Information processing method, robot, electronic equipment and medium
By selecting attention calculation methods based on modal information density in embodied intelligent robots, combining sparse and full attention calculations, the problem of low computing efficiency in multimodal information processing is solved, and efficient information processing is achieved.
Patent Information
- Application Number
- CN202411062030.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-02
- Publication Date
- 2024-11-22
AI Technical Summary
When processing multimodal information, the embodied intelligent robot has low computing efficiency due to the large amount of data, and consumes a lot of computing power and video memory.
By obtaining at least two modal information for the target task, selecting appropriate attention calculation methods based on the density of each modal information, and using information processing methods combining sparse attention calculation and full attention calculation to reduce computing power requirements And improve computing efficiency.
It effectively reduces the computing power demand, improves the computing efficiency, improves the efficiency and accuracy of multimodal information processing, and is suitable for embodied intelligent robots to perform target tasks.
Smart Images

Figure CN119004365A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of robot control technology, and in particular to an information processing method, a robot, an electronic device and a medium. Background Art
[0002] Embodied Intelligence is a multidisciplinary research field. Embodied intelligent robots generally need to process multimodal information when performing tasks. Multimodal information may include information with a large amount of data. When embodied intelligent robots perform related tasks based on multimodal information, they consume a lot of computing power and video memory, resulting in low computing efficiency. Summary of the invention
[0003] In view of this, the embodiments of the present application provide an information processing method, a robot, an electronic device and a medium, which can reduce computing power requirements and improve computing efficiency.
[0004] In a first aspect, an embodiment of the present application provides an information processing method, comprising: obtaining at least two modal information for a target task, the at least two modal information including first modal information and second modal information, the density of the first modal information being higher than the density of the second modal information; performing a first attention calculation based on a first query vector group of the second modal information and a partial value vector of the first modal information to obtain first output information of the first modal information, the first attention calculation including a sparse attention calculation; performing a second attention calculation based on a second query vector group of the second modal information and at least a partial value vector of the second modal information to obtain second output information of the second modal information; performing fusion processing based on the first output information and the second output information to obtain output information for the target task, the output information being used to execute the target task.
[0005] In a second aspect, an embodiment of the present application provides an information processing device, comprising: an acquisition module, used to acquire at least two modal information for a target task, the at least two modal information comprising first modal information and second modal information, the density of the first modal information being higher than the density of the second modal information; a calculation module, used to perform a first attention calculation based on a first query vector group of the second modal information and a partial value vector of the first modal information to obtain first output information of the first modal information, the first attention calculation comprising a sparse attention calculation, and a second attention calculation is performed based on a second query vector group of the second modal information and at least a partial value vector of the second modal information to obtain second output information of the second modal information; a fusion module, used to perform fusion processing based on the first output information and the second output information to obtain output information for the target task, the output information being used to execute the target task.
[0006] In a third aspect, an embodiment of the present application provides a robot, comprising a control module, wherein the control module is used to execute the information processing method described in the first aspect above.
[0007] In a fourth aspect, an embodiment of the present application provides an electronic device, comprising: a processor; and a memory for storing processor executable instructions, wherein the processor is used to execute the information processing method described in the first aspect above.
[0008] In a fifth aspect, an embodiment of the present application provides a computer-readable storage medium, wherein the storage medium stores a computer program, and the computer program is used to execute the information processing method described in the first aspect above.
[0009] In a sixth aspect, an embodiment of the present application provides a computer program product, which includes a computer program. When the computer program is executed by a processor of a computer device, the computer device is enabled to execute the information processing method described in the first aspect above.
[0010] In a seventh aspect, an embodiment of the present application provides a chip, comprising: a processor; and a memory for storing processor executable instructions, wherein the processor is used to execute the information processing method described in the first aspect above.
[0011] The embodiments of the present application provide an information processing method, a robot, an electronic device, and a medium. By obtaining at least two modal information for a target task and selecting a suitable attention calculation method according to the density of each modal information, the computing power requirement can be reduced and the computing efficiency can be improved. BRIEF DESCRIPTION OF THE DRAWINGS
[0012] Figure 1 Shown is a schematic diagram of the system architecture of an information processing system provided by an exemplary embodiment of the present application.
[0013] Figure 2 Shown is a flowchart of an information processing method provided by an exemplary embodiment of the present application.
[0014] Figure 3 Shown is a schematic diagram of an attention calculation method between multiple modal information in a combination provided by an exemplary embodiment of the present application.
[0015] Figure 4a It is a schematic diagram showing a process of collecting signals at multiple first moments by a tactile sensor provided by an exemplary embodiment of the present application.
[0016] Figure 4b It is a schematic diagram showing a process of collecting signals at multiple second moments by an image sensor provided by an exemplary embodiment of the present application.
[0017] Figure 4c Shown is a schematic diagram of signals collected by a language collection device in a preset time period provided by an exemplary embodiment of the present application.
[0018] Figure 5 Shown is a flowchart of an information processing method provided by another exemplary embodiment of the present application.
[0019] Figure 6 Shown is a schematic diagram of a process for obtaining output information provided by an exemplary embodiment of the present application.
[0020] Figure 7 Shown is a schematic diagram of the structure of an information processing device provided by an exemplary embodiment of the present application.
[0021] Figure 8 FIG. 1 is a block diagram of an electronic device for executing an information processing method provided by an exemplary embodiment of the present application. DETAILED DESCRIPTION
[0022] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.
[0023] Application Overview
[0024] Embodied Intelligence is a multidisciplinary research field that combines theories and technologies from multiple fields such as cognitive science, artificial intelligence, robotics, neuroscience and psychology. The core of embodied intelligence is to understand how intelligent behavior is generated through the interaction between the body and the environment. Embodied intelligence emphasizes that intelligence is not just a product of the brain or computing process, but the result of the interaction between the body, brain and environment.
[0025] The field of embodied intelligence involves multimodal technology. In traditional fields, multimodal technology is mainly aimed at image processing, while multimodal technology in the field of embodied intelligence adds the processing of tactile information. Generally, tactile information can be obtained through tactile sensors, such as tactile sensors that simulate the perception of human skin and convert physical contact into electrical signals, which can be used as tactile information. Robots or interactive devices can perceive the surrounding environment through tactile information and interact with the environment.
[0026] Image information is a type of information with a high data volume, and tactile information is also a type of information with a high data volume. A single frame of tactile information can be understood as an image matrix (x, y, z), which contains the friction force in the x and y directions of the contact surface and the magnitude of the force (expressed in the z dimension). The sampling frequency of the tactile sensor is very high, usually at 100 Hz to 1000 Hz. Therefore, the data density of tactile information is very high, which is not conducive to calculation and will consume a lot of computing power and video memory.
[0027] In order to address the technical problem of low computing efficiency caused by the large amount of multimodal information data, an embodiment of the present application provides an information processing method, which can reduce computing power requirements and improve computing efficiency by selecting a suitable attention calculation method according to the density of each modal information in the multimodal information.
[0028] Exemplary Systems
[0029] Figure 1 FIG. 1 is a schematic diagram of a system architecture of an information processing system provided by an exemplary embodiment of the present application. Figure 1 As shown, the information processing system 100 may include a computer device 110, on which a controller 111 may be installed. The controller 111 may be used to execute the information processing method provided in the embodiment of the present application.
[0030] Exemplarily, the computer device may be a server, a personal computer, a mobile terminal, a personal digital assistant, or a robot or other interactive device. The controller 111 may process the input information to obtain output information. For example, the controller 111 may process the input information through a neural network structure to obtain output information. Further, the controller 111 may execute a target task according to the output information.
[0031] The input information may be multimodal information, for example, including at least two of image information, tactile information, and language information.
[0032] The target task can be a target object type recognition task, a target object grasping task, etc.
[0033] In an application scenario example, the user of the computer device 110 may be a technician responsible for training a neural network structure, and he hopes to train the neural network structure through the information processing system 100, and the output information of the neural network structure can be used to perform a target task.
[0034] In another application scenario example, the computer device 110 may be a robot, which may include a trained neural network structure, and the robot may perform a target task according to the output information of the trained neural network structure. In other words, the user of the robot may be a technician who is responsible for using the robot to perform the target task, and who hopes to perform the target task through the information processing system 100.
[0035] In one example, the computer device 110 may obtain first modal information and second modal information for a target task, wherein the density of the first modal information is higher than that of the second modal information. The computer device 110 may perform sparse attention calculation based on the first query vector group of the second modal information and the partial value vector of the first modal information to obtain first output information of the first modal information. The computer device 110 may perform full attention calculation based on the second query vector group of the second modal information and all value vectors of the second modal information to obtain second output information of the second modal information. The computer device 110 may perform fusion processing based on the first output information and the second output information to obtain output information for the target task. Further, when the target task is a target object type recognition task, the computer device 110 may identify the type of the target object according to the output information; when the target task is a target object grasping task, the computer device 110 may control the computer device 110 to grasp the target object according to the output information.
[0036] It should be understood that the above application scenario examples are only shown to facilitate understanding of the spirit and principle of the present application, and the embodiments of the present application are not limited thereto. On the contrary, the embodiments of the present application can be applied to any scenario that may be applicable.
[0037] Exemplary Methods
[0038] Figure 2 Shown is a flowchart of an information processing method provided by an exemplary embodiment of the present application. Figure 2 The method can be Figure 1 The computer equipment in the Figure 2 As shown, the information processing method may include the following contents.
[0039] 210: Obtain at least two modal information for a target task, where the at least two modal information include first modal information and second modal information, and density of the first modal information is higher than density of the second modal information.
[0040] Specifically, the target task may be a task executable by the robot, such as a target object type recognition task, a target object grasping task, a target object moving task, and other tasks that require analysis in combination with multimodal information.
[0041] The at least two modal information, namely, multimodal information, includes first modal information and second modal information. The first modal information and the second modal information are of different types. For example, the first modal information may be tactile information, and the second modal information may be image information.
[0042] In one example, the density of modal information may be an indicator of the amount of information contained in a unit space or a unit time. For example, the density of the first modal information being higher than the density of the second modal information may mean that the sampling frequency of the first modal information is higher than the sampling frequency of the second modal information.
[0043] 220: Perform a first attention calculation based on the first query vector group of the second modal information and the partial value vector of the first modal information to obtain first output information of the first modal information, where the first attention calculation includes a sparse attention calculation.
[0044] Specifically, each modal information corresponds to multiple query vectors and multiple value vectors.
[0045] In one example, the query vector can be used to represent the degree to which the current element pays attention to other elements, and the value vector can carry the representation information of each element. The relationship between the current element and the other element can be calculated based on the query vector of the current element and the value vector of the other element, so that the representation vector of the other element can be updated. This process can be considered as an attention calculation process.
[0046] In one example, sparse attention is an optimized attention mechanism that can obtain multiple output vectors based on multiple query vectors and partial value vectors, thereby reducing computation and memory consumption.
[0047] The first query vector group may include all or part of the query vectors corresponding to the second modal information. Sparse attention calculation is performed based on the first query vector group of the second modal information and the partial value vector of the first modal information to obtain the first output information of the first modal information.
[0048] 230: Perform a second attention calculation based on a second query vector group of the second modal information and at least a portion of the value vector of the second modal information to obtain second output information of the second modal information.
[0049] Specifically, the second query vector group may include all or part of the query vectors corresponding to the second modal information. The second query vector group may be the same as or different from the first query vector group, and may be specifically set according to actual conditions.
[0050] In one example, the second attention calculation may be a full attention calculation. Specifically, the second attention calculation may be performed based on the second query vector group of the second modal information and all value vectors of the second modal information to obtain the second output information of the second modal information.
[0051] In another example, the second attention calculation may be a sparse attention calculation. Specifically, the second attention calculation may be performed based on a second query vector group of the second modal information and a partial value vector of the second modal information to obtain second output information of the second modal information. Optionally, the sparsity of the second attention calculation may be greater than that of the first attention calculation.
[0052] 240: Perform fusion processing based on the first output information and the second output information to obtain output information for the target task, and the output information is used to execute the target task.
[0053] Specifically, the first output information and the second output information may be numerical values, vectors, or matrices. The fusion process may include a splicing process. For example, the first output information and the second output information may be numerical values, and a vector may be obtained through the splicing process, and the vector may be output information for the target task.
[0054] An embodiment of the present application provides an information processing method, which reduces computing power requirements and improves computing efficiency by obtaining at least two modal information for a target task and selecting a suitable attention calculation method according to the density of each modal information.
[0055] According to an embodiment of the present application, the information processing method further includes: performing a first attention calculation based on a third query vector group of the first modal information and a partial value vector of the first modal information to obtain third output information of the first modal information; performing a second attention calculation based on a fourth query vector group of the first modal information and at least a partial value vector of the second modal information to obtain fourth output information of the second modal information. In this embodiment, step 240 may include: performing fusion processing based on the first output information, the second output information, the third output information, and the fourth output information to obtain output information for the target task.
[0056] Specifically, the third query vector group may include some query vectors among all query vectors corresponding to the first modal information. The fourth query vector group may include some query vectors among all query vectors corresponding to the first modal information. The third query vector group may be the same as or different from the fourth query vector group, and may be specifically set according to actual conditions.
[0057] The third output information and the fourth output information are similar to the first output information and the second output information in the foregoing text, that is, the third output information and the fourth output information may be numerical values, vectors, or matrices. For example, the first output information, the second output information, the third output information, and the fourth output information may be numerical values, and the fusion processing may include splicing processing. By splicing the first output information and the second output information, a vector may be obtained; by splicing the third output information and the fourth output information, another vector may be obtained. Splicing these two vectors may obtain a matrix, which may be the output information for the target task.
[0058] In this embodiment, each modal information can be used to perform attention calculation with all modal information (including the modal information itself) to obtain output information. Different attention algorithms can be used according to the density of the modal information being calculated. For example, when the modal information is used to perform attention calculation with modal information with high density, sparse attention calculation can be used to reduce computing power. In addition, the embodiment of the present application can take into account the correlation between each modal information and all modal information, so as to ensure the accuracy of the output information, thereby ensuring the reliability of the execution of the target task.
[0059] According to an embodiment of the present application, step 220 may include: obtaining a first weight group based on the first query vector group; and performing weighted summation on partial value vectors of the first modal information according to the first weight group to obtain first output information.
[0060] Specifically, the first weight group may include multiple weights. The weights may be obtained according to the query vector. For example, the weights corresponding to the value vectors of the first modal information may be obtained by a multi-layer perceptron (MLP) and the query vector of the second modal information. Different weight values corresponding to value vectors may be obtained according to different query vectors to obtain the first query vector group. According to the weights corresponding to each value vector in the partial value vectors of the first modal information, these value vectors are weighted and summed to obtain the first output information.
[0061] Optionally, other methods may be used to obtain weights according to the query vector, which is not limited in the embodiments of the present application.
[0062] In this embodiment, by obtaining the weight corresponding to the value vector of the first modal information based on the query vector of the second modal information, the two modal information can be combined and the degree of influence of the second modal information on the first modal information can be put into the calculation process. This can highlight the key features of the first modal information and thereby improve the reliability of the first output information.
[0063] According to an embodiment of the present application, weighted summing of partial value vectors of the first modal information according to the first weight group to obtain first output information includes: dividing the first weight group based on a preset rule to obtain multiple sub-weight groups; weighted summing of partial value vectors of the first modal information according to the first sub-weight group in the multiple sub-weight groups to obtain the first output information. In this embodiment, step 230 includes: weighted summing of at least partial value vectors of the second modal information according to the second sub-weight group in the multiple sub-weight groups to obtain the second output information.
[0064] Specifically, the preset rules may be rules set manually, or rules for dividing weight groups that are automatically generated by a neural network structure after continuous training.
[0065] According to the preset rules, the first weight group can be divided into multiple sub-weight groups, and the number of multiple sub-weight groups can be consistent with the number of multimodal information. The number of each sub-weight group can be the same or different. Specifically, the number of each sub-weight group can be determined by the preset rules. For example, when the number of multimodal information is 2, the first weight group (assuming that 100 weights are included) can be divided into two groups, the number of weights of the first sub-weight group can be 10, and the number of weights of the second sub-weight group can be 90. For another example, when the number of multimodal information is 3, the first weight group (assuming that 100 weights are included) can be divided into three groups, the number of weights of the first sub-weight group can be 10, the number of weights of the second sub-weight group can be 80, and the number of weights of the third sub-weight group can be 10. The more weights of the sub-weight groups, the more critical or important the corresponding modal information is. In one example, partial value vectors can be collected from all value vectors of the first modal information through sampling and weighted summation can be performed with the first sub-weight group. The number of partial value vectors can be consistent with the number of weights of the first sub-weight group, that is, determined by preset rules. The specific sampling positions of the partial value vectors can be set manually or automatically determined by the neural network structure through continuous training.
[0066] In this embodiment, by dividing the first weight group according to a preset rule, more weight can be assigned to important modal information, thereby improving the calculation efficiency while ensuring the reliability of the output information.
[0067] Furthermore, while obtaining the weight according to the query vector, the bias corresponding to the weight can be obtained according to the query vector, that is, each weight group can correspond to a bias group, and each sub-weight group can correspond to a sub-bias group. The division rule of the bias group can be consistent with the division rule of the weight group. The bias can be used to indicate the sampling position. For example, bias 1 can indicate that the value vector is found at pixel point 1 of the first modal information; the sampling position indicated by the bias can be a spatial sampling position (position on the spatial dimension) and / or a time sampling position (position on the time dimension). According to the sampling position, the corresponding value vector can be collected from the modal information, and the value vector and the weight are weighted and summed. Taking the first weight group of the second modal information as an example, the first weight group and the first bias group corresponding to the value vector of the first modal information can be obtained through a multilayer perceptron and the query vector of the second modal information. The acquisition method of the weight group and the bias group of other modal information is similar to it, and it will not be repeated.
[0068] According to an embodiment of the present application, the first attention calculation includes a multi-scale deformable attention (MSDA) calculation. Specifically, the MSDA model is an improvement on the Transformer architecture in the field of vision, and its algorithm can reduce the complexity of the dense self-attention model from O(n^2) to O(m), that is, it reduces one complexity. The embodiment of the present application applies MSDA to the attention calculation between multimodal information, which can improve the calculation efficiency. Optionally, in other embodiments, the first attention calculation can be implemented by other methods that can implement sparse attention calculation.
[0069] According to an embodiment of the present application, the second attention calculation includes a multi-head attention calculation. Optionally, in other embodiments, the second attention calculation can be implemented by other methods that can implement full attention calculation.
[0070] In one example, the second attention calculation can be implemented through ViT (Vision Transformer). ViT is a neural network architecture mainly used for computer vision tasks. It adopts the structure of the Transformer model, which was originally designed for natural language processing tasks. ViT divides the image into a series of small patches (patches), and then processes these small patches as sequences, using the self-attention mechanism of the Transformer to capture the relationship between different image patches. In this example, each pixel contained in the first modal information and the second modal information can be put into the self-attention model (the core operation step in the Transformer), and the relationship between all pixels is calculated.
[0071] According to an embodiment of the present application, the fusion processing includes normalization processing. The fusion processing is performed based on the first output information, the second output information, the third output information, and the fourth output information to obtain output information for the target task, including: performing normalization processing based on the first output information and the second output information to obtain first normalized information; performing normalization processing based on the third output information and the fourth output information to obtain second normalized information; and obtaining output information for the target task based on the first normalized information and the second normalized information.
[0072] Specifically, in one example, a weight group is obtained according to multiple query vectors corresponding to the second modal information, and a first attention calculation is performed on a partial value vector of the first modal information according to a partial weight in the weight group, so that the first output information can be obtained. Similarly, a second attention calculation is performed on at least a partial value vector of the second modal information according to the remaining weights in the weight group, so that the second output information can be obtained. Similarly, a weight group is obtained according to multiple query vectors corresponding to the first modal information, and a first attention calculation is performed on a partial value vector of the first modal information according to a partial weight in the weight group, so that the third output information can be obtained. Similarly, a second attention calculation is performed on at least a partial value vector of the second modal information according to the remaining weights in the weight group, so that the fourth output information can be obtained.
[0073] Further, for the second modal information, the output information corresponding to the multiple modal information obtained based on its query vector (such as the first output information and the second output information) can be normalized to obtain the first normalized information. Specifically, the first normalized information can be a vector formed by concatenating the normalized first output information and the normalized second output information. Similarly, for the first modal information, the output information corresponding to the multiple modal information obtained based on its query vector (such as the third output information and the fourth output information) can be normalized to obtain the second normalized information. Specifically, the second normalized information can be a vector formed by concatenating the normalized third output information and the normalized fourth output information. The first normalized information and the second normalized information can be combined into a matrix, which can be the output information for the target task.
[0074] It should be understood that after obtaining a weight group based on multiple query vectors, the multiple weights in the weight group can be normalized, and then attention calculation is performed on the value vectors of each modal information based on the normalized weight group to obtain corresponding output information.
[0075] Preferably, after obtaining a weight group according to multiple query vectors, the multiple weights in the weight group may not be normalized, and attention calculation may be performed directly on the value vectors of each modal information based on the weight group to obtain the corresponding output information. Furthermore, after obtaining output information corresponding to multiple modal information based on the query vector corresponding to the current modal information, the output information corresponding to the multiple modal information is normalized, which can comprehensively consider the correlation between the current modal information and the multiple modal information, and improve the reliability of the normalized information.
[0076] In this embodiment, the output information corresponding to multiple modal information obtained based on query vectors of different modal information can be normalized separately, so that the degree of influence of the same modal information on all other modal information can be compared together, and the degree of influence of different modal information on all other modal information can be compared separately, so as to intuitively reflect the importance of the same modal information to all other modal information, thereby improving the reliability of the output information for the target task.
[0077] According to an embodiment of the present application, step 210 may include: splitting at least two types of original information into multiple groups respectively, wherein the at least two modal information includes a group corresponding to each of the at least two types of original information.
[0078] Specifically, each type of modal information may be split from a type of original information. The type of the original information is the same as the type of the modal information, and can be considered as complete modal information. The original information is split into multiple groups, and the information of each group is the modal information mentioned in step 210. Each type of original information is divided separately, and the number of multiple groups obtained based on each original information can be the same or different.
[0079] In one example, at least two kinds of original information may include first original information and second original information. By dividing the first original information, multiple groups of first modal information can be obtained; by dividing the second original information, multiple groups of second modal information can be obtained. For each group of first modal information, attention calculation can be performed based on its query vector and the value vector of each group of second modal information in the multiple groups of second modal information, and attention calculation can be performed based on its query vector and the value vector of each group of first modal information in the multiple groups of first modal information. Similarly, for each group of second modal information, attention calculation can be performed based on its query vector and the value vector of each group of first modal information in the multiple groups of first modal information, and attention calculation can be performed based on its query vector and the value vector of each group of second modal information in the multiple groups of second modal information.
[0080] Optionally, for each group of first modal information, attention calculation may be performed based on its query vector and the value vector of each group of second modal information in at least one group of second modal information, and attention calculation may be performed based on its query vector and the value vector of each group of first modal information in at least one group of first modal information. Similarly, for each group of second modal information, attention calculation may be performed based on its query vector and the value vector of each group of first modal information in at least one group of first modal information, and attention calculation may be performed based on its query vector and the value vector of each group of second modal information in at least one group of second modal information.
[0081] For example, the at least two modal information include first modal information and second modal information, and any group of first modal information can be combined with any group of second modal information, and each combination is as follows: Figure 3 In each combination, the first attention calculation can be performed based on the query vector of the first modal information and the value vector of the first modal information, that is, Figure 3 The first square in the first row of the matrix shown in the figure; the second attention calculation is performed based on the query vector of the first modal information and the value vector of the second modal information, that is, corresponding to Figure 3 The second square in the first row of the matrix shown in the figure; the first attention calculation is performed based on the query vector of the second modal information and the value vector of the first modal information, that is, corresponding to Figure 3 The first square in the second row of the matrix shown in the figure; the second attention calculation is performed according to the query vector of the second modal information and the value vector of the second modal information, that is, corresponding to Figure 3 The second square in the second row of the matrix shown.
[0082] Further, at least two modal information also include third modal information. Any group of first modal information, any group of second modal information and any group of third modal information can be combined, and each combination is as follows: Figure 3 In each combination, the second attention calculation can be performed based on the query vector of the first modal information and the value vector of the third modal information, that is, Figure 3 The third square in the first row of the matrix shown in the figure; the second attention calculation is performed based on the query vector of the second modal information and the value vector of the third modal information, that is, corresponding to Figure 3 The third square in the second row of the matrix shown in the figure; the first attention calculation is performed based on the query vector of the third modal information and the value vector of the first modal information, that is, corresponding to Figure 3 The first square in the third row of the matrix shown in the figure. The second attention calculation is performed based on the query vector of the third modal information and the value vector of the second modal information, that is, the corresponding Figure 3 The second square in the third row of the matrix shown in the figure; the second attention calculation is performed based on the query vector of the third modal information and the value vector of the third modal information, that is, corresponding to Figure 3 The third square in the third row of the matrix shown.
[0083] The attention calculation process between the third modal information and all modal information can refer to the attention calculation process between the first modal information and all modal information mentioned above. To avoid repetition, it will not be repeated here.
[0084] In one example, the first modality information may be tactile information, the second modality information may be image information, and the third modality information may be language information. In one example, when the amount of image information is large, Figure 3 Each square in the second column of the shown matrix may be calculated using the first attention.
[0085] In one example, the method for splitting the modal information into multiple groups may be an equal division method, or a method for splitting the modal information into multiple groups according to the importance of the information. Of course, other methods may also be used to split the modal information into multiple groups.
[0086] In this embodiment, at least two types of original information are split into multiple groups to obtain multiple groups of first modal information and multiple groups of second modal information, and any group of first modal information and any group of second modal information are combined. In each combination, attention calculation is performed on the first modal information and the second modal information, which can reduce the computing power in each combination and improve computing efficiency. In addition, by splitting the modal information into multiple groups, the modal information in each group can be taken into account, improving information utilization and improving the reliability of output information.
[0087] According to an embodiment of the present application, the fusion processing includes normalization processing and splicing processing, and step 240 may include: performing normalization processing based on the first output information and the second output information to obtain first normalized information; performing splicing processing on the first normalized information corresponding to each group in the multiple groups to obtain output information for the target task.
[0088] Specifically, by concatenating the first output information and the second output information, a vector can be obtained. For example, Figure 3 The output information corresponding to each block in the first row of the matrix shown in FIG. 1 is concatenated to obtain a vector; similarly, another vector can be obtained by concatenating the third output information and the fourth output information. For example, Figure 3 The output information corresponding to each block in the second row of the matrix shown is concatenated to obtain another vector.
[0089] Furthermore, the two vectors are concatenated to obtain a matrix. The matrices corresponding to each of the multiple combinations of the first modal information and the second modal information are concatenated (i.e., the first normalized information corresponding to each group is concatenated, and the second normalized information corresponding to each group is concatenated), and the output information for the target task can be obtained.
[0090] According to one embodiment of the present application, the first modal information includes tactile information, and the tactile information includes signals collected by a tactile sensor at multiple first moments. The second modal information includes image information or language information, and the image information includes multiple frames of images collected by an image acquisition device at multiple second moments. The language information includes signals collected by a language acquisition device during a preset time period.
[0091] Specifically, the tactile information includes signals collected by the tactile sensor at multiple first moments. For example, for a scenario in which a tactile sensor is used to determine the material of a target object, the tactile sensor can be controlled to slide across the surface of the target object to sense the surface roughness of the target object, so as to form continuous different frames in the tactile sensor. The material of the target object can be determined based on the continuous different frames. The sampling frequency of the tactile sensor can be a high frequency, such as a sampling frequency of 100 Hz to 1000 Hz. The number of tactile sensors can be one or more.
[0092] In one example, the tactile sensor may include an array of multiple sensor units, each of which can be regarded as a pixel point, capable of detecting and responding to local tactile information. The tactile information collected by the tactile sensor in the embodiment of the present application may include information in a certain spatial dimension; in addition, the tactile information may include information in a time dimension, that is, multiple frames. Figure 4a The figure shows a process of a tactile sensor collecting signals at multiple first moments, wherein T represents the tactile sensor and the subscript of T represents the moment.
[0093] In one example, the image information includes multiple frames of images collected by an image acquisition device at multiple second moments, and the image acquisition device may be an image sensor (or a camera). The sampling frequency of the image sensor may be a medium frequency, such as the sampling frequency of the image sensor may be lower than the sampling frequency of the tactile sensor. The number of image sensors may be one or more. The image information collected by the image sensor in the embodiment of the present application may include information on a certain spatial dimension; in addition, the image information may include information on a time dimension, that is, it may include multiple frames of images. In one example, the image information may be the original image information collected by the image sensor, or a high-dimensional feature vector obtained by extracting features from the original image information. The amount of data of the high-dimensional feature vector may be less than the amount of data of the original image information. For example, when the image information is the original image information, the first attention algorithm may be used to perform attention calculation on the image information; when the image information is a high-dimensional feature vector, the second attention algorithm may be used to perform attention calculation on the image information. Figure 4b The process of an image sensor collecting signals at multiple second moments is shown, wherein T represents the image sensor and the subscript of T represents the moment.
[0094] In one example, the language information includes a signal collected by a language collection device during a preset time period, and the language collection device may be a microphone or other device. The sampling frequency of the language collection device may be a low frequency, such as the sampling frequency of the language collection device may be lower than the sampling frequency of the image sensor. The number of language collection devices may be one or more. Figure 4c The figure shows a signal collected by a language collection device in a preset time period, wherein T represents the language collection device and the subscript of T may represent the preset time period.
[0095] In one example, tactile information can be considered as semi-dense high-frequency data, image information can be considered as dense medium-frequency data, and language information can be considered as low-density low-frequency data. Semi-dense and dense can represent the amount of data collected at a moment, and the amount of data corresponding to semi-dense can be less than the amount of data corresponding to dense; the information sampling frequency corresponding to high frequency is lower than the information sampling frequency corresponding to medium frequency, and the information sampling frequency corresponding to medium frequency is lower than the information sampling frequency corresponding to low frequency.
[0096] In this embodiment, the tactile information includes signals collected by the tactile sensor at multiple first moments, and the image information includes multiple frames of images collected by the image acquisition device at multiple second moments, so that the tactile information and the image information may include information in the time dimension. When processing the tactile information and the image information, the embodiment of the present application can design the attention in the time dimension as sparse attention, which can help the network learn the key parts of high-frequency data (such as tactile information) while reducing the consumption of computing video memory and computing power. Further, when the image information includes information collected by multiple image sensors, the embodiment of the present application can perform sparse attention calculation between multi-view image information and tactile information, which can help efficient information transmission between multi-view image information and realize efficient real-time operation of multiple modal terminals. In addition, the embodiment of the present application can extend the calculation of a single frame image in the multi-scale deformable attention calculation method to the temporal multi-modal method, and use the multi-scale deformable attention calculation method to process the high-density first modal information, which can improve the calculation efficiency.
[0097] Figure 5 Shown is a flowchart of an information processing method provided by another exemplary embodiment of the present application. Figure 5 The embodiment is Figure 2 For the example of the embodiment, in order to avoid repetition, the same points can be referred to the description in the above embodiment, which will not be repeated here. Figure 5 As shown, the information processing method may include the following contents.
[0098] 510: Acquire three kinds of original information for the target task, where the three kinds of original information include first original information, second original information, and third original information.
[0099] 520: Split the three types of original information into multiple groups respectively to obtain multiple groups of first modal information, multiple groups of second modal information and multiple groups of third modal information, and combine any group of the first modal information, any group of the second modal information and any group of the third modal information to obtain multiple combinations.
[0100] The first modality information includes tactile information, the second modality information includes image information, and the third modality information includes language information. The density of the first modality information is higher than the density of the second modality information and higher than the density of the third modality information. The sizes of the three original information may be consistent or inconsistent.
[0101] 530: In each combination, the query vector group based on the first modal information performs attention calculation with the partial value vectors of the first modal information, the entire value vectors of the second modal information, and the entire value vectors of the third modal information respectively; the query vector group based on the second modal information performs attention calculation with the partial value vectors of the first modal information, the entire value vectors of the second modal information, and the entire value vectors of the third modal information respectively; the query vector group based on the third modal information performs attention calculation with the partial value vectors of the first modal information, the entire value vectors of the second modal information, and the entire value vectors of the third modal information respectively, to obtain output information corresponding to the combination.
[0102] Combination Figure 3 and Figure 6 , under each combination, a weight group and a bias group can be obtained based on the query vector group of the first modal information, and the weight group is divided into the first, second and third sub-weight groups, and the bias group is divided into the first, second and third sub-bias groups. According to the first sub-weight group and the first sub-bias group, a weighted sum (first attention calculation) is performed on the partial value vectors of the first modal information to obtain the output information (tactile information value vector, i.e., tactile information Value); according to the second sub-weight group and the second sub-bias group, a weighted sum (second attention calculation) is performed on at least part of the value vectors of the second modal information to obtain the output information (image information value vector, i.e., image information Value); according to the third sub-weight group and the third sub-bias group, a weighted sum (second attention calculation) is performed on at least part of the value vectors of the third modal information to obtain the output information (language information value vector, i.e., language information Value). It should be understood that the first, second and third in the embodiment of the present application are only used to distinguish different information and are not specific.
[0103] Similarly, based on the query vector group of the second modal information, a weight group and a bias group can be obtained, and the weight group is divided into the first, second and third sub-weight groups, and the bias group is divided into the first, second and third sub-bias groups. According to the first sub-weight group and the first sub-bias group, a weighted sum (first attention calculation) is performed on the partial value vectors of the first modal information to obtain output information (tactile information value vector, i.e., tactile information Value); according to the second sub-weight group and the second sub-bias group, a weighted sum (second attention calculation) is performed on at least part of the value vectors of the second modal information to obtain output information (image information value vector, i.e., image information Value); according to the third sub-weight group and the third sub-bias group, a weighted sum (second attention calculation) is performed on at least part of the value vectors of the third modal information to obtain output information (language information value vector, i.e., language information Value).
[0104] Similarly, based on the query vector group of the third modal information, a weight group and a bias group can be obtained, and the weight group is divided into the first, second and third sub-weight groups, and the bias group is divided into the first, second and third sub-bias groups. According to the first sub-weight group and the first sub-bias group, a weighted sum (first attention calculation) is performed on the partial value vectors of the first modal information to obtain output information (tactile information value vector, i.e., tactile information Value); according to the second sub-weight group and the second sub-bias group, a weighted sum (second attention calculation) is performed on at least part of the value vectors of the second modal information to obtain output information (image information value vector, i.e., image information Value); according to the third sub-weight group and the third sub-bias group, a weighted sum (second attention calculation) is performed on at least part of the value vectors of the third modal information to obtain output information (language information value vector, i.e., language information Value).
[0105] According to the above description, the output information under each combination can be obtained, and the output information can be a 3*3 matrix. Specifically, the first row of the 3*3 matrix can be the output information (tactile information Value, image information Value, and language information Value) obtained based on the query vector group of the first modal information; the second row of the 3*3 matrix can be the output information (tactile information Value, image information Value, and language information Value) obtained based on the query vector group of the second modal information; the third row of the 3*3 matrix can be the output information (tactile information Value, image information Value, and language information Value) obtained based on the query vector group of the third modal information.
[0106] 540: Concatenate the output information corresponding to each combination to obtain output information for the target task.
[0107] Specifically, the output information corresponding to each combination can be concatenated, and the obtained output tensor is the output information for the target task.
[0108] The information processing method provided in the embodiment of the present application can be used for the robot to perform a target task, which requires processing of multimodal information. Specifically, the information processing method provided in the embodiment of the present application can be applied to the process of embodied intelligent real-time strategy network reasoning, and network reasoning can refer to the process of entering the network from the input end to complete the reasoning. The information processing method provided in the embodiment of the present application can be deployed on the terminal side, and the running speed is fast, which can reach 10hz and complete the reasoning within 100ms.
[0109] Exemplary Devices
[0110] Figure 7 FIG. 7 is a schematic diagram of the structure of an information processing device 700 provided by an exemplary embodiment of the present application. Figure 7 As shown, the information processing device 700 includes: an acquisition module 710 , a calculation module 720 and a fusion module 730 .
[0111] The acquisition module 710 is used to acquire at least two modal information for the target task, the at least two modal information include first modal information and second modal information, and the density of the first modal information is higher than the density of the second modal information. The calculation module 720 is used to: perform a first attention calculation based on a first query vector group of the second modal information and a partial value vector of the first modal information to obtain first output information of the first modal information, the first attention calculation includes a sparse attention calculation; perform a second attention calculation based on a second query vector group of the second modal information and at least a partial value vector of the second modal information to obtain second output information of the second modal information. The fusion module 730 is used to perform fusion processing based on the first output information and the second output information to obtain output information for the target task, and the output information is used to execute the target task.
[0112] An embodiment of the present application provides an information processing device, which obtains at least two modal information for a target task and selects a suitable attention calculation method according to the density of each modal information, thereby reducing computing power requirements and improving computing efficiency.
[0113] According to an embodiment of the present application, the calculation module 720 is further used to: perform a first attention calculation based on the third query vector group of the first modal information and a partial value vector of the first modal information to obtain third output information of the first modal information; perform a second attention calculation based on the fourth query vector group of the first modal information and at least a partial value vector of the second modal information to obtain fourth output information of the second modal information. The fusion module 730 is used to: perform fusion processing based on the first output information, the second output information, the third output information and the fourth output information to obtain output information for the target task.
[0114] According to an embodiment of the present application, the calculation module 720 is used to: obtain a first weight group based on the first query vector group; and perform weighted summation on partial value vectors of the first modal information according to the first weight group to obtain first output information.
[0115] According to an embodiment of the present application, the calculation module 720 is used to: divide the first weight group based on a preset rule to obtain multiple sub-weight groups; perform weighted summation on the partial value vectors of the first modal information according to the first sub-weight group in the multiple sub-weight groups to obtain the first output information. The calculation module 720 is used to: perform weighted summation on at least a partial value vector of the second modal information according to the second sub-weight group in the multiple sub-weight groups to obtain the second output information.
[0116] According to an embodiment of the present application, the first attention calculation includes multi-scale deformable attention calculation.
[0117] According to one embodiment of the present application, the second attention calculation includes multi-head attention calculation.
[0118] According to an embodiment of the present application, the fusion process includes normalization processing. The fusion module 730 is used to: perform normalization processing based on the first output information and the second output information to obtain first normalized information; perform normalization processing based on the third output information and the fourth output information to obtain second normalized information; and obtain output information for the target task based on the first normalized information and the second normalized information.
[0119] According to an embodiment of the present application, the acquisition module 710 is used to: split at least two types of original information into multiple groups respectively, wherein the at least two modal information includes a group corresponding to each of the at least two types of original information.
[0120] According to an embodiment of the present application, the fusion process includes normalization and splicing. The fusion module 730 is used to: perform normalization based on the first output information and the second output information to obtain first normalized information; and perform splicing on the first normalized information corresponding to each group in the multiple groups to obtain output information for the target task.
[0121] According to one embodiment of the present application, the first modal information includes tactile information, and the tactile information includes signals collected by a tactile sensor at multiple first moments. The second modal information includes image information or language information, and the image information includes multiple frames of images collected by an image acquisition device at multiple second moments. The language information includes signals collected by a language acquisition device during a preset time period.
[0122] It should be understood that the operations and functions of the acquisition module 710, the calculation module 720 and the fusion module 730 in the above embodiment can refer to the above Figure 2 or Figure 5In order to avoid repetition, the description of the information processing method provided in the embodiment will not be repeated here.
[0123] The embodiment of the present application also provides a robot, the robot includes a control module, and the control module is used to execute the information processing method provided by any embodiment of the present application. In some embodiments, the robot can be an embodied intelligent robot.
[0124] Figure 8 FIG. 8 is a block diagram of an electronic device 800 for executing an information processing method provided by an exemplary embodiment of the present application.
[0125] Reference Figure 8 , the electronic device 800 includes a processing component 810, which further includes one or more processors, and a memory resource represented by a memory 820 for storing instructions executable by the processing component 810, such as an application. The application stored in the memory 820 may include one or more modules, each corresponding to a set of instructions. In addition, the processing component 810 is configured to execute instructions to perform the above-mentioned information processing method.
[0126] The electronic device 800 may also include a power supply component configured to perform power management of the electronic device 800, a wired or wireless network interface configured to connect the electronic device 800 to a network, and an input / output (I / O) interface. The electronic device 800 may be operated based on an operating system stored in the memory 820, such as Windows Server 2000. TM , Mac OSX TM , Unix TM , Linux TM , FreeBSD TM or similar.
[0127] A non-transitory computer-readable storage medium, when instructions in the storage medium are executed by a processor of the electronic device 800, enables the electronic device 800 to execute an information processing method.
[0128] All the above optional technical solutions can be arbitrarily combined to form optional embodiments of the present application, and will not be described one by one here.
[0129] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.
[0130] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.
[0131] In the several embodiments provided in the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.
[0132] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.
[0133] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.
[0134] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application can be essentially or partly embodied in the form of a software product that contributes to the prior art. The computer software product is stored in a storage medium, including several instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the methods described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program check codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.
[0135] It should be noted that, in the description of this application, the terms "first", "second", "third", etc. are used for descriptive purposes only and cannot be understood as indicating or implying relative importance. In addition, in the description of this application, unless otherwise specified, the meaning of "plurality" is two or more.
[0136] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and provide corresponding operation entrances for users to choose to authorize or refuse.
[0137] The above description is only a preferred embodiment of the present application and is not intended to limit the present application. Any modifications, equivalent substitutions, etc. made within the spirit and principles of the present application shall be included in the protection scope of the present application.
Claims
1. An information processing method, characterized in that: include: Acquire at least two modal information for a target task, the at least two modal information comprising first modal information and second modal information, wherein density of the first modal information is higher than density of the second modal information; Performing a first attention calculation based on a first query vector group of the second modal information and a partial value vector of the first modal information to obtain first output information of the first modal information, wherein the first attention calculation includes a sparse attention calculation; Performing a second attention calculation based on a second query vector group of the second modal information and at least a portion of the value vectors of the second modal information to obtain second output information of the second modal information; A fusion process is performed based on the first output information and the second output information to obtain output information for the target task, and the output information is used to execute the target task.
2. The information processing method according to claim 1, characterized in that: Also includes: Performing the first attention calculation based on a third query vector group of the first modal information and a partial value vector of the first modal information to obtain third output information of the first modal information; performing the second attention calculation based on a fourth query vector group of the first modal information and at least a portion of the value vectors of the second modal information to obtain fourth output information of the second modal information, The fusing process based on the first output information and the second output information to obtain the output information for the target task includes: The fusion processing is performed based on the first output information, the second output information, the third output information and the fourth output information to obtain output information for the target task.
3. The information processing method according to claim 1, characterized in that: The first attention calculation is performed based on the first query vector group of the second modal information and the partial value vector of the first modal information to obtain first output information of the first modal information, including: Obtaining a first weight group based on the first query vector group; The first output information is obtained by performing weighted summation on partial value vectors of the first modal information according to the first weight group.
4. The information processing method according to claim 3, characterized in that: The step of performing weighted summation on partial value vectors of the first modal information according to the first weight group to obtain the first output information includes: Dividing the first weight group based on a preset rule to obtain a plurality of sub-weight groups; performing weighted summation on partial value vectors of the first modal information according to a first sub-weight group among the multiple sub-weight groups to obtain the first output information, The performing of a second attention calculation based on a second query vector group of the second modal information and at least a portion of the value vector of the second modal information to obtain second output information of the second modal information includes: The second output information is obtained by performing weighted summation on at least part of the value vectors of the second modal information according to a second sub-weight group among the multiple sub-weight groups.
5. The information processing method according to claim 1, characterized in that: The first attention calculation includes a multi-scale deformable attention calculation.
6. The information processing method according to claim 1, characterized in that: The second attention calculation includes multi-head attention calculation.
7. The information processing method according to claim 2, characterized in that: The fusion processing includes normalization processing, and the fusion processing is performed based on the first output information, the second output information, the third output information, and the fourth output information to obtain output information for the target task, including: Performing normalization processing based on the first output information and the second output information to obtain first normalized information; Performing normalization processing based on the third output information and the fourth output information to obtain second normalized information; Output information for the target task is obtained according to the first normalized information and the second normalized information.
8. The information processing method according to claim 1, characterized in that: The obtaining of at least two modal information for the target task includes: The at least two types of original information are respectively divided into a plurality of groups, wherein the at least two types of modal information include a group corresponding to each of the at least two types of original information.
9. The information processing method according to claim 8, characterized in that: The fusion processing includes normalization processing and splicing processing, and the fusion processing based on the first output information and the second output information to obtain output information for the target task includes: Performing normalization processing based on the first output information and the second output information to obtain first normalized information; The first normalized information corresponding to each group in the multiple groups is concatenated to obtain output information for the target task.
10. The information processing method according to any one of claims 1 to 9, characterized in that: The first modal information includes tactile information, which includes signals collected by a tactile sensor at multiple first moments. The second modal information includes image information or language information, which includes multiple frames of images collected by an image acquisition device at multiple second moments. The language information includes signals collected by a language acquisition device during a preset time period.
11. A robot, characterized in that: It comprises a control module, wherein the control module is used to execute the information processing method according to any one of claims 1 to 10.
12. An electronic device, characterized in that: include: processor; a memory for storing instructions executable by the processor, Wherein, the processor is used to execute the information processing method described in any one of claims 1 to 10.
13. A computer-readable storage medium, characterized in that: The storage medium stores a computer program, and the computer program is used to execute the information processing method according to any one of claims 1 to 10.
14. A computer program product, characterized in that The computer program product includes a computer program, and when the computer program is executed by a processor of a computer device, the computer device is enabled to execute the information processing method according to any one of claims 1 to 10.
