Joint attention processing method and system and electronic equipment

Abstract

The invention provides a joint attention processing method and system and electronic equipment. The joint attention processing method comprises the steps that a to-be-processed target is acquired; sorting the to-be-processed targets in at least one target dimension to obtain re-sorted to-be-processed targets; performing joint attention processing on the reordered to-be-processed targets to obtain an initial result; and recovering the sequence of the initial result in the target dimension to be consistent with the to-be-processed target so as to obtain a processing result of the to-be-processed target. According to the joint attention processing method, the dimension of joint attention processing can be expanded, the processing efficiency is improved, and the flexibility and mobility of joint attention processing are enhanced.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of deep learning, and relates to a joint attention processing method, in particular to a joint attention processing method, a system and an electronic device. BACKGROUND

[0002] In the fields of computer vision and natural language processing, self-attention mechanism has become a core component. Although the traditional global self-attention mechanism is powerful, it has high computational complexity, and when processing high-resolution images or long text sequences, its quadratic complexity leads to excessive demand for computing resources and low efficiency. Window multi-head self-attention emerges as the times require, which divides the feature map into non-overlapping local windows and independently calculates multi-head self-attention in each window, and the computational complexity is reduced to linear relationship, greatly improving the efficiency of processing high-resolution data. However, window multi-head self-attention is calculated in local windows, which limits the transmission of information between windows and makes it difficult to capture global information. SUMMARY

[0003] The application aims to provide a joint attention processing method, a system and an electronic device, which can solve the problem that the attention calculation in the prior art cannot balance the computational efficiency and global information.

[0004] In a first aspect, the application provides a joint attention processing method. The joint attention processing method comprises: obtaining a target to be processed; performing sorting processing on the target to be processed in at least one target dimension to obtain a reordered target to be processed; performing joint attention processing on the reordered target to be processed to obtain an initial result; and restoring the order of the initial result in the target dimension to be consistent with the target to be processed to obtain a processing result of the target to be processed.

[0005] In the application, the target to be processed is sorted in the target dimension, and the joint attention processing is performed on the reordered target to be processed to obtain an initial result. The initial result is restored to the original order of the target to be processed in the target dimension to obtain a processing result of the target to be processed. The joint attention processing method of the application can expand the dimension of joint attention processing, improve the processing efficiency, and enhance the flexibility and migration of joint attention processing.

[0006] In an implementation manner of the first aspect, the at least one target dimension is obtained according to the dimension of the target to be processed and the dimension of the joint attention processing on the target to be processed.

[0007] In an implementation form of the first aspect, when the target to be processed is a one-dimensional target to be processed, the obtaining the processing result of the one-dimensional target to be processed comprises: performing one-dimensional causal convolution processing on the one-dimensional target to be processed to obtain a causal convolution result; performing sorting processing on a target dimension of the one-dimensional target to be processed to obtain a reordered one-dimensional target to be processed; performing one-dimensional window causal self-attention processing on the reordered one-dimensional target to be processed to obtain a window causal self-attention result; and restoring an order of the window causal self-attention result in the target dimension to be consistent with the one-dimensional target to be processed to combine the causal convolution result to obtain the processing result of the one-dimensional target to be processed.

[0008] In an implementation form of the first aspect, when the target to be processed is a one-dimensional target to be processed, the obtaining the processing result of the one-dimensional target to be processed comprises: performing encoding processing on the target dimension to obtain a position encoding of the one-dimensional target to be processed; performing one-dimensional window causal self-attention processing on the one-dimensional target to be processed to obtain a first window causal self-attention result; performing sorting processing on a target dimension of the one-dimensional target to be processed to obtain a reordered one-dimensional target to be processed; performing one-dimensional window causal self-attention processing on the reordered one-dimensional target to be processed to obtain a second window causal self-attention result; and restoring an order of the second window causal self-attention result in the target dimension to be consistent with the one-dimensional target to be processed to combine the first window causal self-attention result to obtain the processing result of the one-dimensional target to be processed.

[0009] In an implementation form of the first aspect, when the target to be processed is a two-dimensional target to be processed, the obtaining the processing result of the two-dimensional target to be processed comprises: performing two-dimensional depthwise separable convolution processing on the two-dimensional target to be processed to obtain a depthwise separable convolution result; performing sorting processing on a target dimension of the two-dimensional target to be processed to obtain a reordered two-dimensional target to be processed; performing window self-attention processing on the reordered two-dimensional target to be processed to obtain a window self-attention result; and restoring an order of the window self-attention result in the target dimension to be consistent with the two-dimensional target to be processed to combine the depthwise separable convolution result to obtain the processing result of the two-dimensional target to be processed.

[0010] In an implementation form of the first aspect, when the target to be processed is a three-dimensional target to be processed, the obtaining the processing result of the three-dimensional target to be processed comprises: performing encoding processing on another dimension outside a target dimension of the three-dimensional target to be processed to obtain a position encoding of the three-dimensional target to be processed; performing two-dimensional depth-wise separable convolution processing on the three-dimensional target to be processed to obtain a two-dimensional depth-wise separable convolution result; performing sorting processing on the target dimension of the three-dimensional target to be processed to obtain a reordered three-dimensional target to be processed; performing window self-attention processing on the reordered three-dimensional target to be processed to obtain a window self-attention result; and restoring the order of the window self-attention result in the target dimension to be consistent with the three-dimensional target to be processed to combine the two-dimensional depth-wise separable convolution result to obtain the processing result of the three-dimensional target to be processed.

[0011] In an implementation form of the first aspect, when the target to be processed is a three-dimensional target to be processed, the obtaining the processing result of the three-dimensional target to be processed comprises: performing three-dimensional depth-wise separable convolution processing on the three-dimensional target to be processed to obtain a three-dimensional depth-wise separable convolution result; performing sorting processing on a target dimension of the three-dimensional target to be processed to obtain a reordered three-dimensional target to be processed; performing three-dimensional window self-attention processing on the reordered three-dimensional target to be processed to obtain a three-dimensional window self-attention result; and restoring the order of the three-dimensional window self-attention result in the target dimension to be consistent with the three-dimensional target to be processed to combine the three-dimensional depth-wise separable convolution result to obtain the processing result of the three-dimensional target to be processed.

[0012] In an implementation form of the first aspect, when the target to be processed is a three-dimensional target to be processed, the obtaining the processing result of the three-dimensional target to be processed comprises: performing two-dimensional depth-wise separable convolution processing on the three-dimensional target to be processed to obtain a two-dimensional depth-wise separable convolution result; performing sorting processing on a target dimension of the three-dimensional target to be processed to obtain a reordered three-dimensional target to be processed; performing three-dimensional window self-attention processing on the reordered three-dimensional target to be processed and restoring the order of the three-dimensional window self-attention result in the target dimension to be consistent with the three-dimensional target to be processed to obtain a three-dimensional window self-attention result; performing one-dimensional depth-wise separable convolution processing on the three-dimensional target to be processed in a depth dimension to obtain a one-dimensional depth-wise separable convolution result; and obtaining the processing result of the three-dimensional target to be processed according to the two-dimensional depth-wise separable convolution result, the three-dimensional window self-attention result and the one-dimensional depth-wise separable convolution result.

[0013] In a second aspect, the present application provides a joint attention processing system. The joint attention processing system comprises: a data acquisition module configured to acquire a target to be processed; a sorting processing module configured to perform sorting processing on the target to be processed in at least one target dimension to obtain a re-ordered target to be processed; a joint processing module configured to perform joint attention processing on the re-ordered target to be processed to obtain an initial result; and a result acquisition module configured to restore the order of the initial result in the target dimension to be consistent with the target to be processed to obtain a processing result of the target to be processed.

[0014] In a third aspect, the present application provides an electronic device. The electronic device comprises: a memory configured to store a computer program; and a processor configured to execute the computer program stored in the memory, so that the electronic device performs the joint attention processing method according to any one of the first aspect. BRIEF DESCRIPTION OF DRAWINGS

[0015] Figure 1A An application scenario diagram of the joint attention processing method according to the present application is shown.

[0016] Figure 1A A structure diagram of an end-cloud interaction scenario according to the implementation modes is shown.

[0017] Figure 2A A flow diagram of the joint attention processing method according to the embodiments of the present application is shown.

[0018] Figure 2A A diagram of the joint attention processing method according to the embodiments of the present application is shown.

[0019] Figure 3 A diagram of the joint attention processing method according to the embodiments of the present application is shown.

[0020] Figure 4 A diagram of the joint attention processing method according to the embodiments of the present application is shown.

[0021] Figure 5 A diagram of the joint attention processing method according to the embodiments of the present application is shown.

[0022] Figure 6 A diagram of the joint attention processing method according to the embodiments of the present application is shown.

[0023] Figure 7 A diagram of the joint attention processing method according to the embodiments of the present application is shown.

[0024] Figure 8A joint attention processing diagram of a three-dimensional target to be processed is shown.

[0025] Figure 9A A diagram of obtaining a processing result is shown.

[0026] Figure 9B A diagram of obtaining a processing result is shown.

[0027] Figure 9C A diagram of obtaining a processing result is shown.

[0028] Figure 10 A structural diagram of a joint attention processing system is shown.

[0029] Figure 11 A structural diagram of an electronic device is shown.

[0030] Element number explanation

[0031] 1 Joint attention processing device

[0032] 11 Storage device

[0033] 12 Local processor

[0034] 13 Display terminal

[0035] 2 End-cloud interaction system

[0036] 20 Terminal

[0037] 21 Cloud server

[0038] 100 Joint attention processing system

[0039] 110 Data obtaining module

[0040] 120 Sorting processing module

[0041] 130 Joint processing module

[0042] 140 Result obtaining module

[0043] 200 Electronic device

[0044] 210 Memory

[0045] 220 Processor

[0046] 230 Display

[0047] S11-S14 steps DETAILED DESCRIPTION

[0048] Those skilled in the art will readily understand that the application can have other implementations that are equally valid and thus fall within the scope of the application. Such implementations can be based on different embodiments, different examples, or different details. Therefore, the embodiments described in the specification are illustrative only and should not be considered restrictive.

[0049] It should be noted that the terms "may" and "exemplary" are used herein in the sense of "by way of example." Any embodiment or design scheme described as "may" or "exemplary" should not be construed as preferred or advantageous over other embodiments or design schemes. Rather, the terms "may" and "exemplary" are used to present concepts in a particular manner.

[0050] In the embodiments of the present application, "at least one" means one or more, and "multiple" means two or more. The association relationship between the associated objects is described by "and / or", which means that there can be three relationships, for example, A and / or B, which can represent the following three cases: A exists alone, A and B exist together, and B exists alone, where A and B can be singular or plural. The character " / " generally represents an "or" relationship between the associated objects before and after it. "At least one of the following" or similar expressions means any combination of these items, including any combination of single item or multiple items. For example, at least one of a, b, or c can represent a, b, c, a-b, a-c, b-c, or a-b-c, where a, b, and c can be single or multiple.

[0051] It should be noted that the diagrams provided in the following embodiments only schematically illustrate the basic concepts of the present application, and only the components related to the present application are shown in the diagrams, not the number, shape, and size of the components when actually implemented. The actual implementation of each component can be a random change in shape, number, and proportion, and the layout pattern of the components can be more complex.

[0052] To address the quadratic complexity of self-attention in Vision Transformers, SwinTransformer proposes window self-attention. This involves dividing the image into non-overlapping windows, independently computing self-attention within each window, and incorporating relative positional encoding. A shifted window is introduced to solve the information isolation problem between non-overlapping windows. By alternating between W-MSA (Multi-Head Window Self-Attention) and SW-MSA (Shifted Window Self-Attention) in two layers, global attention is achieved while maintaining linear complexity. However, implementing shifted window self-attention involves complex masking and shifting mechanisms, making it difficult to implement.

[0053] To address at least the aforementioned problems, embodiments of this application provide a joint attention processing method. The joint attention processing method includes: acquiring a target to be processed; sorting the target to be processed in at least one target dimension to obtain a reordered target to be processed; performing joint attention processing on the reordered target to obtain an initial result; and restoring the order of the initial result in the target dimension to be consistent with the target to be processed, thereby obtaining a processing result for the target to be processed.

[0054] In this embodiment, the targets to be processed are sorted along the target dimension, and joint attention processing is applied to the reordered targets to obtain an initial result. The initial result is then restored to the original order of the targets along the target dimension to obtain the processing result. The joint attention processing method of this application can expand the dimensions of joint attention processing, improve processing efficiency, and enhance the flexibility and transferability of joint attention processing.

[0055] Figure 1A This diagram illustrates an application scenario of the joint attention processing method described in this application. The joint attention processing device 1 can be used to implement the joint attention processing method provided in the embodiments of this application, but the application scenarios of the joint attention processing method provided in the embodiments of this application are not limited to this. Figure 1A The joint attention processing device 1 shown is as follows. Figure 1A As shown, the joint attention processing device 1 includes a storage device 11, a local processor 12, and a display terminal 13. The joint attention processing method provided in this application embodiment can be applied to the local processor 12.

[0056] in, Figure 1A The local processor 12 can be a single local processor, a cluster of multiple local processors, or a cloud computing center, etc., and is not specifically limited here. Although Figure 1A Only one storage device 11, one local processor 12, and one display terminal 13 are shown, but it should be understood that... Figure 1AThe examples in the above description are only for understanding the present solution, and the number of local processors 12 and display terminals 13 should be determined flexibly according to actual conditions.

[0057] In some other implementations, the joint attention processing apparatus 1 can not include the display terminal 13, but only include the local processor 12 with display function and the storage device 11. The joint attention processing method provided by the embodiments of the present application can be applied to the local processor 12 with display function. The local processor 12 with display function can include a tablet computer, a notebook computer, a palm computer, a mobile phone, a personal computer, a voice interactive device, a monitoring device, a face recognition device, etc., which are not limited herein.

[0058] In some other implementations, the joint attention processing method described in the present application can be applied to an end-cloud interaction scenario. Figure 1B FIG. 1 shows a structural schematic diagram of the end-cloud interaction scenario in some implementations. As shown in FIG. 1, the end-cloud interaction system 2 includes a terminal 20 and a cloud server 21, and the terminal 20 and the cloud server 21 can communicate with each other, and the communication mode is not limited to a wired or wireless mode. Figure 1B

[0059] The terminal 20 can be mobile or fixed, for example, the terminal 20 can be a wireless terminal or a wired terminal, and the wireless terminal can be a device with wireless transceiving function, which can be deployed indoors, outdoors or in an industrial workshop. The terminal 20 can be a mobile phone, a tablet computer, a notebook computer, etc., which are not limited herein. The cloud server 21 can include one or more servers, or include one or more processing nodes, or include one or more virtual machines running on a server, and the cloud server 21 can also be referred to as a server cluster, a management platform, a data processing center, etc., which are not limited herein.

[0060] The technical solutions in the embodiments of the present application will be described in detail below with reference to the drawings in the embodiments of the present application.

[0061] The joint attention processing method provided in the embodiments of the present application can be implemented by, for example, the local processor 12 shown in FIG. 2 or the cloud server 21 shown in FIG. 3. Figure 1A Figure 1B The joint attention processing method provided in the embodiments of the present application can be implemented by, for example, the local processor 12 shown in FIG. 2 or the cloud server 21 shown in FIG. 3. Figure 2A FIG. 4 shows a flow schematic diagram of the joint attention processing method described in the embodiments of the present application. As shown in FIG. 4, the joint attention processing method includes steps S11-S14. Figure 2A

[0062] Step S11, obtaining a target to be processed.

[0063] ​​​Step S12, sorting the to-be-processed target in at least one target dimension to obtain a re-ordered to-be-processed target.

[0064] Step S13, performing joint attention processing on the re-ordered to-be-processed target to obtain an initial result.

[0065] Step S14, restoring the initial result to the order of the target dimension consistent with the to-be-processed target to obtain a processing result of the to-be-processed target.

[0066] In some possible implementation manners, Figure 2B A schematic diagram of the joint attention processing method is shown in the embodiments of the present application. As shown in the figure, Figure 2B the to-be-processed target is obtained, and the to-be-processed target is sorted in at least one target dimension to obtain a re-ordered to-be-processed target. The target dimension is any or several dimensions of the to-be-processed target, and the target dimension is determined by the dimension of the to-be-processed target and the dimension of the joint attention processing. The re-ordered to-be-processed target is processed by joint attention to obtain an initial result. The joint attention processing is a high-efficiency joint attention processing mechanism U-MSA (Unified Multi-head Self Attention), which is used to realize the joint processing of window self-attention or window causal self-attention and depth separation convolution or causal depth separation convolution. The joint attention processing includes, for example, at least twice window causal self-attention calculation, or at least once depth separation convolution calculation and at least once self-attention calculation, but the present application is not limited thereto. The order of the initial result in the target dimension is restored to be consistent with the to-be-processed target to obtain a processing result of the to-be-processed target.

[0067] In the embodiments of the present application, the to-be-processed target is sorted in the target dimension, the re-ordered to-be-processed target is processed by joint attention to obtain an initial result, and the initial result is restored to the original order of the to-be-processed target in the target dimension to obtain a processing result of the to-be-processed target. The joint attention processing method of the present application can expand the dimension of the joint attention processing, improve the processing efficiency, and enhance the flexibility and migration of the joint attention processing.

[0068] In an embodiment of the present application, obtaining at least one target dimension includes: obtaining the target dimension according to the dimension of the to-be-processed target and the dimension of the joint attention processing on the to-be-processed target.

[0069] In some possible implementations, the dimensions of the target to be processed include one-dimensional, two-dimensional, and three-dimensional dimensions, and the dimensions of the joint attention processing also include one-dimensional, two-dimensional, and three-dimensional dimensions. The dimension requiring reordering is determined based on the dimensions of the joint attention processing and the dimensions of the target to be processed, and is used as the target dimension. For example, video frames of a three-dimensional video target to be processed are sorted to obtain a reordered video, where the target dimension is the dimension of each video frame. As another example, the text order of a one-dimensional text target to be processed is sorted to obtain a reordered text, where the target dimension is the dimension of the text context order.

[0070] Figure 3 This diagram illustrates the joint attention processing of a one-dimensional target as described in an embodiment of this application. Figure 3 As shown, when the target to be processed is a one-dimensional target, obtaining the processing result of the one-dimensional target to be processed includes: performing one-dimensional causal convolution processing on the one-dimensional target to be processed to obtain a causal convolution result; sorting the target dimension of the one-dimensional target to be processed to obtain a reordered one-dimensional target to be processed; performing one-dimensional window causal self-attention processing on the reordered one-dimensional target to be processed to obtain a window causal self-attention result; restoring the order of the window causal self-attention result in the target dimension to be consistent with the one-dimensional target to be processed, so as to obtain the processing result of the one-dimensional target to be processed by combining the causal convolution result.

[0071] In some possible implementations, the one-dimensional target to be processed includes audio, language, or other types. Taking language as an example, a one-dimensional target to be processed (B, N, C) is obtained, where N is the dimension of the one-dimensional target. The one-dimensional target is padded with zeros on the left and then subjected to one-dimensional causal convolution to obtain the causal convolution result. The target dimension N of the one-dimensional target is sorted to obtain a reordered one-dimensional target. One-dimensional windowed causal self-attention is applied to the reordered one-dimensional target to obtain the windowed causal self-attention result. The order of the windowed causal self-attention result in the target dimension is restored to be consistent with the one-dimensional target. The windowed causal self-attention result with the causal convolution result is summed, or either output is used as input to another process to obtain the processing result of the one-dimensional target. In this embodiment, one-dimensional causal convolution processing is used to establish local relationships between adjacent tokens, and one-dimensional window causal self-attention processing is used to establish relationships between distant tokens. The results of the two are organically combined to construct the relationship between any two tokens in the one-dimensional target to be processed, thus solving the problems of computational complexity of global attention and limited contextual semantic connectivity of the large oracle model.

[0072] Figure 4A joint attention processing schematic diagram of a one-dimensional target to be processed is shown in the embodiments of the present application. As shown in Figure 4 When the target to be processed is a one-dimensional target, obtaining a processing result of the one-dimensional target to be processed includes: performing encoding processing on the target dimension to obtain a position encoding of the one-dimensional target to be processed; performing one-dimensional window causal self-attention processing on the one-dimensional target to be processed to obtain a first window causal self-attention result; performing sorting processing on the target dimension of the one-dimensional target to be processed to obtain a reordered one-dimensional target to be processed; performing one-dimensional window causal self-attention processing on the reordered one-dimensional target to be processed to obtain a second window causal self-attention result; and restoring the order of the second window causal self-attention result in the target dimension to be consistent with the one-dimensional target to be processed to obtain a processing result of the one-dimensional target to be processed in combination with the first window causal self-attention result.

[0073] In some possible implementations, the one-dimensional target to be processed includes audio and language. Taking language as an example, a one-dimensional target to be processed (B, N, C) is obtained, and N is the dimension of the one-dimensional target to be processed. Encoding processing is performed on the target dimension N to obtain a position encoding of the one-dimensional target to be processed. One-dimensional window causal self-attention processing is performed on the one-dimensional target to be processed to obtain a first window causal self-attention result. Sorting processing is performed on the target dimension of the one-dimensional target to be processed to obtain a reordered one-dimensional target to be processed. One-dimensional window causal self-attention processing is performed on the reordered one-dimensional target to be processed to obtain a second window causal self-attention result. The order of the second window causal self-attention result in the target dimension N is restored to be consistent with the one-dimensional target to be processed. The reordered second window causal self-attention result is added to the first window causal self-attention result, or the result of any output is taken as the input of another processing to obtain a processing result of the one-dimensional target to be processed.

[0074] In the embodiments of the present application, one-dimensional window causal self-attention calculation is performed on the one-dimensional target to be processed in the original order to obtain the local relationship of adjacent tokens, one-dimensional window causal self-attention calculation is performed on the reordered one-dimensional target to be processed to obtain the relationship of distant tokens, and the results of the two are organically combined to construct the relationship between any two tokens in the one-dimensional target to be processed, thereby solving the problems of complex global attention calculation and limited context semantic connection on a large prediction model.

[0075] Figure 5 A joint attention processing schematic diagram of a two-dimensional target to be processed is shown in the embodiments of the present application. As shown in Figure 5As shown, when the to-be-processed target is a two-dimensional to-be-processed target, the processing result of the two-dimensional to-be-processed target is obtained by performing two-dimensional depth separation convolution processing on the two-dimensional to-be-processed target to obtain a depth separation convolution result, performing sorting processing on a target dimension of the two-dimensional to-be-processed target to obtain a reordered two-dimensional to-be-processed target, performing window self-attention processing on the reordered two-dimensional to-be-processed target to obtain a window self-attention result, and restoring the order of the window self-attention result in the target dimension to be consistent with the two-dimensional to-be-processed target to obtain the processing result of the two-dimensional to-be-processed target in combination with the depth separation convolution result.

[0076] In some possible implementation manners, taking an image as an example of the two-dimensional to-be-processed target (B, H, W, C), the two dimensions are H and W. The two-dimensional to-be-processed target is processed by two-dimensional depth separation convolution to obtain a depth separation convolution result. The dimensions of the two-dimensional to-be-processed target are H and W, and the target dimension of the two-dimensional to-be-processed target is sorted to obtain a reordered two-dimensional to-be-processed target. The reordered two-dimensional to-be-processed target is processed by window self-attention to obtain a window self-attention result. The order of the window self-attention result in the target dimension is restored to be consistent with the two-dimensional to-be-processed target. The window self-attention result after the order is restored is added to the depth separation convolution result, or the result of any output is taken as the input of another processing, to obtain the processing result of the two-dimensional to-be-processed target.

[0077] In the embodiments of the present application, the target dimension of the two-dimensional to-be-processed target is sorted, and the depth separation convolution processing and the window self-attention processing are used to establish local relationships and long-distance relationships, respectively. The results of the two are organically combined to establish relationships between arbitrary tokens, thereby reducing the computational complexity and improving the processing efficiency.

[0078] Figure 6 A joint attention processing schematic diagram of a three-dimensional to-be-processed target is shown in the embodiments of the present application. As shown in FIG. 6, the three-dimensional to-be-processed target is processed by performing sorting processing on a target dimension of the three-dimensional to-be-processed target to obtain a reordered three-dimensional to-be-processed target, performing window self-attention processing on the reordered three-dimensional to-be-processed target to obtain a window self-attention result, and restoring the order of the window self-attention result in the target dimension to be consistent with the three-dimensional to-be-processed target to obtain the processing result of the three-dimensional to-be-processed target in combination with the depth separation convolution result. Figure 6As shown, when the to-be-processed target is a three-dimensional to-be-processed target, the processing result of the three-dimensional to-be-processed target is obtained by: performing encoding processing on another dimension outside the target dimension of the three-dimensional to-be-processed target to obtain a position encoding of the three-dimensional to-be-processed target; performing two-dimensional depth separable convolution processing on the three-dimensional to-be-processed target to obtain a two-dimensional depth separable convolution result; performing sorting processing on the target dimension of the three-dimensional to-be-processed target to obtain a reordered three-dimensional to-be-processed target; performing window self-attention processing on the reordered three-dimensional to-be-processed target to obtain a window self-attention result; and restoring the order of the window self-attention result in the target dimension to be consistent with the three-dimensional to-be-processed target to obtain the processing result of the three-dimensional to-be-processed target in combination with the two-dimensional depth separable convolution result. The target dimension can also be a dimension that is not subjected to position encoding.

[0079] In some possible implementation manners, the three-dimensional to-be-processed target includes a video, a 3D point cloud or other types, and the application is not limited thereto. Taking the video as an example, a two-dimensional to-be-processed target (B, D, H, W, C) is obtained, and three dimensions D, H and W are referred to as three dimensions. Encoding processing is performed on another dimension D outside the target dimension of the three-dimensional to-be-processed target to obtain a position encoding of the three-dimensional to-be-processed target. Two-dimensional depth separable convolution processing is performed on the three-dimensional to-be-processed target to obtain a two-dimensional depth separable convolution result. Sorting processing is performed on the target dimension H and the dimension W of the three-dimensional to-be-processed target to obtain a reordered three-dimensional to-be-processed target. Window self-attention processing is performed on the tokens of all frames of the reordered three-dimensional to-be-processed target to obtain a window self-attention result. The order of the window self-attention result in the target dimension is restored to be consistent with the three-dimensional to-be-processed target. The window self-attention result after the order is restored is added to the two-dimensional depth separable convolution result, or the result of any output is taken as the input of another processing to obtain the processing result of the three-dimensional to-be-processed target.

[0080] In the embodiments of the application, the two-dimensional depth separable convolution processing is used to establish the local relationship between single-frame tokens, the window self-attention processing is used to establish the relationship between distant tokens in a single frame and the relationship between some tokens in different frames, the results of the two are organically combined to establish the relationship between any tokens, the calculation complexity is reduced, and the processing efficiency is improved.

[0081] Figure 7 A joint attention processing schematic diagram of the three-dimensional to-be-processed target described in the embodiments of the application is shown. As shown in FIG. 1, the three-dimensional to-be-processed target is a video, and the video is processed by the joint attention processing method. Figure 7As shown, when the to-be-processed target is a three-dimensional to-be-processed target, obtaining the processing result of the three-dimensional to-be-processed target comprises: performing three-dimensional depth separation convolution processing on the three-dimensional to-be-processed target to obtain a three-dimensional depth separation convolution result; performing sorting processing on a target dimension of the three-dimensional to-be-processed target to obtain a reordered three-dimensional to-be-processed target; performing three-dimensional window self-attention processing on the reordered three-dimensional to-be-processed target to obtain a three-dimensional window self-attention result; and restoring the order of the three-dimensional window self-attention result in the target dimension to be consistent with the three-dimensional to-be-processed target to obtain the processing result of the three-dimensional to-be-processed target in combination with the three-dimensional depth separation convolution result.

[0082] In some possible implementation manners, the three-dimensional to-be-processed target comprises a video, a 3D point cloud or other types, and the application is not limited thereto. Taking the video as an example, a two-dimensional to-be-processed target (B, D, H, W, C) is obtained, and three dimensions of D, H and W are three-dimensional. The three-dimensional to-be-processed target is subjected to three-dimensional depth separation convolution processing to obtain a three-dimensional depth separation convolution result. The target dimensions D, H and W of the three-dimensional to-be-processed target are subjected to sorting processing to obtain a reordered three-dimensional to-be-processed target. Then, the reordered three-dimensional to-be-processed target is divided into non-overlapping dhw-shaped blocks (such as Figure 7 a black cuboid shown), three-dimensional window self-attention processing is performed in the blocks to obtain a three-dimensional window self-attention result. The order of the three-dimensional window self-attention result in the target dimension is restored to be consistent with the three-dimensional to-be-processed target. The window self-attention result after the order is restored is added to the three-dimensional depth separation convolution result, or the result of any output is taken as the input of another processing to obtain the processing result of the three-dimensional to-be-processed target.

[0083] In the embodiment of the application, the three-dimensional depth separation convolution processing is used to establish the local relationship between tokens of consecutive frames, and the window self-attention processing is used to establish the relationship between distant tokens in a single frame and the relationship between some tokens between frames in part of frames. The results of the two are organically combined to establish the relationship between any tokens, which reduces the calculation complexity and improves the processing efficiency.

[0084] Figure 8 A joint attention processing schematic diagram of the three-dimensional to-be-processed target described in the embodiment of the application is shown. As Figure 8As shown, when the to-be-processed target is a three-dimensional to-be-processed target, obtaining the processing result of the three-dimensional to-be-processed target comprises: performing two-dimensional depth separation convolution processing on the three-dimensional to-be-processed target to obtain a two-dimensional depth separation convolution result; performing sorting processing on a target dimension of the three-dimensional to-be-processed target to obtain a re-sequenced three-dimensional to-be-processed target; performing three-dimensional window self-attention processing on the re-sequenced three-dimensional to-be-processed target and restoring the order of the target dimension to be consistent with the three-dimensional to-be-processed target to obtain a three-dimensional window self-attention result; performing one-dimensional depth separation convolution processing on the three-dimensional to-be-processed target in the depth dimension to obtain a one-dimensional depth separation convolution result; and obtaining the processing result of the three-dimensional to-be-processed target according to the two-dimensional depth separation convolution result, the three-dimensional window self-attention result and the one-dimensional depth separation convolution result.

[0085] In some possible implementation manners, the three-dimensional to-be-processed target comprises a video, a 3D point cloud or other types, and the application is not limited thereto. Taking the video as an example, a two-dimensional to-be-processed target (B, D, H, W, C) is obtained, and three dimensions of D, H and W are three-dimensional. Two-dimensional depth separation convolution processing is performed on the three-dimensional to-be-processed target to obtain a two-dimensional depth separation convolution result. Sorting processing is performed on the three dimensions of D, H and W of the target dimension of the three-dimensional to-be-processed target to obtain a re-sequenced three-dimensional to-be-processed target. Three-dimensional window self-attention processing is performed on the re-sequenced three-dimensional to-be-processed target and the order of the target dimension is restored to be consistent with the three-dimensional to-be-processed target to obtain a three-dimensional window self-attention result. One-dimensional depth separation convolution processing is performed on the three-dimensional to-be-processed target in the depth dimension D to obtain a one-dimensional depth separation convolution result. The processing result of the three-dimensional to-be-processed target is obtained by performing summation calculation according to the two-dimensional depth separation convolution result, the three-dimensional window self-attention result and the one-dimensional depth separation convolution result or taking the calculation result of one of them as the input of another calculation.

[0086] In the embodiment of the application, the two-dimensional depth separation convolution processing is used for the local relationship between tokens of a single frame, the window self-attention processing is used for establishing the relationship between distant tokens of a single frame and the relationship between some tokens of frames and frames of part of frames, and the one-dimensional depth separation convolution processing is used for establishing the relationship between D tokens at the same spatial position (H, W) of each frame. The results of the three are organically combined to establish the relationship between any tokens, reduce the calculation complexity and improve the processing efficiency.

[0087] In an embodiment of the application, summation calculation is performed on the calculation results of at least two times of joint attention processing, or the input of any one calculation is taken as the output of another calculation to obtain the processing result of the to-be-processed target.

[0088] In some possible implementation manners,Figure 9A A schematic diagram for obtaining a processing result is shown in the embodiments of the present application. As shown in the figure, Figure 9A z l is an input target to be processed, serial number 1 represents sorting processing on the target dimension of the target to be processed, and serial number 2 represents recovering the order of the target dimension of the target to be processed, l+1 z is a processing result of the target to be processed obtained by adding and calculating the window self-attention result after recovering the order and the depth separated convolution result.

[0089] In some possible implementation manners, Figure 9B A schematic diagram for obtaining a processing result is shown in the embodiments of the present application. As shown in the figure, Figure 9B z l is an input target to be processed, a depth separated convolution result of the target to be processed is obtained, the depth separated convolution result is taken as an input of window self-attention processing, serial number 1 represents sorting processing on the target dimension of the input data, the order of the target dimension is recovered at serial number 2 after the window self-attention processing, and then a processing result z l+1 of the target to be processed is obtained.

[0090] In some possible implementation manners, Figure 9C A schematic diagram for obtaining a processing result is shown in the embodiments of the present application. As shown in the figure, Figure 9C z l is an input target to be processed, serial number 1 represents sorting processing on the target dimension of the input data, the order of the target dimension is recovered at serial number 2, a window self-attention result of the target to be processed is obtained, the window self-attention result is taken as an input of depth separated convolution processing, and then a processing result z l+1 of the target to be processed is obtained.

[0091] It should be noted that the above is only a general description of obtaining a processing result of a target to be processed in the embodiments of the present application, and is applicable to different joint attention processing manners of targets to be processed with different dimensions. The calculation manner in a specific application process is determined according to the joint attention processing process, and is not limited thereto.

[0092] Figure 10 A structural schematic diagram of a joint attention processing system is shown in the embodiments of the present application. As shown in the figure, Figure 10 the joint attention processing system 100 includes a data obtaining module 110, a sorting processing module 120, a joint processing module 130, and a result obtaining module 140.

[0093] The data obtaining module 110 is configured to obtain a target to be processed.

[0094] The sorting processing module 120 performs sorting processing on the to-be-processed target in at least one target dimension, to obtain a re-ordered to-be-processed target.

[0095] The joint processing module 130 is configured to perform joint attention processing on the re-ordered to-be-processed target, to obtain an initial result.

[0096] The result obtaining module 140 is configured to restore the order of the initial result in the target dimension to be consistent with the to-be-processed target, to obtain a processing result of the to-be-processed target.

[0097] It should be noted that the above modules 110-140 included in the joint attention processing system 100 correspond to steps S11-S14 in the joint attention processing method shown in FIG. 1, and thus will not be described here. Figure 2A

[0098] In several embodiments provided in the present application, it should be understood that the disclosed system, device or method can be implemented by other means. For example, the above-described device embodiments are only illustrative, for example, the division of the modules / units is only a logical function division, and actual implementation can have another division manner, for example, a plurality of modules or units can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between the displayed or discussed modules / units can be indirect coupling or communication connection through some interfaces, devices or modules / units, which can be electrical, mechanical or other forms.

[0099] The modules / units described as separate components can or can not be physically separated, and the components shown as modules / units can or can not be physical modules, i.e., can be located in one place, or can be distributed on multiple network units. According to actual needs, part or all of the modules / units can be selected to achieve the purpose of the embodiments of the present application. For example, the functional modules / units in each embodiment of the present application can be integrated in one processing module, or each module / unit can be physically present separately, or two or more modules / units can be integrated in one module / unit.

[0100] ​Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0101] This application also provides an electronic device. Figure 11 The diagram shown is a structural schematic of the electronic device described in an embodiment of this application. Figure 11 As shown, in this embodiment, the electronic device 200 includes a memory 210 and a processor 220.

[0102] The memory 210 is used to store computer programs; preferably, the memory 210 includes various media that can store program code, such as ROM, RAM, magnetic disk, USB flash drive, memory card or optical disk.

[0103] Specifically, memory 210 may include computer system readable media in the form of volatile memory, such as random access memory (RAM) and / or cache memory. Electronic device 200 may further include other removable / non-removable, volatile / non-volatile computer system storage media. Memory 210 may include at least one program product having a set (e.g., at least one) of program modules configured to perform the functions of the embodiments of this application. It is understood that memory 210 may be volatile memory or non-volatile memory, or both. Non-volatile memory may be read-only memory (ROM) or programmable read-only memory (PROM), which serves as an external cache. By way of example, but not limitation, many forms of RAM are available, such as static random access memory (SRAM) and synchronous static random access memory (SSRAM). The memories described in the embodiments of this invention are intended to include, but are not limited to, these and any other suitable categories of memories.

[0104] The processor 220 is connected with the memory 210, and is configured to execute a computer program stored in the memory 210, so that the electronic device 200 performs the joint attention processing method according to any one of the embodiments of the present application.

[0105] Optionally, the processor 220 can be a general processor, including a central processing unit (CPU), a network processor (NP), etc. ; and can also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic device, a discrete gate or transistor logic device, a discrete hardware component.

[0106] Optionally, the electronic device 200 in the embodiment can further include a display 230. The display 230 is connected with the memory 210 and the processor 220 in communication, and is configured to display a related graphical user interface (GUI) interactive interface of the joint attention processing method according to the embodiments of the present application.

[0107] The terms "component", "module", "system", and the like used in the present specification are used to represent computer-related entities, hardware, firmware, combinations of hardware and software, software, or software in execution. For example, a component can be, but is not limited to, a process running on a processor, a processor, an object, an executable, a thread of execution, a program, and / or a computer. By way of illustration, both an application running on a computing device and the computing device can be a component. One or more components can reside within a process and / or thread of execution, and a component can be localized on one computer and / or distributed between two or more computers. In addition, these components can execute from various computer readable media having various data structures stored thereon. The components can communicate via local and / or remote processes such as in accordance with a signal having one or more data packets (e.g., data from programs, data included in a management information base, etc.), such as data generated / produced by one component and transmitted to another component in accordance with the related embodiments.

[0108] The descriptions of the corresponding flow or structure of each of the above figures are each focused on, and the parts not described in detail in a certain flow or structure can be referred to the related description of other flow or structure.

[0109] The above embodiments are only illustrative of the principles of the present application and its effects, and are not intended to limit the present application. Any modification or change made by any person skilled in the art without departing from the spirit and scope of the present application shall be covered by the claims of the present application.

Claims

1. A joint attention processing method, characterized by, The method comprises: obtaining a target to be processed; sorting the target to be processed in at least one target dimension to obtain a reordered target to be processed; performing joint attention processing on the reordered target to be processed to obtain an initial result; and restoring the order of the initial result in the target dimension to be consistent with the target to be processed to obtain a processing result of the target to be processed.

2. The joint attention processing method of claim 1, wherein, Obtaining at least one target dimension comprises: obtaining the target dimension according to the dimension of the target to be processed and the dimension of the joint attention processing on the target to be processed.

3. The joint attention processing method of claim 1, wherein, When the target to be processed is a one-dimensional target to be processed, obtaining a processing result of the one-dimensional target to be processed comprises: performing one-dimensional causal convolution processing on the one-dimensional target to be processed to obtain a causal convolution result; sorting the target dimension of the one-dimensional target to be processed to obtain a reordered one-dimensional target to be processed; performing one-dimensional window causal self-attention processing on the reordered one-dimensional target to be processed to obtain a window causal self-attention result; and restoring the order of the window causal self-attention result in the target dimension to be consistent with the one-dimensional target to be processed to obtain a processing result of the one-dimensional target to be processed in combination with the causal convolution result.

4. The joint attention processing method of claim 1, wherein, When the target to be processed is a one-dimensional target to be processed, obtaining a processing result of the one-dimensional target to be processed comprises: performing encoding processing on the target dimension to obtain position encoding of the one-dimensional target to be processed; performing one-dimensional window causal self-attention processing on the one-dimensional target to be processed to obtain a first window causal self-attention result; sorting the target dimension of the one-dimensional target to be processed to obtain a reordered one-dimensional target to be processed; performing one-dimensional window causal self-attention processing on the reordered one-dimensional target to be processed to obtain a second window causal self-attention result; and restoring the order of the second window causal self-attention result in the target dimension to be consistent with the one-dimensional target to be processed to obtain a processing result of the one-dimensional target to be processed in combination with the first window causal self-attention result.

5. The joint attention processing method of claim 1, wherein, When the target to be processed is a two-dimensional target to be processed, obtaining a processing result of the two-dimensional target to be processed comprises: performing two-dimensional depthwise separable convolution processing on the two-dimensional target to be processed to obtain a depthwise separable convolution result; sorting the target dimension of the two-dimensional target to be processed to obtain a reordered two-dimensional target to be processed; performing window self-attention processing on the reordered two-dimensional target to be processed to obtain a window self-attention result; and restoring the order of the window self-attention result in the target dimension to be consistent with the two-dimensional target to be processed to obtain a processing result of the two-dimensional target to be processed in combination with the depthwise separable convolution result.

6. The joint attention processing method of claim 1, wherein, When the target to be processed is a three-dimensional target to be processed, obtaining a processing result of the three-dimensional target to be processed comprises: performing encoding processing on another dimension outside the target dimension of the three-dimensional target to be processed to obtain position encoding of the three-dimensional target to be processed; performing two-dimensional depthwise separable convolution processing on the three-dimensional target to be processed to obtain a two-dimensional depthwise separable convolution result; The target dimension of the three-dimensional target to be processed is sorted to obtain a re-ordered three-dimensional target to be processed; The re-ordered three-dimensional target to be processed is subjected to window self-attention processing to obtain a window self-attention result; The order of the window self-attention result in the target dimension is restored to be consistent with the three-dimensional target to be processed to obtain a processing result of the three-dimensional target to be processed in combination with the two-dimensional depth separation convolution result.

7. The joint attention processing method of claim 1, wherein, When the target to be processed is a three-dimensional target to be processed, obtaining a processing result of the three-dimensional target to be processed includes: The three-dimensional target to be processed is subjected to three-dimensional depth separation convolution processing to obtain a three-dimensional depth separation convolution result; The target dimension of the three-dimensional target to be processed is sorted to obtain a re-ordered three-dimensional target to be processed; The re-ordered three-dimensional target to be processed is subjected to three-dimensional window self-attention processing to obtain a three-dimensional window self-attention result; The order of the three-dimensional window self-attention result in the target dimension is restored to be consistent with the three-dimensional target to be processed to obtain a processing result of the three-dimensional target to be processed in combination with the three-dimensional depth separation convolution result.

8. The joint attention processing method of claim 1, wherein, When the target to be processed is a three-dimensional target to be processed, obtaining a processing result of the three-dimensional target to be processed includes: The three-dimensional target to be processed is subjected to two-dimensional depth separation convolution processing to obtain a two-dimensional depth separation convolution result; The target dimension of the three-dimensional target to be processed is sorted to obtain a re-ordered three-dimensional target to be processed; The re-ordered three-dimensional target to be processed is subjected to three-dimensional window self-attention processing and the order of the three-dimensional window self-attention result in the target dimension is restored to be consistent with the three-dimensional target to be processed to obtain a three-dimensional window self-attention result; The three-dimensional target to be processed is subjected to one-dimensional depth separation convolution processing in the depth dimension to obtain a one-dimensional depth separation convolution result; A processing result of the three-dimensional target to be processed is obtained according to the two-dimensional depth separation convolution result, the three-dimensional window self-attention result and the one-dimensional depth separation convolution result.

9. A joint attention processing system, characterized by, It includes: A data acquisition module is configured to acquire a target to be processed; A sorting processing module is configured to sort the target to be processed in at least one target dimension to obtain a re-ordered target to be processed; A joint processing module is configured to perform joint attention processing on the re-ordered target to be processed to obtain an initial result; A result acquisition module is configured to restore the order of the initial result in the target dimension to be consistent with the target to be processed to obtain a processing result of the target to be processed.

10. An electronic device, comprising: The electronic device includes: A memory is configured to store a computer program; A processor is configured to execute the computer program stored in the memory to enable the electronic device to perform the joint attention processing method according to any one of claims 1 to 8.

Images