Multi-view target detection or model training method and device, electronic equipment and medium

Abstract

The invention discloses a multi-view target detection or model training method and device, electronic equipment and a medium. The method comprises the following steps: carrying out feature extraction processing on collected images of at least two view angles of the same scene to obtain interested visual features and interested position features corresponding to the view angles; performing linear processing on the interested visual features corresponding to the visual angle according to the interested position features corresponding to the visual angle to obtain an attention key matrix corresponding to the visual angle under at least one attention head; calculating an attention result between every two view angles; fusing the attention results between the two visual angles under the attention heads to obtain multi-head attention results of the two visual angles; determining a target feature of each view angle; and determining a target detection result of the collected image of each view angle according to the target feature of each view angle. According to the embodiment of the invention, multi-view image features can be fully utilized, and the target detection accuracy is greatly improved.

Description

Technical Field

[0001] The present invention relates to the field of image processing, and in particular to a multi-view target detection or model training method, device, electronic equipment and medium. Background Art

[0002] Object detection is usually a detection method to detect whether there is a target object in an image.

[0003] Currently, target detection algorithms have high detection accuracy in single-view images, but for images with two or more views, the improvement in detection accuracy is limited. Summary of the invention

[0004] The present invention provides a multi-view target detection or model training method, device, electronic device and medium, which can fully utilize multi-view image features and greatly improve the accuracy of target detection.

[0005] According to one aspect of the present invention, a multi-view target detection method is provided, the method comprising:

[0006] Performing feature extraction processing on the collected images of at least two viewing angles of the same scene to obtain the visual features of interest and the position features of interest corresponding to each of the viewing angles;

[0007] For each of the viewing angles, determining an attention key matrix corresponding to the viewing angle under at least one attention head according to the visual features of interest and the position features of interest corresponding to the viewing angle;

[0008] For each of the attention heads, according to the attention key matrix corresponding to each of the perspectives, calculating the attention result between every two perspectives;

[0009] For every two perspectives, the attention results between the two perspectives under each of the attention heads are fused to obtain multi-head attention results of the two perspectives;

[0010] Determining target features of each of the perspectives according to the multi-head attention results of each of the two perspectives and the visual features of interest of each of the perspectives;

[0011] According to the target features of each of the viewing angles, the target detection results of the captured images of each of the viewing angles are determined.

[0012] According to one aspect of the present invention, a method for training a multi-view object detection model is provided, the method comprising:

[0013] Through the twin network feature extraction network in the multi-view target detection model, feature extraction processing is performed on sample images of at least two viewpoints of the same scene to obtain the visual features of interest and the position features of interest corresponding to each of the viewpoints;

[0014] By using a multi-head cross attention network in the multi-view target detection model, for each of the viewpoints, according to the interesting visual features and interesting position features corresponding to the viewpoint, determining an attention key matrix corresponding to the viewpoint under at least one attention head;

[0015] By using the multi-head cross attention network, for each of the attention heads, according to the attention key matrix corresponding to each of the perspectives, the attention result between each two perspectives is calculated;

[0016] Through the multi-head cross attention network, for each two perspectives, the attention results between the two perspectives under each of the attention heads are fused to obtain the multi-head attention results of the two perspectives;

[0017] Determining target features of each of the views according to the multi-head attention results of each of the two views and the visual features of interest of each of the views through a regression classification network in the multi-view target detection model;

[0018] Determine the predicted detection results of the sample images of each of the viewing angles according to the target features of each of the viewing angles through the regression classification network;

[0019] According to the difference between the standard detection results and the predicted detection results of the sample images of each perspective, the parameters of the twin network feature extraction network and the multi-head cross attention network are adjusted.

[0020] According to another aspect of the present invention, a multi-view target detection device is provided, the device comprising:

[0021] The twin network feature extraction network in the multi-view target detection model is used to perform feature extraction processing on the collected images of at least two viewpoints of the same scene to obtain the visual features of interest and the position features of interest corresponding to each of the viewpoints;

[0022] The multi-head cross attention network in the multi-view target detection model is used to determine, for each of the view angles, an attention key matrix corresponding to the view angle under at least one attention head according to the visual features of interest and the position features of interest corresponding to the view angle;

[0023] The multi-head cross attention network is used to calculate the attention result between each two perspectives according to the attention key matrix corresponding to each perspective for each attention head;

[0024] The multi-head cross attention network is used to fuse the attention results between the two perspectives under each of the attention heads for each two perspectives to obtain the multi-head attention results of the two perspectives;

[0025] The regression classification network in the multi-view target detection model is used to determine the target features of each of the two views according to the multi-head attention results of each of the two views and the visual features of interest of each of the views;

[0026] The regression classification network is used to determine the target detection results of the captured images of each of the viewing angles based on the target features of each of the viewing angles.

[0027] According to another aspect of the present invention, a training device for a multi-view object detection model is provided, the device comprising:

[0028] The twin network feature extraction network in the multi-view target detection model is used to perform feature extraction processing on sample images of at least two viewpoints of the same scene to obtain the visual features of interest and the position features of interest corresponding to each of the viewpoints;

[0029] The multi-head cross attention network in the multi-view target detection model is used to determine, for each of the view angles, an attention key matrix corresponding to the view angle under at least one attention head according to the visual features of interest and the position features of interest corresponding to the view angle;

[0030] The multi-head cross attention network is used to calculate the attention result between each two perspectives according to the attention key matrix corresponding to each perspective for each attention head;

[0031] The multi-head cross attention network is used to fuse the attention results between the two perspectives under each of the attention heads for each two perspectives to obtain the multi-head attention results of the two perspectives;

[0032] The regression classification network in the multi-view target detection model is used to determine the target features of each of the two views according to the multi-head attention results of each of the two views and the visual features of interest of each of the views;

[0033] The regression classification network is used to determine the predicted detection results of the sample images of each of the viewing angles according to the target features of each of the viewing angles;

[0034] The model training module is used to adjust the parameters of the twin network feature extraction network and the multi-head cross attention network according to the difference between the standard detection results and the predicted detection results of the sample images of each perspective.

[0035] According to another aspect of the present invention, an electronic device is provided, the electronic device comprising:

[0036] at least one processor; and

[0037] a memory communicatively connected to the at least one processor; wherein,

[0038] The memory stores a computer program that can be executed by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute the multi-view target detection method or the training method of the multi-view target detection model described in any embodiment of the present invention.

[0039] According to another aspect of the present invention, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the multi-view target detection method or the multi-view target detection model training method described in any embodiment of the present invention when executed.

[0040] The technical solution of the embodiment of the present invention obtains captured images of multiple perspectives of the same scene and performs feature extraction to obtain visual features of interest and position features of interest, and determines the attention key matrix corresponding to each perspective under each attention head for the visual features of interest and position features of interest corresponding to each perspective, calculates the attention results of each perspective for other perspectives for each attention head, and fuses the attention results of different attention heads to obtain multi-head attention results of each perspective for other perspectives, and finally updates the visual features of interest of the perspective based on the multi-head attention results of the perspective for other perspectives, determines the target features of the perspective, and determines the target detection results of the perspective based on the target features, so that the scene information of other perspectives can be added to the visual features of interest of each perspective, enriching the image features of each perspective, improving the representativeness of the visual features of interest, realizing image data fusion of multiple perspectives, and thus improving the accuracy of target detection.

[0041] It should be understood that the contents described in this section are not intended to identify the key or important features of the embodiments of the present invention, nor are they intended to limit the scope of the present invention. Other features of the present invention will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0043] Figure 1 is a flow chart of a multi-view target detection method provided according to an embodiment of the present invention;

[0044] Figure 2 is a flow chart of a multi-view target detection method provided according to an embodiment of the present invention;

[0045] Figure 3 is a flowchart of a training method for a multi-view target detection model provided according to an embodiment of the present invention;

[0046] Figure 4 is a scene diagram of a training method for a multi-view target detection model provided by an embodiment of the present invention;

[0047] Figure 5 is a schematic structural diagram of a multi-view target detection device provided according to an embodiment of the present invention;

[0048] Figure 6 is a structural schematic diagram of a training device for a multi-view target detection model provided according to an embodiment of the present invention;

[0049] Figure 7 It is a structural schematic diagram of an electronic device for implementing the multi-view target detection method or the training method of a multi-view target detection model according to an embodiment of the present invention. DETAILED DESCRIPTION

[0050] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0051] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0052] Figure 1A flowchart of a multi-view target detection method provided by an embodiment of the present invention. The embodiment of the present invention is applicable to the case of performing target detection based on multi-view captured images. The method can be executed by a multi-view target detection device, which can be implemented in the form of hardware and / or software. The multi-view target detection device can be configured in an electronic device that carries a multi-view target detection function, such as a client or a server. The client can include: a mobile phone, a tablet computer, a laptop computer, a desktop computer, a wearable device, etc.

[0053] See also Figure 1 The multi-view target detection method shown includes:

[0054] S101, performing feature extraction processing on images collected from at least two viewing angles of the same scene to obtain visual features of interest and position features of interest corresponding to each of the viewing angles.

[0055] The same scene may refer to the same space or the same product. Image acquisition devices with different positions and / or angles may be set, and images of at least two viewing angles may be acquired by at least one image acquisition device. Images of at least two viewing angles may be acquired by image acquisition devices at different positions, or images of at least two viewing angles may be acquired by image acquisition devices at the same position at different angles. One acquired image corresponds to one viewing angle. Visual features of interest are used to describe image features of interest at one viewing angle. Position features of interest are used to describe position features of interest at one viewing angle. Feature extraction processing is performed on the acquired image to determine at least one region of interest, and visual features and position features of each region of interest are determined. The visual features may refer to feature vectors obtained by extracting image features of the region of interest. Position features may refer to positioning information of the region of interest, for example, the position features include key point coordinates, width and height, etc., wherein the region of interest is a rectangle, and the key point coordinates may include the coordinates of the upper left vertex, the upper right vertex, the lower left vertex, or the lower right vertex of the region of interest.

[0056] In some embodiments, a twin network feature extraction module can be used to process the captured image to obtain the visual features of interest and the position features of interest of at least one region of interest. For example, the twin network feature extraction module can be a Faster R-CNN (Faster Region-based Convolutional Neural Networks, region-based fast convolutional neural network) network, and Faster R-CNN includes a parameter-shared Backbone, parameter-independent FPN (Feature Pyramid Networks, Feature Pyramid Network) and RPN (Region Proposal Network, Region Candidate Network). In one example, for a certain perspective, the captured image of the perspective is input into the Backbone for processing, and the feature map of the perspective is output. One perspective corresponds to one Backbone, and the parameters of the Backbones corresponding to different perspectives are shared. Parameter sharing not only reduces the memory usage, but also avoids redundant calculations, and further improves the computational efficiency of the network. The feature map of the perspective is input into the FPN for processing, and the feature vector set F of the visual of the region of interest is output as the visual features of interest of the perspective; and the feature map is input into the RPN for processing, and the position set P of the region of interest is output as the position features of interest of the perspective. Where F includes the visual feature vectors of m regions of interest, and P includes the positions of m regions of interest. , features of the location of interest , , is the coordinate of the upper left corner of the region of interest, is the width of the region of interest, For example, the interested visual feature F1 and the interested position feature P1 of the first viewing angle, and the interested visual feature F2 and the interested position feature P2 of the second viewing angle are obtained.

[0057] S102. For each of the viewing angles, determine an attention key matrix corresponding to the viewing angle under at least one attention head according to the interesting visual features and interesting position features corresponding to the viewing angle.

[0058] Among them, the attention head is used to describe the model's focus dimension on key information in the image. The attention key matrix is ​​used to calculate the attention weight to assist the model in paying attention to key information. Based on the Transform algorithm, the attention key matrix of at least one attention head can be obtained by processing the visual features of interest and the position features of interest. One attention head corresponds to one attention key matrix. The attention key matrix includes query, key, and value matrices.

[0059] In some embodiments, a linear transformation may be performed on the visual features of interest and the position features of interest to obtain a set of QKV matrices as the attention key matrix. When different linear transformations are performed, multiple sets of QKV matrices, i.e., multiple attention key matrices, may be obtained. One linear transformation corresponds to an attention key matrix of an attention head.

[0060] In some embodiments, the visual features of interest may be linearly changed to obtain a QKV matrix, and the QKV matrix may be adjusted according to the features of the positions of interest, and the adjusted QKV matrix may be used as an attention key matrix.

[0061] S103. For each of the attention heads, calculate the attention result between every two perspectives according to the attention key matrix corresponding to each perspective.

[0062] The attention result may refer to the degree of attention paid from one perspective to the features of another perspective. The attention result between each two perspectives may refer to the attention result of the first perspective to the second perspective, and the attention result of the second perspective to the first perspective. Multiple perspectives may be combined in pairs to obtain at least one group of two perspectives.

[0063] S104: For every two perspectives, the attention results between the two perspectives under each of the attention heads are fused to obtain a multi-head attention result of the two perspectives.

[0064] Among them, the multi-head attention result can refer to the degree of attention from one perspective to the features of another perspective in multiple dimensions. The attention results between the two perspectives under each attention head are fused, and the attention results of multiple attention heads are actually fused for a group of two perspectives. In some embodiments, a group of two perspectives includes perspective A and perspective B, and there are 3 attention heads in this group of perspectives. The attention results of perspective A on perspective B in the 3 attention heads can be fused to obtain the multi-head attention result of perspective A on perspective B; and the attention results of perspective B on perspective A in the 3 attention heads can be fused to obtain the multi-head attention result of perspective B on perspective A.

[0065] S105 , determining target features of each of the perspectives according to the multi-head attention results of the two perspectives and the visual features of interest of each of the perspectives.

[0066] The target feature may refer to the feature obtained by enhancing the visual feature of interest through the multi-head attention result. The target feature focuses on the key information of the view for other views, which can effectively add more representative key information of images from other views to the visual features of the view, thus enriching the visual features of the view. The fusion result of all multi-head attention results of a view and the visual feature of interest of the view is obtained as the target feature of the view.

[0067] S106: Determine target detection results of the captured images of each viewing angle according to the target features of each viewing angle.

[0068] Among them, the target features of a perspective can be nonlinearly transformed, expressed and predicted to obtain the target detection result of the captured image of the perspective. The target detection result includes at least one detection frame and the classification label of each detection frame. In some embodiments, the transported items can be photographed to obtain the captured image, and the captured image can be subjected to target detection. For example, the object of target detection is the items transported on the conveyor belt, and the position of the items is detected to sort the items, etc. For another example, the object of target detection is dangerous objects, and whether there are dangerous objects on the items is detected to conduct security checks on the items, etc.

[0069] The technical solution of the embodiment of the present invention obtains captured images of multiple perspectives of the same scene and performs feature extraction to obtain visual features of interest and position features of interest, and determines the attention key matrix corresponding to each perspective under each attention head for the visual features of interest and position features of interest corresponding to each perspective, calculates the attention results of each perspective for other perspectives for each attention head, and fuses the attention results of different attention heads to obtain multi-head attention results of each perspective for other perspectives, and finally updates the visual features of interest of the perspective based on the multi-head attention results of the perspective for other perspectives, determines the target features of the perspective, and determines the target detection results of the perspective based on the target features, so that the scene information of other perspectives can be added to the visual features of interest of each perspective, enriching the image features of each perspective, improving the representativeness of the visual features of interest, realizing image data fusion of multiple perspectives, and thus improving the accuracy of target detection.

[0070] Figure 2A flow chart of a multi-perspective target detection method provided by an embodiment of the present invention. Based on the above embodiments, the embodiment of the present invention concretizes "calculating the attention result between each two perspectives according to the attention key matrix corresponding to each of the perspectives" as follows: for each two perspectives, the query matrix in the attention key matrix of the first perspective and the key matrix in the attention key matrix of the second perspective are merged to obtain a weight matrix of the first perspective to the second perspective; based on the weight matrix of the first perspective to the second perspective, and the median matrix of the attention key matrix of the second perspective, the attention matrix of the first perspective to the second perspective is calculated as the attention result between the two perspectives. It should be noted that for the parts not described in detail in the embodiments of the present invention, please refer to the description of other embodiments.

[0071] See also Figure 2 The multi-view target detection method shown includes:

[0072] S201. Perform feature extraction processing on images collected from at least two viewing angles of the same scene to obtain visual features of interest and position features of interest corresponding to each of the viewing angles.

[0073] S202. For each of the viewing angles, determine an attention key matrix corresponding to the viewing angle under at least one attention head according to the interesting visual features and interesting position features corresponding to the viewing angle.

[0074] In an optional embodiment, the method linearly processes the visual features of interest corresponding to the view according to the position features of interest corresponding to the view to obtain a key attention matrix corresponding to the view under at least one attention head, including: linearly processing the visual features of interest corresponding to the view to obtain an initial attention matrix of the view under at least one attention head; calculating the center coordinates of at least one position feature of interest corresponding to the view; concatenating the center coordinates of each position feature of interest to obtain a center vector of interest corresponding to the view; encoding the center vector of interest corresponding to the view to obtain a position code corresponding to the view; and for each attention head, fusing the initial attention matrix of the view and the corresponding position code to obtain a key attention matrix corresponding to the view.

[0075] Among them, the initial attention matrix includes a QKV matrix. The initial attention matrix represents the QKV matrix obtained by processing the visual features of interest of the perspective through the attention mechanism. The center of interest vector is used to describe the position of the region of interest. The central coordinates may refer to the coordinates of the center point of the region of interest, the region of interest is a rectangle, and the center point is the intersection of the diagonals of the rectangle. According to the coordinates and width and height of the key points in each position of interest feature, the center coordinates of the position of interest feature are calculated. The center coordinates of different position of interest features of the same perspective are spliced ​​to obtain the center of interest vector of the perspective. The center of interest vector is encoded for feature extraction of the center of interest vector. The initial attention matrix is ​​fused with the position coding to realize the fusion of the initial attention matrix of the perspective with the position information to enrich the feature information of the initial attention matrix. In some embodiments, the position coding is accumulated into the query matrix of the initial attention matrix, the query matrix is ​​updated, the position coding is accumulated into the key matrix of the initial attention matrix, the key matrix is ​​updated, and the value matrix of the initial attention matrix may not be changed, and the updated initial attention matrix is ​​the attention key matrix.

[0076] In some embodiments, the visual features F1 of the region of interest of the first perspective are transformed into initial attention matrices of multiple attention heads through multiple linear transformations. An initial attention matrix includes query, key, and value matrices:

[0077] , ,

[0078] Among them, W 1,q , W 1,k and W 1,v are learnable parameters in model training, which are weight matrices of query matrix, key matrix and value matrix.

[0079] Correspondingly, the visual features F2 of the region of interest of the second perspective are transformed into the initial attention matrices of multiple attention heads through multiple linear transformations:

[0080] , ,

[0081] The position encoding is usually a vector with the same dimension as the feature map (visual feature of interest), which is added element by element to the original feature, so that the model can maintain sensitivity to position while focusing on the feature content. The coordinates of the center point of each region of interest are encoded by sine-cosine position:

[0082]

[0083]

[0084]

[0085] in, is the coordinate of the center point of the nth region of interest in the first perspective, is the center of interest vector, which is the vector formed by stitching the coordinates of the center points of all the regions of interest from the first perspective. is the position encoding of all regions of interest from the first perspective obtained by linear transformation, d is the number of regions of interest, and d is also the dimension of the center of interest vector, is the mapping matrix, is the bias term.

[0086] Among them, the query matrix and the key matrix in the initial attention matrix of the first perspective are respectively accumulated with the position encoding to obtain the updated query matrix and key matrix :

[0087] ,

[0088] Similarly, after the query matrix and key matrix in the initial attention matrix of the second perspective are respectively accumulated with position encoding, the updated query matrix is ​​obtained Sum and bond matrix :

[0089] ,

[0090] It can be seen that by adding the position coding obtained by encoding the center coordinates of the position features of interest to the initial attention matrix obtained by the linear change of the visual features of the viewing angle, the visual features and the position features can be fused. While paying attention to the image feature content, the sensitivity to the position can be maintained. By encoding the center coordinates to obtain the position coding, the operation of obtaining the position features can be simplified, and the position features that meet the geometric features of the region of interest can be obtained as the fusion object, so that the position features of interest and the visual features of interest are naturally fused. Finally, based on the fused attention key matrix, richer feature information can be extracted to improve the accuracy of target detection.

[0091] S203. For every two perspectives, the query matrix in the attention key matrix of the first perspective and the key matrix in the attention key matrix of the second perspective are merged to obtain a weight matrix of the first perspective to the second perspective.

[0092] In each group of two perspectives, one perspective is determined as the first perspective, and the other perspective is determined as the second perspective. The fusion method can be to calculate the dot product of the query matrix and the key matrix. The weight matrix is ​​used to describe the degree of attention of each region of interest in the first perspective to each region of interest in the second perspective.

[0093] In some embodiments, the weight matrix of the first perspective to the second perspective can be calculated using the following formula: :

[0094]

[0095] in, represents the dimension of the key vector, is a scaling factor to avoid the dot product being too large.

[0096] In some embodiments, the weight matrix Attention of the second perspective to the first perspective 2→1 .

[0097] S204. Calculate the attention matrix of the first perspective to the second perspective according to the weight matrix of the first perspective to the second perspective and the median matrix of the attention key matrix of the second perspective as the attention result between the two perspectives.

[0098] The attention matrix is ​​used to describe the degree of attention paid to the features of the second perspective from the first perspective. The attention matrix is ​​obtained by fusing the weight matrix of the first perspective to the second perspective and the median matrix of the attention key matrix of the second perspective.

[0099] In some embodiments, the following formula can be used, based on the weight matrix Attention 1→2 Perform weighted summation on the value matrix V2 of the second perspective to obtain the attention matrix :

[0100]

[0101] In some embodiments, the weight matrix of the second perspective to the first perspective is .

[0102] S205: For every two perspectives, the attention results between the two perspectives under each of the attention heads are fused to obtain multi-head attention results of the two perspectives.

[0103] In an optional embodiment, the attention results between the two perspectives under each of the attention heads are fused to obtain a multi-head attention result of the two perspectives, including: fusing the attention matrices of the first perspective to the second perspective under each of the attention heads to obtain a fused attention matrix of the first perspective to the second perspective; wherein the multi-head attention result of the two perspectives includes a fused attention matrix of the first perspective to the second perspective.

[0104] Among them, the attention matrices of the first perspective to the second perspective under each attention head are spliced ​​to obtain the fusion attention matrix of the first perspective to the second perspective. In some embodiments, the attention matrices are expanded from a single attention head to multiple single-head attention heads, and the attention matrices are spliced, and then the final multi-head cross attention result is obtained through linear transformation, that is, the following formula can be used to calculate the fusion attention matrix of the first perspective to the second perspective :

[0105]

[0106] Among them, the multi-head attention result can also include the fusion attention matrix of the second perspective to the first perspective .

[0107] It can be seen that by splicing the attention matrices of the first perspective to the second perspective in each attention head, a fused attention matrix of the first perspective to the second perspective is obtained. Multiple dimensions of key information can be added to the fused attention matrix to improve the representativeness of the fused attention matrix.

[0108] S206: Determine the target feature of each of the perspectives according to the multi-head attention results of the two perspectives and the visual features of interest of each of the perspectives.

[0109] In an optional embodiment, the target features of each of the viewpoints are determined based on the multi-head attention results of the two viewpoints and the visual features of interest of each of the viewpoints, including: for each of the viewpoints, updating the visual features of interest of the viewpoint based on the fusion attention matrix of the two viewpoints to which the viewpoint belongs; for each of the viewpoints, performing feature extraction processing on the updated visual features of interest of the viewpoint to obtain the target features of the viewpoint.

[0110] Among them, for each perspective, the fusion attention matrix of the perspective to each other perspective is adjusted and updated to the visual feature of interest of the perspective to obtain the updated visual feature of interest of the perspective. In some embodiments, the fusion attention matrix of the perspective to each other perspective is superimposed on the visual feature of interest to obtain the updated visual feature of interest. The updated visual feature of interest is processed nonlinearly to obtain the target feature. For example, the fusion attention matrix of the i-th perspective to the j-th perspective is superimposed on the visual feature of interest of the i-th perspective to obtain the updated visual feature of interest of the i-th perspective. Among them, i is not equal to j, i∈R, j∈R, and R is the set of perspectives.

[0111] In one example, the number of perspectives is two, and the visual features of interest from the first perspective and the fusion attention matrix of the first perspective to the second perspective are processed by residual connection and layer normalization to obtain the updated visual features of interest. :

[0112]

[0113] Updated first-person perspective visual features of interest After the feedforward neural network (FFM) increases nonlinearity and expression ability, the target feature Fffm,1 is obtained.

[0114] It can be seen that by fusion attention matrix of the two perspectives to which each perspective belongs, the visual features of interest of the perspective are updated, and the feature extraction process is continued to obtain the target features of the perspective, so as to achieve full fusion of multi-perspective feature information and improve the accuracy of target detection.

[0115] S207 . Determine target detection results of the captured images of each of the viewing angles according to the target features of each of the viewing angles.

[0116] The technical solution of the embodiment of the present invention generates a cross-perspective attention matrix by performing attention mechanism calculation on the query matrix of one perspective and the key matrix and value matrix of the other perspective for every two perspectives, thereby focusing on multiple relevant features of one perspective in another perspective, and fully integrating the features of the two perspectives from a global perspective, enriching the image features of each perspective, and increasing the representativeness of the image features of each perspective, thereby performing target detection based on the fused target features, thereby improving the accuracy of target detection.

[0117] In a scene, there are two perspectives. Feature extraction is performed on the captured images of the two perspectives of the same scene to obtain the visual features of interest and the position features of interest corresponding to each perspective. For each perspective, the visual features of interest corresponding to the perspective are linearly processed to obtain the initial attention matrix of the perspective. For each perspective, the center coordinates of at least one position feature of interest corresponding to the perspective are calculated, and the center vector of interest corresponding to the perspective is obtained by splicing. The center vector of interest corresponding to the perspective is encoded to obtain the position code corresponding to the perspective. For each attention head, the initial attention matrix and the corresponding position code corresponding to the perspective are fused to obtain the attention key matrix corresponding to the perspective. For each attention head, the query matrix in the attention key matrix of the first perspective and the key matrix in the attention key matrix of the second perspective are fused to obtain the weight matrix of the first perspective to the second perspective. According to the weight matrix of the first perspective to the second perspective and the median matrix of the attention key matrix of the second perspective, the attention matrix of the first perspective to the second perspective is calculated. The attention matrix of the first perspective to the second perspective under each attention head is fused to obtain the attention matrix of the first perspective to the second perspective. a fusion attention matrix; according to the visual features of interest of the first perspective and the fusion attention matrix of the first perspective to the second perspective, the visual features of interest of the first perspective are updated; the updated visual features of interest of the first perspective are processed to obtain the target features of the first perspective; according to the target features of the first perspective, the target detection result of the captured image of the first perspective is determined; for each attention head, the query matrix in the attention key matrix of the second perspective and the key matrix in the attention key matrix of the first perspective are fused to obtain the weight matrix of the second perspective to the first perspective; according to the weight matrix of the second perspective to the first perspective and the median matrix of the attention key matrix of the first perspective, the attention matrix of the second perspective to the first perspective is calculated; the attention matrices of the second perspective to the first perspective under each attention head are fused to obtain the fusion attention matrix of the second perspective to the first perspective; according to the visual features of interest of the second perspective and the fusion attention matrix of the second perspective to the first perspective, the visual features of interest of the second perspective are updated; the updated visual features of interest of the second perspective are processed to obtain the target features of the second perspective; according to the target features of the second perspective, the target detection result of the captured image of the second perspective is determined.

[0118] Figure 3A flowchart of a method for training a multi-view target detection model provided by an embodiment of the present invention. The embodiment of the present invention is applicable to the case of training a multi-view target detection model. The method can be executed by a training device for a multi-view target detection model. The training device for a multi-view target detection model can be implemented in the form of hardware and / or software. The training device for a multi-view target detection model can be configured in an electronic device that carries the training function of the multi-view target detection model, such as a client or a server. The client may include: a mobile phone, a tablet computer, a laptop computer, a desktop computer, a wearable device, etc.

[0119] See also Figure 3 The training method of the multi-view target detection model shown includes:

[0120] S301. Through the twin network feature extraction network in the multi-view target detection model, feature extraction processing is performed on sample images of at least two viewpoints of the same scene to obtain visual features of interest and position features of interest corresponding to each of the viewpoints.

[0121] Among them, multi-view image acquisition is performed on the same scene to obtain sample images. In some embodiments, images of multiple objects under multiple viewpoints can be first acquired as sample images, and the images of the same object under multiple viewpoints can be marked with the same identification information, and the objects can be marked in each sample image respectively, and the target frame of the marked object can be used as the standard detection result. Each sample image is divided into a training set and a test set according to the quantity of 4:1. For example, the object is a logistics package. In the embodiment of the present invention, the sample image is an image of the same object under multiple viewpoints.

[0122] Among them, the twin network feature extraction network is used to extract visual features of interest and position features of interest.

[0123] S302. Through the multi-head cross-attention network in the multi-view target detection model, for each of the viewpoints, according to the interesting visual features and interesting position features corresponding to the viewpoint, determine the attention key matrix corresponding to the viewpoint under at least one attention head.

[0124] Among them, the multi-head cross-attention network is used to generate the initial attention matrix, and superimpose the position encoding to obtain the attention key matrix, and realize the fusion of the attention key matrix across perspectives to obtain the target features of each perspective, and obtain the target detection results based on the target feature detection.

[0125] S303. Through the multi-head cross attention network, for each of the attention heads, according to the attention key matrix corresponding to each of the perspectives, calculate the attention result between every two perspectives.

[0126] S304. Through the multi-head cross attention network, for every two perspectives, the attention results between the two perspectives under each of the attention heads are fused to obtain the multi-head attention results of the two perspectives.

[0127] S305. Determine the target features of each of the perspectives according to the multi-head attention results of each of the two perspectives and the visual features of interest of each of the perspectives through the multi-head cross attention network.

[0128] S306. Determine the predicted detection results of the sample images of each of the perspectives based on the target features of each of the perspectives through the multi-head cross attention network.

[0129] Among them, the predicted detection result is the target detection result output by the multi-head cross-attention network.

[0130] S307. Adjust the parameters of the twin network feature extraction network and the multi-head cross attention network according to the difference between the standard detection results and the predicted detection results of the sample images of each perspective.

[0131] Among them, the standard detection result is the correct target detection result in the sample image, which can be the target detection result obtained by manually annotating the sample image. For each perspective, the difference between the standard detection result and the predicted detection result is calculated. According to the difference in the perspective, the loss value of the perspective is calculated, the loss values ​​of all perspectives are accumulated, and the parameters of the twin network feature extraction network and the multi-head cross attention network are adjusted. When the loss value is minimized, the loss value converges, or the number of iterations is greater than or equal to the number threshold, it can be determined that the training of the twin network feature extraction network and the multi-head cross attention network is completed, that is, the training of the multi-view target detection model is completed. The multi-view target detection model can be published for application.

[0132] In some embodiments, the loss value L may be calculated based on the following formula:

[0133]

[0134] Among them, L i,cls is the classification loss value of the i-th perspective, L i,reg is the regression loss value of the i-th view. The above formula is used to calculate the loss value L when the number of viewpoints is two. Among them, the classification loss value is calculated using the cross entropy loss. The regression loss value is calculated using DIoU (Distance Intersection over Union). is the weight coefficient.

[0135] The technical solution of the embodiment of the present invention obtains sample images of multiple perspectives of the same scene and performs feature extraction to obtain visual features of interest and position features of interest, and determines the attention key matrix corresponding to each perspective under each attention head for the visual features of interest and position features of interest corresponding to each perspective, calculates the attention results of each perspective for other perspectives for each attention head, and fuses the attention results of different attention heads to obtain multi-head attention results of each perspective for other perspectives, and finally updates the visual features of interest of the perspective based on the multi-head attention results of the perspective for other perspectives, determines the target features of the perspective, and determines the target detection results of the perspective based on the target features, so that the scene information of other perspectives can be added to the visual features of interest of each perspective, enriching the image features of each perspective, improving the representativeness of the visual features of interest, and realizing effective fusion of image data from multiple perspectives. Through cross-perspective feature fusion, the model learns the common feature information between different perspectives, has stronger generalization ability, and improves the target detection accuracy of the model.

[0136] In a scenario, such as Figure 4 As shown, two viewing angles are used to capture images of the same scene, and a sample image corresponding to the viewing angle is obtained.

[0137] The first stage (blue line) is the region-based fast convolutional neural network - learnable candidate regions (Faster R-CNN Learnable Proposals): The collected images are processed through the Faster R-CNN network to obtain the visual features of the region of interest and the position of interest. Based on the region of interest pooling method, two parameter-sharing backbones are used to extract the features of the sample images of the two perspectives to obtain the feature maps of each perspective. For each perspective, the feature map of the perspective is input into the parameter-independent FPN for processing, and the feature vector set F of the visual region of interest is output as the visual feature of interest for that perspective; and the feature map is input into the RPN for processing, and the position set P of the region of interest is output as the position feature of interest for that perspective.

[0138] The second stage (red line): Through a multi-head cross-attention network, the visual features of interest and the position features of interest are processed to obtain a fused attention matrix as the multi-head attention result. Specifically, for each perspective, the visual features of interest corresponding to the perspective are linearly processed to obtain the Q matrix, K matrix and V matrix as the initial attention matrix of the perspective. For each perspective, the center coordinates of at least one position feature of interest corresponding to the perspective are calculated, and the center vector of interest corresponding to the perspective is obtained by splicing; the center vector of interest corresponding to the perspective is encoded to obtain the position code corresponding to the perspective; for each attention head, the initial attention matrix corresponding to the perspective and the corresponding position code are fused to obtain the attention key matrix corresponding to the perspective, that is, Figure 4 Q1, K1, V1, Q2, K2 and V2. Use the CrossAttention network to calculate the weight matrix of the first perspective to the second perspective , based on the weight matrix Perform weighted summation on the value matrix V2 of the second perspective to obtain the attention matrix The attention matrices are concatenated, and then linearly transformed to obtain the final multi-head cross attention result, that is, to calculate the fusion attention matrix of the first perspective to the second perspective , through residual connection and layer normalization (Add Norm), the visual features of interest from the first perspective and the fusion attention matrix of the first perspective on the second perspective are processed to obtain the visual features of interest from the first perspective , thereby obtaining cross-view features. Similarly, the visual features of interest from the second perspective are calculated .

[0139] The third stage (green line): The visual features of interest are processed through the regression classification network to obtain the target detection results, that is, the target box and the classification labels of each target box. The visual features of interest from the first perspective are processed through the feedforward neural network. Perform nonlinear changes to obtain the target feature F from the first perspective ffm,1 Through residual connection and layer normalization (Add Norm), the visual features of interest from the first perspective are and the target feature F ffm,1 , determine the target detection results, that is, the target boxes and classification labels in the captured image of the first perspective. Similarly, detect the target boxes and classification labels in the captured image of the second perspective.

[0140] in, Figure 4The target boxes of different colors in the captured images represent different classification labels. For example, the green box, blue box, and red box represent objects of different categories. After experiments, single-view and dual-view algorithm comparison experiments were conducted on the dataset. The recall rates were calculated when the false positive ratios on each image were controlled to be 0.5 and 1, respectively, as shown in Table 1 below:

[0141] Table 1

[0142] Viewing angle R@0.5 R@1 Single view 1 0.9465 0.9551 Single View 2 0.9186 0.9306 Dual perspective 0.9568 0.9646

[0143] From the experimental results, the multi-view target detection method proposed in the embodiment of the present invention improves detection accuracy and reduces false positives. False Positive (FP) refers to the situation where the model predicts that the target box is a positive class (positive), but the actual classification label of the target box is a negative class.

[0144] The embodiment of the present invention adopts a twin network feature extraction network to share some parameters of the control part, thereby reducing video memory occupancy and computational complexity. By adopting the high-dimensional features of the center coordinates of the region of interest after linear transformation as the position code, the center point coordinates are more in line with the geometric features of the region of interest than the complex spatial information decomposed by sine-cosine coding, so that the position code and the features of the region of interest are more naturally integrated; the multi-view feature information is fully integrated through the multi-head cross-attention mechanism to achieve efficient cross-view feature fusion and improve the target detection accuracy; through cross-view feature fusion and position coding, the model can learn the common feature information between different viewpoints, have stronger generalization ability, achieve flexible multi-view adaptability, and improve the model's target detection accuracy.

[0145] Figure 5 A schematic diagram of the structure of a multi-view target detection device provided by an embodiment of the present invention. The embodiment of the present invention can be applied to the case of performing target detection based on multi-view captured images, the device can execute a multi-view target detection method, the device can be implemented in the form of hardware and / or software, and the device can be configured in an electronic device that carries a multi-view target detection function.

[0146] See also Figure 5 The multi-view target detection device shown includes:

[0147] The twin network feature extraction network 501 in the multi-view target detection model is used to perform feature extraction processing on the collected images of at least two viewpoints of the same scene to obtain the visual features of interest and the position features of interest corresponding to each of the viewpoints;

[0148] The multi-head cross attention network 502 in the multi-view object detection model is used to determine, for each of the view angles, an attention key matrix corresponding to the view angle under at least one attention head according to the visual features of interest and the position features of interest corresponding to the view angle;

[0149] The multi-head cross attention network 502 is used to calculate the attention result between each two perspectives according to the attention key matrix corresponding to each perspective for each attention head;

[0150] The multi-head cross attention network 502 is used to fuse the attention results between the two perspectives under each of the attention heads for each two perspectives to obtain the multi-head attention results of the two perspectives;

[0151] The regression classification network 503 in the multi-view target detection model is used to determine the target features of each of the two views according to the multi-head attention results of each of the two views and the visual features of interest of each of the views;

[0152] The regression classification network 503 is used to determine the target detection results of the captured images of each of the viewing angles according to the target features of each of the viewing angles.

[0153] The technical solution of the embodiment of the present invention obtains captured images of multiple perspectives of the same scene and performs feature extraction to obtain visual features of interest and position features of interest, and determines the attention key matrix corresponding to each perspective under each attention head for the visual features of interest and position features of interest corresponding to each perspective, calculates the attention results of each perspective for other perspectives for each attention head, and fuses the attention results of different attention heads to obtain multi-head attention results of each perspective for other perspectives, and finally updates the visual features of interest of the perspective based on the multi-head attention results of the perspective for other perspectives, determines the target features of the perspective, and determines the target detection results of the perspective based on the target features, so that the scene information of other perspectives can be added to the visual features of interest of each perspective, enriching the image features of each perspective, improving the representativeness of the visual features of interest, realizing image data fusion of multiple perspectives, and thus improving the accuracy of target detection.

[0154] Optionally, the multi-head cross attention network 502 is specifically used for:

[0155] For every two perspectives, the query matrix in the attention key matrix of the first perspective and the key matrix in the attention key matrix of the second perspective are merged to obtain a weight matrix of the first perspective to the second perspective;

[0156] According to the weight matrix of the first perspective to the second perspective and the median matrix of the attention key matrix of the second perspective, the attention matrix of the first perspective to the second perspective is calculated as the attention result between the two perspectives.

[0157] Optionally, the multi-head cross attention network 502 is specifically used for:

[0158] The attention matrices of the first perspective to the second perspective under each of the attention heads are fused to obtain a fused attention matrix of the first perspective to the second perspective; wherein the multi-head attention result of the two perspectives includes the fused attention matrix of the first perspective to the second perspective.

[0159] Optionally, the regression classification network 503 is specifically used for:

[0160] For each of the perspectives, updating the visual features of interest of the perspective according to the fusion attention matrix of the two perspectives to which the perspective belongs;

[0161] For each of the viewing angles, feature extraction processing is performed on the updated visual features of interest of the viewing angle to obtain the target features of the viewing angle.

[0162] Optionally, the multi-head cross attention network 502 is specifically used for:

[0163] Linearly process the visual features of interest corresponding to the perspective to obtain an initial attention matrix of the perspective under at least one attention head;

[0164] Calculate the center coordinates of at least one interesting position feature corresponding to the viewing angle;

[0165] The center coordinates of each of the features at the position of interest are spliced ​​to obtain a center vector of interest corresponding to the viewing angle;

[0166] Encoding a center of interest vector corresponding to the viewing angle to obtain a position code corresponding to the viewing angle;

[0167] For each of the attention heads, the initial attention matrix of the perspective and the corresponding position encoding are fused to obtain the attention key matrix corresponding to the perspective.

[0168] The multi-view target detection device provided in the embodiment of the present invention can execute the multi-view target detection method provided in any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of the execution method.

[0169] Figure 6A schematic diagram of the structure of a training device for a multi-view target detection model provided by an embodiment of the present invention. The embodiment of the present invention can be applied to the case of training a multi-view target detection model, the device can execute a training method for a multi-view target detection model, the device can be implemented in the form of hardware and / or software, and the device can be configured in an electronic device that carries the training function of the multi-view target detection model.

[0170] See also Figure 6 The training device of the multi-view target detection model shown includes:

[0171] The twin network feature extraction network 601 in the multi-view target detection model is used to perform feature extraction processing on sample images of at least two viewpoints of the same scene to obtain the visual features of interest and the position features of interest corresponding to each of the viewpoints;

[0172] The multi-head cross attention network 602 in the multi-view object detection model is used to determine, for each of the view angles, an attention key matrix corresponding to the view angle under at least one attention head according to the interesting visual features and interesting position features corresponding to the view angle;

[0173] The multi-head cross attention network 602 is used to calculate the attention result between each two perspectives according to the attention key matrix corresponding to each perspective for each attention head;

[0174] The multi-head cross attention network 602 is used to fuse the attention results between the two perspectives under each of the attention heads for each two perspectives to obtain the multi-head attention results of the two perspectives;

[0175] The regression classification network 603 in the multi-view target detection model is used to determine the target features of each of the two views according to the multi-head attention results of each of the two views and the visual features of interest of each of the views;

[0176] The regression classification network 603 is used to determine the prediction detection results of the sample images of each of the perspectives according to the target features of each of the perspectives;

[0177] The model training module 604 is used to adjust the parameters of the twin network feature extraction network and the multi-head cross attention network according to the difference between the standard detection results and the predicted detection results of the sample images of each perspective.

[0178] The technical solution of the embodiment of the present invention obtains sample images of multiple perspectives of the same scene and performs feature extraction to obtain visual features of interest and position features of interest, and determines the attention key matrix corresponding to each perspective under each attention head for the visual features of interest and position features of interest corresponding to each perspective, calculates the attention results of each perspective for other perspectives for each attention head, and fuses the attention results of different attention heads to obtain multi-head attention results of each perspective for other perspectives, and finally updates the visual features of interest of the perspective based on the multi-head attention results of the perspective for other perspectives, determines the target features of the perspective, and determines the target detection results of the perspective based on the target features, so that the scene information of other perspectives can be added to the visual features of interest of each perspective, enriching the image features of each perspective, improving the representativeness of the visual features of interest, and realizing effective fusion of image data from multiple perspectives. Through cross-perspective feature fusion, the model learns the common feature information between different perspectives, has stronger generalization ability, and improves the target detection accuracy of the model.

[0179] The training device for the multi-view target detection model provided in the embodiment of the present invention can execute the training method for the multi-view target detection model provided in any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of the execution method.

[0180] In the technical solutions of the embodiments of the present invention, the acquisition, storage and application of the data involved are in compliance with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0181] Figure 7 A schematic diagram of the structure of an electronic device 700 that can be used to implement an embodiment of the present invention is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices (such as helmets, glasses, watches, etc.) and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present invention described and / or required herein.

[0182] like Figure 7As shown, the electronic device 700 includes at least one processor 701, and a memory connected to the at least one processor 701 in communication, such as a read-only memory (ROM) 702, a random access memory (RAM) 703, etc., wherein the memory stores a computer program that can be executed by at least one processor, and the processor 701 can perform various appropriate actions and processes according to the computer program stored in the read-only memory (ROM) 702 or the computer program loaded from the storage unit 708 to the random access memory (RAM) 703. In the RAM 703, various programs and data required for the operation of the electronic device 700 can also be stored. The processor 701, the ROM 702, and the RAM 703 are connected to each other via a bus 704. An input / output (I / O) interface 705 is also connected to the bus 704.

[0183] Multiple components in the electronic device 700 are connected to the I / O interface 705, including: an input unit 706, such as a keyboard, a mouse, etc.; an output unit 707, such as various types of displays, speakers, etc.; a storage unit 708, such as a disk, an optical disk, etc.; and a communication unit 709, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 709 allows the electronic device 700 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0184] The processor 701 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the processor 701 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any appropriate processor, controller, microcontroller, etc. The processor 701 performs the various methods and processes described above, such as a multi-view target detection method or a training method for a multi-view target detection model.

[0185] In some embodiments, the multi-view target detection method or the training method of the multi-view target detection model may be implemented as a computer program, which is tangibly contained in a computer-readable storage medium, such as a storage unit 708. In some embodiments, part or all of the computer program may be loaded and / or installed on the electronic device 700 via the ROM 702 and / or the communication unit 709. When the computer program is loaded into the RAM 703 and executed by the processor 701, one or more steps of the multi-view target detection method or the training method of the multi-view target detection model described above may be performed. Alternatively, in other embodiments, the processor 701 may be configured to execute the multi-view target detection method or the training method of the multi-view target detection model by any other appropriate means (e.g., by means of firmware).

[0186] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0187] Computer programs for implementing the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that when the computer program is executed by the processor, the functions / operations specified in the flow chart and / or block diagram are implemented. The computer program may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0188] In the context of the present invention, a computer-readable storage medium may be a tangible medium that may contain or store a computer program for use by or in conjunction with an instruction execution system, device, or equipment. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices, or equipment, or any suitable combination of the foregoing. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0189] To provide interaction with a user, the systems and techniques described herein may be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and a pointing device (e.g., a mouse or trackball) through which the user can provide input to the electronic device. Other types of devices may also be used to provide interaction with the user; for example, the feedback provided to the user may be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user may be received in any form (including acoustic input, voice input, or tactile input).

[0190] The systems and techniques described herein may be implemented in a computing system that includes backend components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes frontend components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such backend components, middleware components, or frontend components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), a blockchain network, and the Internet.

[0191] A computing system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The client and server relationship is generated by computer programs running on the corresponding computers and having a client-server relationship with each other. The server may be a cloud server, also known as a cloud computing server or cloud host, which is a host product in the cloud computing service system to solve the defects of difficult management and weak business scalability in traditional physical hosts and VPS (Virtual Private Server) services.

[0192] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps described in the present invention can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solution of the present invention can be achieved, and this document does not limit this.

[0193] The above specific implementations do not constitute a limitation on the protection scope of the present invention. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A multi-view target detection method, characterized in that: The method comprises: Performing feature extraction processing on the collected images of at least two viewing angles of the same scene to obtain the visual features of interest and the position features of interest corresponding to each of the viewing angles; For each of the viewing angles, determining an attention key matrix corresponding to the viewing angle under at least one attention head according to the visual features of interest and the position features of interest corresponding to the viewing angle; For each of the attention heads, according to the attention key matrix corresponding to each of the perspectives, calculating the attention result between every two perspectives; For every two perspectives, the attention results between the two perspectives under each of the attention heads are fused to obtain multi-head attention results of the two perspectives; Determining target features of each of the perspectives according to the multi-head attention results of each of the two perspectives and the visual features of interest of each of the perspectives; According to the target features of each of the viewing angles, the target detection results of the captured images of each of the viewing angles are determined.

2. The method according to claim 1, characterized in that The step of calculating the attention result between each two perspectives according to the attention key matrix corresponding to each perspective includes: For every two perspectives, the query matrix in the attention key matrix of the first perspective and the key matrix in the attention key matrix of the second perspective are merged to obtain a weight matrix of the first perspective to the second perspective; According to the weight matrix of the first perspective to the second perspective and the median matrix of the attention key matrix of the second perspective, the attention matrix of the first perspective to the second perspective is calculated as the attention result between the two perspectives.

3. The method according to claim 2, characterized in that The fusing the attention results between the two perspectives under each of the attention heads to obtain the multi-head attention results of the two perspectives includes: The attention matrices of the first perspective to the second perspective under each of the attention heads are fused to obtain a fused attention matrix of the first perspective to the second perspective; wherein the multi-head attention result of the two perspectives includes the fused attention matrix of the first perspective to the second perspective.

4. The method according to claim 3, characterized in that Determining the target features of each of the perspectives according to the multi-head attention results of each of the two perspectives and the visual features of interest of each of the perspectives includes: For each of the perspectives, updating the visual features of interest of the perspective according to the fusion attention matrix of the two perspectives to which the perspective belongs; For each of the viewing angles, feature extraction processing is performed on the updated visual features of interest of the viewing angle to obtain the target features of the viewing angle.

5. The method according to claim 1, characterized in that The determining, according to the visual features of interest and the position features of interest corresponding to the perspective, an attention key matrix corresponding to the perspective under at least one attention head comprises: Linearly process the visual features of interest corresponding to the perspective to obtain an initial attention matrix of the perspective under at least one attention head; Calculate the center coordinates of at least one interesting position feature corresponding to the viewing angle; The center coordinates of each of the features at the position of interest are spliced ​​to obtain a center vector of interest corresponding to the viewing angle; Encoding a center of interest vector corresponding to the viewing angle to obtain a position code corresponding to the viewing angle; For each of the attention heads, the initial attention matrix of the perspective and the corresponding position encoding are fused to obtain the attention key matrix corresponding to the perspective.

6. A training method for a multi-view target detection model, characterized in that: The method comprises: Through the twin network feature extraction network in the multi-view target detection model, feature extraction processing is performed on sample images of at least two viewpoints of the same scene to obtain the visual features of interest and the position features of interest corresponding to each of the viewpoints; By using a multi-head cross attention network in the multi-view target detection model, for each of the viewpoints, according to the interesting visual features and interesting position features corresponding to the viewpoint, determining an attention key matrix corresponding to the viewpoint under at least one attention head; By using the multi-head cross attention network, for each of the attention heads, according to the attention key matrix corresponding to each of the perspectives, the attention result between each two perspectives is calculated; Through the multi-head cross attention network, for each two perspectives, the attention results between the two perspectives under each of the attention heads are fused to obtain the multi-head attention results of the two perspectives; Determining target features of each of the views according to the multi-head attention results of each of the two views and the visual features of interest of each of the views through a regression classification network in the multi-view target detection model; Determine the predicted detection results of the sample images of each of the viewing angles according to the target features of each of the viewing angles through the regression classification network; According to the difference between the standard detection results and the predicted detection results of the sample images of each perspective, the parameters of the twin network feature extraction network and the multi-head cross attention network are adjusted.

7. A multi-view target detection device, characterized in that: The device comprises: The twin network feature extraction network in the multi-view target detection model is used to perform feature extraction processing on the collected images of at least two viewpoints of the same scene to obtain the visual features of interest and the position features of interest corresponding to each of the viewpoints; The multi-head cross attention network in the multi-view target detection model is used to determine, for each of the view angles, an attention key matrix corresponding to the view angle under at least one attention head according to the visual features of interest and the position features of interest corresponding to the view angle; The multi-head cross attention network is used to calculate the attention result between each two perspectives according to the attention key matrix corresponding to each perspective for each attention head; The multi-head cross attention network is used to fuse the attention results between the two perspectives under each of the attention heads for each two perspectives to obtain the multi-head attention results of the two perspectives; The regression classification network in the multi-view target detection model is used to determine the target features of each of the two views according to the multi-head attention results of each of the two views and the visual features of interest of each of the views; The regression classification network is used to determine the target detection results of the captured images of each of the viewing angles based on the target features of each of the viewing angles.

8. A training device for a multi-view target detection model, characterized in that: The device comprises: The twin network feature extraction network in the multi-view target detection model is used to perform feature extraction processing on sample images of at least two viewpoints of the same scene to obtain the visual features of interest and the position features of interest corresponding to each of the viewpoints; The multi-head cross attention network in the multi-view target detection model is used to determine, for each of the view angles, an attention key matrix corresponding to the view angle under at least one attention head according to the visual features of interest and the position features of interest corresponding to the view angle; The multi-head cross attention network is used to calculate the attention result between each two perspectives according to the attention key matrix corresponding to each perspective for each attention head; The multi-head cross attention network is used to fuse the attention results between the two perspectives under each of the attention heads for each two perspectives to obtain the multi-head attention results of the two perspectives; The regression classification network in the multi-view target detection model is used to determine the target features of each of the two views according to the multi-head attention results of each of the two views and the visual features of interest of each of the views; The regression classification network is used to determine the predicted detection results of the sample images of each of the viewing angles according to the target features of each of the viewing angles; The model training module is used to adjust the parameters of the twin network feature extraction network and the multi-head cross attention network according to the difference between the standard detection results and the predicted detection results of the sample images of each perspective.

9. An electronic device, characterized in that: The electronic device comprises: at least one processor; and a memory communicatively connected to the at least one processor; wherein, The memory stores a computer program that can be executed by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute the multi-view target detection method described in any one of claims 1 to 5, or the training method of the multi-view target detection model described in any one of claim 6.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the multi-view target detection method described in any one of claims 1 to 5, or the training method of a multi-view target detection model described in any one of claim 6 when executed.

Images