A target identification method and device, electronic equipment and storage medium

Abstract

The application relates to the technical field of image recognition, in particular to a target recognition method and device, electronic equipment and a storage medium. The application obtains a plurality of transformed feature maps by performing a plurality of affine transformations on a first feature map of a first frame of image, determines a target rotation angle through rotation invariance scores between each transformed feature map and a second feature map, and then inputs the transformed feature map corresponding to the target rotation angle into a trained target rotation equivalence encoder to obtain a template feature vector. Further, based on the rotation invariance score between a query feature vector of a second frame of image and the template feature vector, it is determined whether a candidate object in the second frame of image is a target object. In this way, continuous recognition of a target object performing three-dimensional rotation in a target video can be realized, and the accuracy of target object recognition can be improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image recognition, and particularly relates to a target recognition method and device, electronic equipment and a storage medium. BACKGROUND

[0002] Target recognition technology is to recognize a target object from an image and give position information of the target object. The target recognition technology has been widely applied in many fields, such as biomedical, satellite remote sensing, robot vision, cargo detection, target tracking, autonomous vehicle navigation, public security, bank, transportation, military, e-commerce and multimedia network communication.

[0003] Generally, in the process of identifying a target object, most of the captured video or time sequence images are two-dimensional, that is, planar. However, when the target object rotates in three dimensions, the general target recognition technology will have the phenomenon of recognition error or inability to recognize.

[0004] At present, the target recognition technology for the target object rotating in three dimensions is to use the initial state of the target object recognized in the video frame as a feature template, and then extract the features of the surrounding area of the target object to form candidate features, and realize continuous recognition when the target moves based on the template features and the candidate features. This method is based on the principle of translation equivalence, although it can identify the rotating target object to some extent, but the recognition accuracy is not high, and there are failure or deviation cases. SUMMARY

[0005] Therefore, the embodiments of the present application provide at least a target recognition method and device, electronic equipment and a storage medium, which can realize continuous recognition of the target object rotating in three dimensions in a video, and can improve the accuracy of recognizing the target object.

[0006] The present application mainly includes the following aspects:

[0007] In a first aspect, an embodiment of the present application provides a target recognition method, the target recognition method comprising: performing multiple affine transformations on a first feature map of a first frame image in a target video to obtain multiple transformed feature maps; calculating a rotation invariance score between each transformed feature map of the first frame image and a second feature map of a second frame image, and determining a target rotation angle corresponding to a highest rotation invariance score; the second frame image is an image located after the first frame image in the target video, and the first frame image and the second frame image are adjacent; inputting the transformed feature map corresponding to the target rotation angle into a target rotation equivalent encoder of a trained target recognition model to obtain a template feature vector, and inputting the second feature map of the second frame image into the target rotation equivalent encoder to obtain a query feature vector; determining whether a candidate object in the second frame image and a target object in the first frame image are the same object based on a rotation invariance score between the query feature vector of the second frame image and the template feature vector of the first frame image.

[0008] In a possible implementation, the performing multiple affine transformations on the first feature map of the first frame image in the target video to obtain the multiple transformed feature maps comprises: predicting a target motion direction of the target object in the first frame image based on change information between the first frame image and the second frame image; and performing multiple affine transformations on the first feature map of the first frame image according to multiple rotation angles associated with the target motion direction to obtain the multiple transformed feature maps.

[0009] In a possible implementation, the target rotation equivalent encoder is trained according to the following steps: obtaining multiple sample images, performing multiple affine transformations on each sample image and fusing to obtain a sample fused feature map corresponding to each sample image; calculating a rotation invariance score between the sample fused feature map and a sample original feature map of each sample image; adjusting network parameters of an initial rotation equivalent encoder based on the rotation invariance score corresponding to each sample image to train the target rotation equivalent encoder.

[0010] In a possible implementation, the performing multiple affine transformations on each sample image and fusing to obtain a sample fused feature map corresponding to each sample image comprises: performing multiple affine transformations on each sample image to obtain multiple first sample transformed feature maps corresponding to each sample image; determining a second sample transformed feature map fused with multiple affine transformations according to a transformation weight of each affine transformation and the first sample transformed feature map corresponding to each sample image under each affine transformation; and obtaining the sample fused feature map corresponding to each sample image based on the second sample transformed feature map corresponding to each channel of each sample image.

[0011] In a possible implementation, the target recognition model comprises a plurality of convolution networks, each of which comprises a target rotation equivalence encoder and a convolution unit; and the target recognition model is trained according to the following steps: obtaining a plurality of sample images and actual position information of sample objects in each sample image; inputting each sample image into an initial recognition model to obtain predicted position information of the sample objects in each sample image; determining a comprehensive joint loss of the plurality of convolution networks according to a position loss between the actual position information and the predicted position information of each sample image, and a score loss of a rotation invariance score corresponding to each sample image; and adjusting network parameters of the initial recognition model based on the comprehensive joint loss corresponding to each sample image to obtain the target recognition model.

[0012] In a possible implementation, the determining, based on the rotation invariance score between the query feature vector of the second frame image and the template feature vector of the first frame image, whether the candidate object in the second frame image and the target object in the first frame image are the same object comprises: determining whether the rotation invariance score between the query feature vector and the template feature vector is greater than or equal to a preset threshold; if yes, determining that the candidate object in the second frame image and the target object in the first frame image are the same object; and if no, determining that the candidate object in the second frame image and the target object in the first frame image are not the same object.

[0013] In a second aspect, an embodiment of the present application further provides a target recognition device, which comprises: a transformation module configured to perform a plurality of affine transformations on a first feature map of a first frame image in a target video to obtain a plurality of transformed feature maps; a first determination module configured to calculate a rotation invariance score between each transformed feature map of the first frame image and a second feature map of a second frame image, and determine a target rotation angle corresponding to a highest rotation invariance score; the second frame image is an image located after the first frame image in the target video, and the first frame image and the second frame image are adjacent; a second determination module configured to input a transformed feature map corresponding to the target rotation angle into a target rotation equivalence encoder of a trained target recognition model to obtain a template feature vector, and input the second feature map of the second frame image into the target rotation equivalence encoder to obtain a query feature vector; and a recognition module configured to determine, based on a rotation invariance score between the query feature vector of the second frame image and the template feature vector of the first frame image, whether a candidate object in the second frame image and a target object in the first frame image are the same object.

[0014] Thirdly, embodiments of this application also provide an electronic device, including: a processor, a memory, and a bus, wherein the memory stores machine-readable instructions executable by the processor, and when the electronic device is running, the processor communicates with the memory through the bus, and the machine-readable instructions are executed by the processor to perform the steps of the target recognition method described in the first aspect or any possible implementation of the first aspect.

[0015] Fourthly, embodiments of this application also provide a computer-readable storage medium storing a computer program, which, when executed by a processor, performs the steps of the target recognition method described in the first aspect or any possible implementation of the first aspect.

[0016] The target recognition method, apparatus, electronic device, and storage medium provided in this application embodiment achieve continuous recognition of target objects rotating in three dimensions in the target video based on the actual motion direction (target rotation angle) of the target object in the first frame image of the target video and the rotational equivalence of the features before and after the feature map undergoes affine transformation. Compared with the prior art based on the principle of translation equivalence, which can recognize rotating target objects to a certain extent but has low recognition accuracy and is prone to failure or deviation, this solution can improve the accuracy of target object recognition.

[0017] To make the above-mentioned objectives, features and advantages of this application more apparent and understandable, preferred embodiments are described below in detail with reference to the accompanying drawings. Attached Figure Description

[0018] To more clearly illustrate the technical solutions of the embodiments of this application, the accompanying drawings used in the embodiments will be briefly introduced below. It should be understood that the following drawings only show some embodiments of this application and should not be regarded as a limitation of the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0019] Figure 1 A flowchart of a target recognition method provided in an embodiment of this application is shown;

[0020] Figure 2 A flowchart of the training target rotary encoder provided in an embodiment of this application is shown;

[0021] Figure 3 This illustration shows one of the functional block diagrams of a target recognition device provided in an embodiment of this application;

[0022] Figure 4 This illustration shows a second functional block diagram of a target recognition device provided in an embodiment of this application;

[0023] Figure 5 A schematic diagram of the structure of an electronic device provided in an embodiment of this application is shown. Detailed Implementation

[0024] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. It should be understood that the drawings in this application are for illustrative and descriptive purposes only and are not intended to limit the scope of protection of this application. Furthermore, it should be understood that the schematic drawings are not drawn to scale. The flowcharts used in this application illustrate operations implemented according to some embodiments of this application. It should be understood that the operations in the flowcharts may not be implemented in sequence, and steps without logical contextual relationships may be reversed or implemented simultaneously. In addition, those skilled in the art, guided by the content of this application, may add one or more other operations to the flowcharts, or remove one or more operations from the flowcharts.

[0025] Furthermore, the described embodiments are merely some, not all, of the embodiments of this application. The components of the embodiments of this application described and illustrated herein can typically be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of this application provided in the accompanying drawings is not intended to limit the scope of the claimed application, but merely to illustrate selected embodiments of the application. All other embodiments obtained by those skilled in the art based on the embodiments of this application without inventive effort are within the scope of protection of this application.

[0026] In order to enable those skilled in the art to use the content of this application and combine it with the specific application scenario of "continuously identifying a three-dimensional rotating target object from a target video", the following implementation method is given. For those skilled in the art, the general principles defined herein can be applied to other embodiments and application scenarios without departing from the spirit and scope of this application.

[0027] The methods, apparatus, electronic devices, or computer-readable storage media described in this application can be applied to any scenario requiring target recognition. This application does not limit specific application scenarios, and any scheme using the target recognition methods and apparatus provided in this application is within the protection scope of this application.

[0028] It is worth noting that prior to this application, existing target recognition technologies for 3D rotating objects used the initial state of the target object identified in a video frame as a feature template. Features from the surrounding area of ​​the target object were then extracted to form candidate features, and continuous recognition during target motion was achieved based on the template features and candidate features. This method is based on the principle of translation equivalence. While it can recognize rotating targets to some extent, its accuracy is low, and it is prone to failure or deviation.

[0029] To address the aforementioned issues, this application embodiment performs multiple affine transformations on the first feature map of the first frame image to obtain multiple transformed feature maps. The rotation invariance score between each transformed feature map and the second feature map is used to determine the target rotation angle. Then, the transformed feature map corresponding to the target rotation angle is input into a trained target rotation equivalence encoder to obtain a template feature vector. Furthermore, based on the rotation invariance score between the query feature vector and the template feature vector of the second frame image, it is determined whether a candidate object in the second frame image is the target object. This enables continuous recognition of target objects undergoing 3D rotation in a target video and improves the accuracy of target object recognition.

[0030] To facilitate understanding of this application, the technical solutions provided in this application will be described in detail below with reference to specific embodiments.

[0031] Figure 1 This is a flowchart illustrating a target recognition method provided in an embodiment of this application. Figure 1 As shown in the embodiments of this application, the target identification method includes the following steps:

[0032] S101: Perform various affine transformations on the first feature map of the first frame image in the target video to obtain multiple transformed feature maps.

[0033] In specific implementation, during the continuous recognition of a 3D rotating target object in a target video, the first frame image of the target video is first acquired, and features are extracted from the first frame image to obtain a first feature map of the first frame image. Then, the target object is identified based on the first feature map of the first frame image. Furthermore, various affine transformations are performed on the first feature map of the first frame image to obtain multiple transformed feature maps corresponding to the first frame image.

[0034] Here, if no frame of the target video has been identified yet, the first frame is the first frame of the target video; if some of the earlier frames of the target video have been identified, the first frame is the middle frame of the target video, and the middle frame is the last frame in which the target object has been identified. That is, the frames in the target video after the middle frame have not yet been identified.

[0035] It should be noted that affine transformation, also known as affine mapping, refers to the geometric transformation of a vector space into another vector space through a linear transformation followed by a translation. Affine transformations include rotation, scaling, and translation, all of which are achieved through vectors, specifically vector multiplication. This can be accomplished by first processing the translation vector and then the rotation vector to achieve the desired translation and rotation of an object to its final position. In this application, affine transformations primarily include rotation and scaling. Different affine transformations applied to the first feature map of the first frame will result in different rotation angles and scaling ratios.

[0036] Here, an affine transformation formula for the first feature map is: ψ r,θ (T)=τ(r)e iθ T; where r represents the scaling magnitude, τ(r) represents the scaling ratio, e is the natural base, i represents the imaginary unit, θ represents the rotation angle, and T represents the first feature map of the first frame image.

[0037] Here, "different rotation angles" represents the possible rotation angles of the target object in the first frame image. Then, by performing an affine transformation on the first feature map of the first frame image according to these rotation angles, a feature map of the target object after moving according to these rotation angles can be obtained, i.e., a transformed feature map. These rotation angles can be determined based on the target object's direction of motion. Specifically, step S101 involves performing various affine transformations on the first feature map of the first frame image in the target video to obtain multiple transformed feature maps, including:

[0038] Step S1011: Based on the change information between the first frame image and the second frame image, predict the target motion direction of the target object in the first frame image.

[0039] In practice, for the target object to be tracked, after identifying the target object from the first frame image, the tendency of the target object to move in the first frame image, i.e., the target motion direction, can be determined based on the motion trend of the target object in the first frame image. Specifically, the target motion direction of the target object in the first frame image can be predicted based on the change information between the first and second frames image. Here, the target motion direction is estimated and cannot be completely equivalent to the actual motion direction.

[0040] Step S1012: Perform multiple affine transformations on the first feature map of the first frame image according to multiple rotation angles associated with the direction of motion of the target, to obtain multiple transformed feature maps.

[0041] In a specific implementation, by determining the target motion direction of the target object in the first frame image, the rotation angle associated with the target motion direction can be determined, that is, the possible rotation angle in the target motion direction. Then, according to each determined rotation angle, an affine transformation is performed on the first feature map of the first frame image to obtain the transformed feature map of the first frame image at each rotation angle.

[0042] S102: Calculate the rotation invariance scores between each transformation feature map of the first frame image and the second feature map of the second frame image, and determine the target rotation angle corresponding to the highest rotation invariance score.

[0043] The second frame image is an image in the target video that is located after the first frame image, and the first frame image and the second frame image are adjacent to each other.

[0044] In specific implementation, the second feature map of the second frame image is also the feature map corresponding to the target object in the first frame image after actual motion. Specifically, multiple transformed feature maps are obtained by performing different affine transformations on the first feature map of the first frame image. The rotation invariance scores between each transformed feature map and the second feature map of the second frame image are calculated, resulting in multiple rotation invariance scores. The rotation invariance score with the highest score is determined from the multiple obtained rotation invariance scores, and the rotation angle corresponding to the highest rotation invariance score is determined and determined as the target rotation angle. Here, the target rotation angle is the estimated three-dimensional rotation Euler angle (θ) in the x-axis, y-axis, and z-axis directions. x ,θ y ,θ z ).

[0045] It should be noted that, for any transformed feature map ψ(T1) corresponding to the first frame image, the rotation invariance score s between this transformed feature map ψ(T1) and the second feature map of the second frame image is formulated as follows: Where T1 is the first feature map of the first frame image, ψ(T1) is the transformed feature map obtained after performing an affine transformation on the first feature map T1, f(x) is the feature vector obtained by encoding x through the target rotary encoder, "·" represents the dot product operation between feature vectors, and "||||" represents the modulus of the feature vector.

[0046] Here, if the target object actually moves according to a certain rotation angle, then the transform feature map corresponding to that rotation angle and the second feature map of the second frame image are essentially the same feature map. In this case, the rotation invariance score between the transform feature map of that rotation angle and the second feature map is the highest; that is, that rotation angle is the target rotation angle. If the target object does not actually move according to a certain rotation angle, then the transform feature map corresponding to that rotation angle is different from the second feature map of the second frame image. In this case, the rotation invariance score between the transform feature map of that rotation angle and the second feature map is not the highest.

[0047] S103: Input the transformation feature map corresponding to the target rotation angle into the target rotation equivalent encoder of the trained target recognition model to obtain the template feature vector, and input the second feature map of the second frame image into the target rotation equivalent encoder to obtain the query feature vector.

[0048] In practice, after determining the actual direction of motion of the target object, that is, the target rotation angle, the transformation feature map corresponding to the target rotation angle is input into a pre-trained target rotation equivalent encoder for encoding to obtain the template feature vector of the first frame image. This template feature vector is used as the template feature vector of the target object. In this way, the target object can be identified from the second frame image using this template feature vector.

[0049] It should be noted that in mathematics, given a function defined in an inner product space, if the parameter values ​​of the function may change with any rotation, but the numerical value of the function remains unchanged, this property is called rotational invariance or rotational symmetry, because the function is symmetric with respect to rotation. For example, suppose the origin of the xyz-reference frame is fixed, and the xyz-reference frame is rotated arbitrarily, while the numerical value of the function f(x, y, z) remains unchanged. Therefore, the function f(x, y, z) is invariant with respect to any rotation, or is symmetric with respect to any rotation. Here, based on the above rotational invariance, this application proposes a target rotational equivalence encoder in a neural network. This target rotational equivalence encoder is an encoder based on the rotational equivalence principle using a multi-convolutional architecture. Before and after an affine transformation, the feature vector output by the target rotational equivalence encoder is the same, and this feature vector represents the features of the target object.

[0050] Here, the target rotation equivalent encoder f(x) in this application is defined as follows: for any element x in the domain, for any affine transformation ψ, f(ψ(x)) = ψ(f(x)); where the affine transformation ψ represents the affine function of rotation and scaling.

[0051] In this application, if x is the target feature map T of the target frame image in the target video, then f(ψ(T)) = ψ(f(T)); ψ(T) is the transformed feature map of the target feature map T after a certain affine transformation, f(T) is the first feature vector obtained by encoding the target feature map T through the target rotation encoder, f(ψ(T)) is the second feature vector obtained by encoding the transformed feature map ψ(T) through the target rotation encoder, and ψ(f(T)) is the third feature vector obtained by the first feature vector after the affine transformation. The second and third feature vectors are equal. That is, the feature vector obtained by inputting an original feature map into the target rotation equivalent encoder for encoding and then performing an affine transformation is the same as the feature vector obtained by performing the affine transformation on the original feature map and then encoding it through the target rotation encoder. In other words, the output obtained by inputting the original feature map into the target rotation equivalent encoder before and after the affine transformation is consistent.

[0052] The training process of the target rotation equivalent encoder is explained below. Figure 2 This is a flowchart illustrating the training target rotation equivalent encoder provided in an embodiment of this application. Figure 2 As shown in the embodiments of this application, the process of training a target rotation equivalent encoder includes the following steps:

[0053] S201: Acquire multiple sample images, perform various affine transformations on each sample image and fuse them to obtain the sample fusion feature map corresponding to each sample image.

[0054] In practice, the target rotational equivalent encoder can be obtained by training the initial rotational equivalent encoder with multiple sample images. Specifically, various affine transformations can be pre-set, and each sample image can undergo multiple different affine transformations. In this way, the initial rotational equivalent encoder can learn different affine transformations, thereby obtaining a sample fusion feature map corresponding to each sample image. The sample fusion feature map corresponding to each sample image fuses multiple different affine transformations.

[0055] Here, during the training of the initial rotational equivalent encoder, the transformation weights of each affine transformation can be continuously adjusted to ensure that the trained target rotational equivalent encoder meets the training accuracy requirements. Each affine transformation can correspond to different rotation angles and scaling ratios. The rotation angle can be any angle within (0, 2π), and the scaling ratio can be any value within (0, m). The transformation weights for different affine transformations can be the same or different. In this way, the initial rotational equivalent encoder can learn the affine transformations corresponding to different rotation angles. Furthermore, each sample image corresponds to feature maps with multiple channels, and affine transformations can be performed individually on the feature maps corresponding to each channel.

[0056] The following describes the process of generating the sample fusion feature map corresponding to each sample image. That is, in step S201, multiple affine transformations are performed on each sample image and fused to obtain the sample fusion feature map corresponding to each sample image. This includes the following steps:

[0057] Step S2011: Perform multiple affine transformations on each sample image to obtain multiple first sample transformation feature maps corresponding to each sample image.

[0058] In practice, multiple affine transformations can be pre-set, and initial affine weights can be assigned to each affine transformation. These initial affine weights are continuously adjusted during training. Here, the affine transformation formula is: Where r represents the scaling magnitude, τ(r) represents the scaling ratio, e is the natural base, i represents the imaginary unit, θ represents the rotation angle, T represents the original feature map of the sample image, and g i Represents a specific transformation, and g∈G={g1,…,g i ,…,g m Let G be the set of all m feature transformations applied to the original feature map T of the samples. The original feature map T of the sample image under a specific transformation g iThe first sample transformation feature map was obtained afterward.

[0059] Step S2012: Based on the transformation weights of each affine transformation and the first sample transformation feature map corresponding to each sample image under each affine transformation, determine the second sample transformation feature map that integrates multiple affine transformations.

[0060] In practice, the original feature map T of each sample image is subjected to all affine transformations ψ and then weighted and fused to form a unified transformation operation: Where Ψ(T) represents the transformed feature map of the second sample. For affine transformation The transformation weights, where m represents the number of affine transformations.

[0061] Step S2013: Based on the second sample transformation feature map corresponding to each channel of each sample image, obtain the sample fusion feature map corresponding to each sample image.

[0062] In practice, each sample image generates C channels of original sample feature maps {T1, T2, ... T}. C Based on the second sample transformation feature map corresponding to each channel of each sample image, the sample fusion feature map corresponding to each sample image is obtained as follows:

[0063] S202: Calculate the rotation invariance score between the sample fused feature map and the original feature map of each sample image.

[0064] In specific implementation, the original feature map of each sample image is input into the initial rotation equivalence encoder for encoding to obtain f(y(I)), and the fused feature map of each sample image is input into the initial rotation equivalence encoder for encoding to obtain f(y(I|Ψ)). Furthermore, the rotation invariance score between the fused feature map and the original feature map of each sample image is calculated using the following formula: In this context, "·" represents the dot product operation between feature vectors, and "||" represents the magnitude of the feature vector. The rotation invariance score is used to characterize whether two feature maps originate from the same target object.

[0065] S203: Based on the rotation invariance scores corresponding to each sample image, adjust the network parameters of the initial rotation equivalent encoder to train the target rotation equivalent encoder.

[0066] In practice, following the steps outlined above, various affine transformations are performed on each sample image, and the rotation invariance score between the original feature map and the fused feature map of each sample image is calculated. The rotation invariance score of each sample image is maximized to adjust the network parameters of the initial encoder, thereby training the target rotation-equivalent encoder.

[0067] It should also be noted that by inputting the target video into the target recognition model, the model outputs the positional information of the target object in each frame of the video. This allows for the continuous identification of target objects undergoing 3D rotation from the target video. The target recognition model is a multi-convolutional architecture model, such as a Convolutional Neural Network (CNN) model.

[0068] Here, the target recognition model includes multiple convolutional networks, each including a target rotation equivalent encoder and a convolutional unit. The target recognition model is trained according to the following steps: acquiring multiple sample images and the actual position information of the sample objects in each sample image; inputting each sample image into the initial recognition model to obtain the predicted position information corresponding to the sample objects in each sample image; determining the comprehensive joint loss of the multiple convolutional networks based on the positional loss between the actual position information and the predicted position information corresponding to each sample image, and the score loss of the rotation invariance score corresponding to each sample image; and adjusting the network parameters of the initial recognition model based on the comprehensive joint loss corresponding to each sample image to obtain the target recognition model.

[0069] Here, the target recognition model includes n convolutional networks A = {a1, a2, ... a...} n For a single target object M and n convolutional networks, the combined joint loss B of the target recognition model is:

[0070] B = β1loss(a1(M|Ψ,f)) + ... + β j loss(a j (M|Ψ,f))+…+β n loss(a n (M|Ψ,f)), where loss is the loss function of the convolutional network, j is a constant, j∈[1,n], and the joint loss corresponding to the j-th convolutional network is loss(a j (M|Ψ,f)), the joint loss corresponding to the j-th convolutional network consists of position and score losses, β j Let be the weights of the loss value for the j-th convolutional network. During the training of the initial recognition model, the comprehensive joint loss is minimized to obtain the target recognition model.

[0071] S104: Based on the rotation invariance score between the query feature vector of the second frame image and the template feature vector of the first frame image, determine whether the candidate object in the second frame image and the target object in the first frame image are the same object.

[0072] In practice, the rotation invariance score can be calculated using the template feature vector of the first frame image and the query feature vector of the second frame image. The higher the rotation invariance score, the greater the probability that the candidate object identified in the second frame image is the same object as the target object identified in the first frame image.

[0073] Here, step S104, which determines whether the candidate object in the second frame image and the target object in the first frame image are the same object based on the rotation invariance score between the query feature vector of the second frame image and the template feature vector of the first frame image, includes the following steps:

[0074] Step S1041: Determine whether the rotation invariance score between the query feature vector and the template feature vector is greater than or equal to a preset threshold.

[0075] In practice, after calculating the rotation invariance score between the template feature vector of the first frame image and the query feature vector of the second frame image, it is determined whether the rotation invariance score is greater than or equal to a preset threshold. Here, the preset threshold is set in advance based on the accuracy requirements of the target rotary encoder.

[0076] Step S1042: If yes, then determine that the candidate object identified in the second frame image is the same object as the target object identified in the first frame image.

[0077] In specific implementation, if the rotation invariance score between the template feature vector of the first frame image and the query feature vector of the second frame image is greater than or equal to a preset threshold, then the candidate object identified in the second frame image is determined to be the same object as the target object identified in the first frame image, thereby realizing the identification of the target object from the second frame image. In this way, continuous identification of three-dimensional rotating target objects in the target video can be achieved.

[0078] Step S1043: If not, then determine that the candidate object in the second frame image is not the same object as the target object in the first frame image.

[0079] In practice, if the rotation invariance score between the template feature vector of the first frame image and the query feature vector of the second frame image is less than a preset threshold, then it is determined that the candidate object identified in the second frame image is not the same object as the target object identified in the first frame image.

[0080] It should be noted that this application proposes a target recognition method based on the principle of rotational equivalence using a multi-convolutional architecture. Rotational equivalence is established based on the concept that target feature groups are equivalent during translation. At the same time, the action of the target object in the target video is continuous. Based on the target motion direction of the target object identified in the first frame of the target video, the estimated rotation direction is used to effectively enhance the robustness of target feature recognition and similarity matching ability, thereby achieving high-performance and continuous target recognition of the target object during three-dimensional rotation.

[0081] Here, this application fully considers the characteristic that the spatial feature groups of each position of the target object are still equivalent when the target object is rotated in three dimensions, establishes a rotation equivalence algorithm into the convolutional network architecture, and judges the rotation direction based on video continuity, which makes up for the insufficient recognition ability of traditional target recognition technology when the target object is rotated in three dimensions, and greatly improves the recognition accuracy of image target recognition technology in video content.

[0082] In this embodiment, multiple transformed feature maps are obtained by performing various affine transformations on the first feature map of the first frame image. The target rotation angle is determined by the rotation invariance score between each transformed feature map and the second feature map. Then, the transformed feature map corresponding to the target rotation angle is input into a trained target rotation equivalent encoder to obtain a template feature vector. Furthermore, based on the rotation invariance score between the query feature vector and the template feature vector of the second frame image, it is determined whether the candidate object in the second frame image is the target object. In this way, continuous recognition of target objects undergoing three-dimensional rotation in the target video can be achieved, and the accuracy of target object recognition can be improved.

[0083] Based on the same application concept, this application also provides a target recognition device corresponding to the target recognition method provided in the above embodiments. Since the principle of the device in this application is similar to the target recognition method in the above embodiments of this application, the implementation of the device can refer to the implementation of the method, and the repeated parts will not be described again.

[0084] like Figure 3 , Figure 4 As shown, Figure 3 This is one of the functional block diagrams of a target recognition device 300 provided in an embodiment of this application. Figure 4 This is a second functional block diagram of a target recognition device 300 provided in an embodiment of this application.

[0085] In the embodiments of this application, such as Figure 3As shown, the target recognition device 300 includes: a transformation module 310, used to perform multiple affine transformations on a first feature map of a first frame image in a target video to obtain multiple transformed feature maps; a first determination module 320, used to calculate the rotation invariance scores between each transformed feature map of the first frame image and a second feature map of a second frame image, and determine the target rotation angle corresponding to the highest rotation invariance score; the second frame image is an image in the target video located after the first frame image, and the first frame image and the second frame image are adjacent; a second determination module 330, used to input the transformed feature map corresponding to the target rotation angle into a target rotation equivalence encoder of a trained target recognition model to obtain a template feature vector, and input the second feature map of the second frame image into the target rotation equivalence encoder to obtain a query feature vector; and a recognition module 340, used to determine whether a candidate object in the second frame image and a target object in the first frame image are the same object based on the rotation invariance scores between the query feature vector of the second frame image and the template feature vector of the first frame image.

[0086] In one possible implementation, such as Figure 4 As shown, the transformation module 310 includes: a determination unit 311, used to predict the target motion direction of the target object in the first frame image based on the change information between the first frame image and the second frame image; and a transformation unit 312, used to perform multiple affine transformations on the first feature map of the first frame image according to multiple rotation angles associated with the target motion direction, to obtain multiple transformed feature maps.

[0087] In one possible implementation, such as Figure 4 As shown, the second determining module 330 includes a training unit 331; the training unit 331 is used to train the target rotation equivalent encoder according to the following steps: acquiring multiple sample images, performing multiple affine transformations on each sample image and fusing them to obtain a sample fusion feature map corresponding to each sample image; calculating the rotation invariance score between the sample fusion feature map and the original feature map of each sample image; and adjusting the network parameters of the initial rotation equivalent encoder based on the rotation invariance scores corresponding to each sample image to train the target rotation equivalent encoder.

[0088] In one possible implementation, such as Figure 4As shown, the training unit 331 is specifically used to perform multiple affine transformations on each sample image and fuse them according to the following steps: performing multiple affine transformations on each sample image to obtain multiple first sample transformation feature maps corresponding to each sample image; determining a second sample transformation feature map fused with multiple affine transformations based on the transformation weight of each affine transformation and the first sample transformation feature map corresponding to each sample image under each affine transformation; and obtaining a sample fusion feature map corresponding to each sample image based on the second sample transformation feature map corresponding to each channel of each sample image.

[0089] In one possible implementation, such as Figure 4 As shown, the target recognition model includes multiple convolutional networks, each convolutional network including a target rotation equivalent encoder and a convolutional unit; the target recognition device 300 also includes a training module 350; the training module 350 is used to train the target recognition model according to the following steps: acquiring multiple sample images and the actual position information of sample objects in each sample image; inputting each sample image into an initial recognition model to obtain the predicted position information corresponding to the sample object in each sample image; determining the comprehensive joint loss of the multiple convolutional networks based on the position loss between the actual position information and the predicted position information corresponding to each sample image, and the score loss of the rotation invariance score corresponding to each sample image; adjusting the network parameters of the initial recognition model based on the comprehensive joint loss corresponding to each sample image to obtain the target recognition model.

[0090] In one possible implementation, such as Figure 4 As shown, the recognition module 340 is used to determine whether the candidate object in the second frame image and the target object in the first frame image are the same object according to the following steps: determining whether the rotation invariance score between the query feature vector and the template feature vector is greater than or equal to a preset threshold; if yes, then determining that the candidate object in the second frame image and the target object in the first frame image are the same object; if no, then determining that the candidate object in the second frame image and the target object in the first frame image are not the same object.

[0091] In this embodiment, the transformation module 310 performs various affine transformations on the first feature map of the first frame image to obtain multiple transformed feature maps. The rotation invariance score between each transformed feature map and the second feature map is determined by the first determination module 320. Then, the transformed feature map corresponding to the target rotation angle is input into a trained target rotation equivalence encoder by the second determination module 330 to obtain a template feature vector. Furthermore, the recognition module 340 determines whether a candidate object in the second frame image is the target object based on the rotation invariance score between the query feature vector and the template feature vector of the second frame image. This enables continuous recognition of target objects undergoing three-dimensional rotation in the target video and improves the accuracy of target object recognition.

[0092] Based on the same application concept, see [link / reference] Figure 5 The diagram shows a schematic of an electronic device 500 provided in an embodiment of this application. It includes a processor 510, a memory 520, and a bus 530. The memory 520 stores machine-readable instructions executable by the processor 510. When the electronic device 500 is running, the processor 510 and the memory 520 communicate through the bus 530. The machine-readable instructions are executed by the processor 510 to perform the steps of any of the target recognition methods described in the above embodiments.

[0093] Specifically, when the machine-readable instructions are executed by the processor 510, they can perform the following processing: performing multiple affine transformations on the first feature map of the first frame image in the target video to obtain multiple transformed feature maps; calculating the rotation invariance scores between each transformed feature map of the first frame image and the second feature map of the second frame image, and determining the target rotation angle corresponding to the highest rotation invariance score; the second frame image is an image in the target video located after the first frame image, and the first frame image and the second frame image are adjacent; inputting the transformed feature map corresponding to the target rotation angle into the target rotation equivalence encoder of the trained target recognition model to obtain a template feature vector, and inputting the second feature map of the second frame image into the target rotation equivalence encoder to obtain a query feature vector; based on the rotation invariance score between the query feature vector of the second frame image and the template feature vector of the first frame image, determining whether the candidate object in the second frame image and the target object in the first frame image are the same object.

[0094] Based on the same concept, embodiments of this application also provide a computer-readable storage medium storing a computer program, which, when run by a processor, executes the steps of the target recognition method provided in the above embodiments.

[0095] Specifically, the storage medium can be a general-purpose storage medium, such as a portable disk or hard disk. When the computer program on the storage medium is run, it can execute the above-mentioned target recognition method, which can realize the continuous recognition of target objects that are rotated in three dimensions in the target video and improve the accuracy of target object recognition.

[0096] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems and devices described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here. In the several embodiments provided in this application, it should be understood that the disclosed systems, devices, and methods can be implemented in other ways. The device embodiments described above are merely illustrative. For example, the division of units is only a logical functional division; in actual implementation, there may be other division methods. Furthermore, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Another point is that the displayed or discussed mutual coupling or direct coupling or communication connection may be through some communication interfaces; the indirect coupling or communication connection of devices or units may be electrical, mechanical, or other forms.

[0097] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0098] In addition, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0099] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a processor-executable, non-volatile, computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0100] The above are merely specific embodiments of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A target recognition method characterized by, The target recognition method comprises: performing multiple affine transformations on a first feature map of a first frame image in a target video to obtain multiple transformed feature maps; calculating a rotation invariance score between each transformed feature map of the first frame image and a second feature map of a second frame image, and determining a target rotation angle corresponding to a highest rotation invariance score; the second frame image is an image located after the first frame image in the target video, and the first frame image and the second frame image are adjacent; inputting the transformed feature map corresponding to the target rotation angle into a target rotation equivalent encoder of a trained target recognition model to obtain a template feature vector, and inputting the second feature map of the second frame image into the target rotation equivalent encoder to obtain a query feature vector; determining whether a candidate object in the second frame image and a target object in the first frame image are the same object based on a rotation invariance score between the query feature vector of the second frame image and the template feature vector of the first frame image; The target rotation equivalent encoder is trained according to the following steps: obtaining multiple sample images, performing multiple affine transformations on each sample image and fusing to obtain a sample fusion feature map corresponding to each sample image; calculating a rotation invariance score between the sample fusion feature map and a sample original feature map of each sample image; adjusting network parameters of an initial rotation equivalent encoder based on the rotation invariance score corresponding to each sample image to train the target rotation equivalent encoder.

2. The object recognition method of claim 1, wherein, The multiple affine transformations performed on the first feature map of the first frame image in the target video to obtain the multiple transformed feature maps comprise: predicting a target motion direction of the target object in the first frame image based on change information between the first frame image and the second frame image; performing multiple affine transformations on the first feature map of the first frame image according to multiple rotation angles associated with the target motion direction to obtain the multiple transformed feature maps.

3. The object recognition method of claim 1, wherein, The multiple affine transformations performed on each sample image and the fusing to obtain the sample fusion feature map corresponding to each sample image comprise: performing multiple affine transformations on each sample image to obtain multiple first sample transformed feature maps corresponding to each sample image; determining a second sample transformed feature map fused with the multiple affine transformations according to a transformation weight of each affine transformation and the first sample transformed feature map corresponding to each sample image under each affine transformation; obtaining the sample fusion feature map corresponding to each sample image based on the second sample transformed feature map corresponding to each channel of each sample image.

4. The object recognition method of claim 1, wherein The target recognition model comprises multiple convolutional networks, and each convolutional network comprises a target rotation equivalent encoder and a convolutional unit; The target recognition model is trained according to the following steps: obtaining multiple sample images and actual position information of a sample object in each sample image; inputting each sample image into an initial recognition model to obtain predicted position information of the sample object in each sample image; determine the comprehensive joint loss of the plurality of convolutional networks according to the position loss between the actual position information and the predicted position information corresponding to each sample image, and the score loss of the rotation invariance score corresponding to each sample image; adjust the network parameters of the initial identification model based on the comprehensive joint loss corresponding to each sample image to obtain the target identification model.

5. The object recognition method of claim 1, wherein, The determination of whether the candidate object in the second frame image and the target object in the first frame image are the same object based on the rotation invariance score between the query feature vector of the second frame image and the template feature vector of the first frame image includes: determining whether the rotation invariance score between the query feature vector and the template feature vector is greater than or equal to a preset threshold value; if yes, determining that the candidate object in the second frame image and the target object in the first frame image are the same object; if no, determining that the candidate object in the second frame image and the target object in the first frame image are not the same object.

6. A target recognition device, characterized by The target identification device includes: a transformation module configured to perform a plurality of affine transformations on a first feature map of a first frame image in a target video to obtain a plurality of transformed feature maps; a first determination module configured to calculate rotation invariance scores between each transformed feature map of the first frame image and a second feature map of a second frame image, and determine a target rotation angle corresponding to a highest score rotation invariance score; the second frame image is an image located after the first frame image in the target video, and the first frame image and the second frame image are adjacent; a second determination module configured to input a transformed feature map corresponding to the target rotation angle into a target rotation equivalence encoder of a trained target identification model to obtain a template feature vector, and input a second feature map of the second frame image into the target rotation equivalence encoder to obtain a query feature vector; an identification module configured to determine whether a candidate object in the second frame image and a target object in the first frame image are the same object based on a rotation invariance score between a query feature vector of the second frame image and a template feature vector of the first frame image; The second determination module trains the target rotation equivalence encoder according to the following steps: obtain a plurality of sample images, perform a plurality of affine transformations on each sample image, and fuse to obtain a sample fused feature map corresponding to each sample image; calculate a rotation invariance score between a sample fused feature map and a sample original feature map of each sample image; adjust network parameters of an initial rotation equivalence encoder based on the rotation invariance score corresponding to each sample image to train the target rotation equivalence encoder.

7. The object recognition apparatus according to claim 6, characterized in that, The transformation module includes: a determination unit configured to predict a target motion direction of the target object in the first frame image based on change information between the first frame image and the second frame image; a transformation unit configured to perform a plurality of affine transformations on a first feature map of the first frame image according to a plurality of rotation angles associated with the target motion direction to obtain a plurality of transformed feature maps.

8. An electronic device, comprising: includes: A processor, a memory, and a bus, the memory storing machine readable instructions executable by the processor, when the electronic device is running, the processor and the memory communicating through the bus, the machine readable instructions being executed by the processor to perform the steps of the target identification method of any one of claims 1 to 5.

9. A computer-readable storage medium, characterized in that, A computer readable storage medium storing a computer program, the computer program being executed by the processor to perform the steps of the target identification method of any one of claims 1 to 5.

Images