Object recognition method and electronic equipment

Abstract

The invention belongs to the technical field of artificial intelligence, and particularly relates to an object recognition method and electronic equipment. The method comprises the steps of obtaining a to-be-recognized image, and performing edge pixel detection on the to-be-recognized image to obtain edge pixel points and non-edge pixel points in the to-be-recognized image; the edge pixel points are pixel points located in the edge area of the content object in the to-be-recognized image, and the non-edge pixel points are pixel points located in the non-edge area of the content object in the to-be-recognized image; selecting a target pooling algorithm matched with the pooling area from a plurality of candidate pooling algorithms according to the distribution proportion of the edge pixel points in the pooling area; the pooling region is a local image region acquired by moving a pooling kernel with a preset size on the to-be-identified image; performing pooling processing on the pooling region according to a target pooling algorithm to obtain a feature map corresponding to the to-be-recognized image; and positioning the distribution position of the to-be-recognized target object in the to-be-recognized image according to the feature map. The object identification and tracking accuracy can be improved.

Description

Technical Field

[0001] The present application belongs to the field of artificial intelligence technology, and specifically relates to an object recognition method, an object recognition device, a computer-readable medium, an electronic device, and a computer program product. Background Art

[0002] Target tracking refers to estimating the state of the target object in subsequent video frames based on the initial state of the target object in a given video frame. As one of the most challenging areas of computer vision, it is widely used in scenarios such as face recognition, intelligent monitoring, and autonomous driving. Currently, target tracking is mainly divided into traditional tracking algorithms based on correlation filtering and tracking algorithms based on deep learning.

[0003] Traditional target tracking methods usually rely on manually designed features (grayscale features, color features, shape features, etc.) and model update strategies. In order to design these manual features and update the model, the overall tracking algorithm is very complex and overly dependent on manually designed features. In addition, the representation ability of these features is limited (the number of features is limited and they are all shallow features), and they are often limited in effect when facing complex scenes.

[0004] The target tracking method based on deep learning relies on convolutional neural networks (CNN) and a large amount of training data. CNN can extract features from images layer by layer through structures such as convolutional layers and pooling layers. From low-level edge and texture features to high-level shape and semantic features, CNN can form a hierarchical feature representation. This hierarchical feature extraction enables CNN to capture more complex patterns and structures in images. After training with a large amount of data, it has stronger representation capabilities and can more effectively handle tracking problems in complex scenes.

[0005] In deep learning-based target tracking methods, image features are generally extracted through convolution operations. Due to the locality of convolution operations, each convolution kernel only processes the local area it covers, resulting in poor ability to capture global context information. In order to increase the receptive field of image feature extraction, a pooling operation can be added after the convolution operation. However, the pooling operation is prone to losing image detail information and being disturbed by noise or outliers, making the image blurred, resulting in poor accuracy in object recognition and tracking. Summary of the invention

[0006] The present application provides an object recognition method, an object recognition device, a computer-readable medium, an electronic device, and a computer program product, the purpose of which is to improve the accuracy of object recognition and tracking.

[0007] Other features and advantages of the present application will become apparent from the following detailed description, or may be learned in part by the practice of the present application.

[0008] According to one aspect of an embodiment of the present application, there is provided an object recognition method, the method comprising: acquiring an image to be recognized, and performing edge pixel detection on the image to be recognized to obtain edge pixel points and non-edge pixel points in the image to be recognized; the edge pixel points are pixel points located in an edge area of ​​a content object in the image to be recognized, and the non-edge pixel points are pixel points located in a non-edge area of ​​the content object in the image to be recognized; according to the distribution ratio of the edge pixel points in a pooling area, selecting a target pooling algorithm that matches the pooling area from a plurality of candidate pooling algorithms; the pooling area is a local image area collected by moving a pooling kernel with a preset size on the image to be recognized; performing pooling processing on the pooling area according to the target pooling algorithm to obtain a feature map corresponding to the image to be recognized; and locating the distribution position of the target object to be recognized in the image to be recognized according to the feature map.

[0009] According to one aspect of an embodiment of the present application, there is provided an object recognition device, the device comprising:

[0010] a detection module configured to acquire an image to be identified, and perform edge pixel detection on the image to be identified to obtain edge pixel points and non-edge pixel points in the image to be identified; the edge pixel points are pixel points located in an edge area of ​​a content object in the image to be identified, and the non-edge pixel points are pixel points located in a non-edge area of ​​the content object in the image to be identified;

[0011] A selection module is configured to select a target pooling algorithm that matches the pooling area from a plurality of candidate pooling algorithms according to the distribution ratio of the edge pixels in the pooling area; the pooling area is a local image area collected by moving a pooling kernel with a preset size on the image to be identified;

[0012] A pooling module is configured to perform pooling processing on the pooling area according to the target pooling algorithm to obtain a feature map corresponding to the image to be recognized;

[0013] The positioning module is configured to locate the distribution position of the target object to be identified in the image to be identified according to the feature map.

[0014] In some embodiments of the present application, based on the above technical solution, the selection module includes:

[0015] A first statistical module is configured to count the distribution ratio of the edge pixels in the image to be identified to obtain the overall area ratio of the edge pixels;

[0016] A second statistical module is configured to count the distribution ratio of the edge pixels in the pooling area to obtain the local area ratio of the edge pixels;

[0017] The algorithm selection module is configured to select a target pooling algorithm that matches the pooling area from a plurality of candidate pooling algorithms based on a numerical relationship between the overall area ratio and the local area ratio.

[0018] In some embodiments of the present application, based on the above technical solution, the candidate pooling algorithms include a maximum pooling algorithm, an average pooling algorithm and a hybrid pooling algorithm, and the hybrid pooling algorithm is a pooling algorithm formed by combining the maximum pooling algorithm and the average pooling algorithm; the algorithm selection module is further configured as: when the local area proportion is greater than the overall area proportion, the maximum pooling algorithm is selected as the target pooling algorithm matching the pooling area; when the local area proportion is less than the overall area proportion and greater than zero, the hybrid pooling algorithm is selected as the target pooling algorithm matching the pooling area; when the local area proportion is equal to zero, the average pooling algorithm is selected as the target pooling algorithm matching the pooling area.

[0019] In some embodiments of the present application, based on the above technical solution, the pooling module is further configured to: perform pooling processing on the pooling area according to the maximum pooling algorithm to obtain a first pooling eigenvalue; perform pooling processing on the pooling area according to the uniform pooling algorithm to obtain a second pooling eigenvalue; use the local area proportion as a weighting coefficient, perform a weighted operation on the first pooling eigenvalue and the second pooling eigenvalue, and obtain a mixed pooling eigenvalue of the pooling area.

[0020] In some embodiments of the present application, based on the above technical solution, the pooling module is further configured to: perform pooling processing on the pooling area according to the maximum pooling algorithm to obtain a first pooling eigenvalue; perform pooling processing on the pooling area according to the uniform pooling algorithm to obtain a second pooling eigenvalue; and take the local area proportion as the selection probability, and randomly select an eigenvalue from the first pooling eigenvalue and the second pooling eigenvalue as the mixed pooling eigenvalue of the pooling area.

[0021] In some embodiments of the present application, based on the above technical solution, the positioning module includes:

[0022] A template acquisition module is configured to acquire a feature template corresponding to a target object to be identified;

[0023] a feature fusion module configured to perform feature fusion processing on the feature template and the candidate region in the feature map to obtain a response map of the candidate region, wherein the response map is used to represent the similarity between the candidate region and the feature template;

[0024] The object positioning module is configured to locate the candidate area with the highest similarity to the feature template as the distribution position of the target object in the image to be identified according to the response map.

[0025] In some embodiments of the present application, based on the above technical solution, the feature template includes a pre-configured static template and a dynamic template dynamically updated during the object recognition process; the feature fusion module is further configured to: perform feature fusion processing on the static template and the candidate area in the feature map to obtain a first fusion feature with spatial information; perform feature fusion processing on the dynamic template and the candidate area in the feature map to obtain a second fusion feature with time information; perform convolution operation on the first fusion feature and the second fusion feature to obtain a response map of the candidate area.

[0026] In some embodiments of the present application, based on the above technical solution, the object recognition device also includes an update module, which is configured to: crop target features from the feature map according to the distribution position of the target object in the image to be recognized; obtain the feature similarity between the target feature and the static template; and update the dynamic template according to the feature similarity.

[0027] In some embodiments of the present application, based on the above technical solution, the feature similarity includes the similarity value at each pixel point; the dynamic template is updated according to the feature similarity, and the update module is further configured to: use the similarity value at each similar point as a weight, perform weighted operation on each pixel value of the dynamic template, and obtain an updated dynamic template.

[0028] According to one aspect of an embodiment of the present application, a computer-readable medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the object recognition method in the above technical solution is implemented.

[0029] According to one aspect of an embodiment of the present application, an electronic device is provided, comprising: a processor; and a memory for storing executable instructions of the processor; wherein the processor is configured to execute the executable instructions to implement an object recognition method as in the above technical solution.

[0030] According to one aspect of an embodiment of the present application, a computer program product is provided, including a computer program, and when the computer program is executed by a processor, the object recognition method in the above technical solution is implemented.

[0031] In the technical solution provided in the embodiment of the present application, the accuracy and efficiency of image recognition are effectively improved through fine edge pixel detection and intelligent pooling algorithm selection. First, the edge and non-edge areas of the content object in the image are distinguished through edge pixel detection, which provides accurate area division for subsequent processing. Then, according to the distribution ratio of edge pixels in the pooling area, the most appropriate pooling algorithm is intelligently selected to ensure the pertinence and effectiveness of feature extraction. Finally, the feature map generated by pooling processing can accurately locate the position of the target object in the image, thereby achieving efficient and accurate image recognition. The technical solution provided in the embodiment of the present application is particularly suitable for target detection in complex scenes, and can significantly improve the recognition accuracy and processing speed.

[0032] It should be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] The drawings herein are incorporated into the specification and constitute a part of the specification, illustrate embodiments consistent with the present application, and together with the specification are used to explain the principles of the present application. Obviously, the drawings described below are only some embodiments of the present application, and for ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0034] Figure 1 The exemplary system architecture block diagram applying the technical solution of the present application is schematically shown.

[0035] Figure 2 The network structure of the twin neural network used to achieve target tracking in the related technology of this application is shown.

[0036] Figure 3 The network architecture of the target tracking network in the related technology of this application is shown.

[0037] Figure 4 The network structure of the feature extraction network AlexNet in the related technology of this application is shown.

[0038] Figure 5 A flow chart of an object recognition method in one embodiment of the present application is shown.

[0039] Figure 6 A schematic diagram showing the principle of maximum pooling is shown.

[0040] Figure 7 A schematic diagram showing the principle of uniform pooling.

[0041] Figure 8A schematic diagram of image grayscale values ​​in an application scenario of an embodiment of the present application is shown.

[0042] Fig. 9 A schematic diagram of an operator of Robert edge computing used in an application scenario in an embodiment of the present application is shown.

[0043] Fig.10 A schematic diagram showing pixel distribution after normalizing the gradient value in an application scenario of an embodiment of the present application

[0044] Fig.11 A binary image of an image edge obtained in an application scenario according to an embodiment of the present application is shown.

[0045] Fig.12 A schematic diagram of a local area in which a maximum pooling algorithm is selected in an application scenario according to an embodiment of the present application is shown.

[0046] Fig.13 A schematic diagram of a local area in which a hybrid pooling algorithm is selected in an application scenario according to an embodiment of the present application is shown.

[0047] Fig.14 A schematic diagram of a local area in which an average pooling algorithm is selected in an application scenario according to an embodiment of the present application is shown.

[0048] Fig.15 A diagram of a parallel spatiotemporal dual-channel feature fusion architecture implemented in an application scenario of an embodiment of the present application is shown.

[0049] Fig.16 A target tracking solution based on initial template update adopted in a related technology of the present application is shown.

[0050] Fig.17 A diagram showing an architecture of a pixel-level template update based on tracking results and an initial template in an application scenario according to an embodiment of the present application.

[0051] Fig.18 A schematic diagram of calculating the similarity between tracking results and initial template features in an application scenario of an embodiment of the present application is shown.

[0052] Fig.19 The accuracy curves of the present application and several related technologies on LaSOT are shown.

[0053] Fig. 20 The success rate curve of the present application and multiple related technologies on LaSOT is shown.

[0054] Fig.21 The structural block diagram of the object recognition device provided in the embodiment of the present application is schematically shown.

[0055] Fig. 22 The structure block diagram of a computer system suitable for implementing an electronic device of an embodiment of the present application is schematically shown. DETAILED DESCRIPTION

[0056] Example embodiments will now be described more fully with reference to the accompanying drawings. However, example embodiments can be implemented in a variety of forms and should not be construed as limited to the examples set forth herein; rather, these embodiments are provided so that this application will be more comprehensive and complete and fully convey the concept of the example embodiments to those skilled in the art.

[0057] In addition, described feature, structure or characteristic can be combined in one or more embodiments in any suitable manner. In the following description, many specific details are provided to provide a full understanding of the embodiments of the present application. However, those skilled in the art will appreciate that the technical scheme of the present application can be put into practice without one or more of the specific details, or other methods, components, devices, steps, etc. can be adopted. In other cases, known methods, devices, realizations or operations are not shown or described in detail to avoid blurring the various aspects of the application.

[0058] In the embodiments of the present application, the term "module" or "unit" refers to a computer program or a part of a computer program with a predetermined function, and works together with other related parts to achieve a predetermined goal, and can be implemented in whole or in part by using software, hardware (such as processing circuits or memories) or a combination thereof. Similarly, a processor (or multiple processors or memories) can be used to implement one or more modules or units. In addition, each module or unit can be part of an overall module or unit that includes the function of the module or unit.

[0059] The block diagrams shown in the accompanying drawings are merely functional entities and do not necessarily correspond to physically independent entities. That is, these functional entities may be implemented in software form, or in one or more hardware modules or integrated circuits, or in different networks and / or processor devices and / or microcontroller devices.

[0060] The flowcharts shown in the accompanying drawings are only exemplary and do not necessarily include all the contents and operations / steps, nor must they be executed in the order described. For example, some operations / steps can be decomposed, and some operations / steps can be combined or partially combined, so the actual execution order may change according to actual conditions.

[0061] Figure 1 The exemplary system architecture block diagram applying the technical solution of the present application is schematically shown.

[0062] like Figure 1As shown, the system architecture 100 may include a terminal device 110, a network 120 and a server 130. The terminal device 110 may include various electronic devices such as smart phones, tablet computers, laptops, desktop computers, smart speakers, smart wearable devices, smart vehicle-mounted devices, and smart payment terminals. The server 130 may be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms. The network 120 may be a communication medium of various connection types that can provide a communication link between the terminal device 110 and the server 130, such as a wired communication link or a wireless communication link.

[0063] According to the implementation requirements, the system architecture in the embodiment of the present application can have any number of terminal devices, networks and servers. For example, the server 130 can be a server group composed of multiple server devices. In addition, the technical solution provided in the embodiment of the present application can be applied to the terminal device 110, can also be applied to the server 130, or can be implemented by the terminal device 110 and the server 130 together, and the present application does not make any special restrictions on this.

[0064] Figure 2 The network structure of the twin neural network used to achieve target tracking in the related technology of this application is shown. In the target tracking method based on deep learning, the twin neural network (Siamese Network) is a typical network structure, which is a coupled architecture composed of two or more neural networks with the same structure and shared weights. Its main feature is that the input data is processed in parallel through two networks, and a similarity measurement function is learned to compare the similarity of two input data.

[0065] like Figure 2 As shown in the figure, the twin neural network inputs the target image z and the search area image x into two identical networks for feature extraction, and then calculates the similarity between the two feature maps to determine the location of the target. This method has the advantages of high computational efficiency and simple model structure, so it has been widely used in practical applications.

[0066] The following are the typical steps and algorithm formulas of a target tracking algorithm based on Siamese Network.

[0067] (1) Feature extraction: Input the target image z (the initial template of the first frame, with a size of 127×127×3) and the search area image x (a frame in the video frame, with a size of 255×255×3). Use a deep neural network (such as AlexNet, VGG, etc.) to extract features from the target image and the search area image to obtain the corresponding feature maps φ(z) and φ(x).

[0068] (2) Feature graph cross-correlation: Calculate the cross-correlation (a similarity calculation method) between the feature graphs φ(z) and φ(x) to obtain the similarity graph g. The cross-correlation operation here can be implemented through a convolution operation. Formula: g = φ(z)*φ(x) (where * represents a convolution operation).

[0069] (3) Similarity map processing: The similarity map g is upsampled (increasing the resolution of the image from a lower to a higher resolution) by bilinear interpolation (an interpolation method in two-dimensional space used in fields such as image and video processing) to make it have the same spatial resolution as the search area image x. A cosine window and scale change penalty are applied to suppress the influence of boundary effects and scale changes on the tracking results.

[0070] (4) Target position determination: Find the position of the maximum value on the similarity graph g, which is the position of the target in the current frame. Formula: (x_t, y_t,) = max(g), where (x_t, y_t,) represents the position of the target in the current frame.

[0071] (5) Repeat the above steps: For each frame in the video sequence, repeat the above steps to achieve continuous tracking of the target.

[0072] The training method of the target tracking neural network includes the following steps.

[0073] (1) Data preparation: Prepare a dataset containing a large number of annotated video frames, each of which is labeled with the location and size of the target. Extract the target template and search area from these video frames as training samples.

[0074] (2) Initialize network parameters: Use a pre-trained convolutional neural network (such as AlexNet, VGG, etc.) to initialize the parameters of the feature extraction sub-network. For the cross-correlation layer and other newly added layers, you can use random initialization or pre-trained parameters.

[0075] (3) Forward propagation: Input the target template and search area into the feature extraction subnetwork respectively to obtain the corresponding feature map. The two feature maps are sent to the cross-correlation layer for cross-correlation operation to obtain the similarity map.

[0076] (4) Calculate the loss: Generate a binary label map on the similarity map based on the annotated target position (the target position is 1, and other positions are 0). Use the logistic loss or cross entropy loss to calculate the loss between the predicted similarity map and the label map.

[0077] (5) Back propagation: Back propagation is performed based on the calculated loss to update the network parameters. Gradient descent algorithms (such as SGD, Adam, etc.) can be used to optimize network parameters.

[0078] (6) Iterative training: Repeat the above steps until the network achieves satisfactory performance on the validation set or reaches the preset number of iterations.

[0079] (7) Model saving: Save the trained model parameters to a file for use in testing or actual applications.

[0080] As deep learning-based target tracking methods have flourished in this field, more and more researchers have begun to use deep learning to design various target tracking neural networks.

[0081] Figure 3 FIG. 1 shows the network architecture of the target tracking network in the related art of this application. Figure 3 As shown in Figure 1, the target tracking network roughly includes the following four modules.

[0082] (1) Feature extraction module: It is responsible for extracting useful feature information from the input image or video frame. These features may include color, texture, shape, etc., which help to distinguish the target from the background.

[0083] (2) Feature fusion module: After extracting the feature vector, this module is responsible for fusing the features between the target template and the candidate region. This is usually done by calculating the response map between the feature vectors in some way (similarity, attention score). The result reflects the degree of match between the candidate region and the target.

[0084] (3) Target positioning module: Based on the results calculated by the feature fusion module, this module is responsible for determining the exact location of the target in the current frame. This usually involves sorting and screening the candidate regions and selecting the region with the highest similarity as the predicted location of the target.

[0085] (4) Template update module: During the target tracking process, the target’s appearance may change due to factors such as lighting, viewing angle, and deformation. The template update module can use the target information in the new frame to update the original target template to adapt to these changes.

[0086] At present, the feature extraction networks of target tracking networks based on deep learning mostly rely on CNN backbone networks (such as AlexNet and VGGNet, etc.). CNN stands for Convolutional Neural Networks, which is a deep learning algorithm that is particularly suitable for processing data with grid structures, such as images and speech.

[0087] Since CNN relies on convolution operations, and convolution is local, that is, each convolution kernel only processes the local area it covers, the receptive field of CNN is usually limited. This means that CNN is poor at capturing global context information. Global context information is crucial for accurately judging the location and state of the target. For example, when the target encounters challenges such as occlusion, deformation, or scale changes. To solve this problem, most CNN-based feature extraction networks need to add a pooling layer to increase the receptive field of the CNN feature extraction module.

[0088] Figure 4 The network structure of the feature extraction network AlexNet in the related art of this application is shown. Figure 4 As shown in , AlexNet is an 8-layer CNN with a pooling layer following the first, second, and fifth convolutional layers. Figure 3-4 Specifically, the pooling layer divides the feature map into multiple sub-regions through certain pooling operations (such as maximum pooling or average pooling), and selects a representative value (such as the maximum value or average value) from these sub-regions as the pixel value of the output feature map. This process reduces the size of the feature map so that each output pixel corresponds to a larger area in the original image.

[0089] Although the pooling layer can increase the receptive field, too many pooling operations may lead to excessive information loss and affect the performance of the model. Commonly used pooling layer methods mainly include maximum pooling (Max Pooling) and average pooling (Mean Pooling or Average Pooling).

[0090] The disadvantage of max pooling is that it will lose some non-maximum feature information, which is useful for some specific visual tasks. In addition, if there are noise or outliers in the input data, max pooling may retain these noise or outliers as important features.

[0091] The disadvantage of average pooling is that due to the calculation of the average value, average pooling may make the image blurred and lose some detail information, which will lead to poor tracking effect.

[0092] Secondly, in the feature fusion module, most of the current feature fusion modules, whether based on Siamese trackers or Transformers, regard target tracking as template matching between the initial template and the current search area. These methods only capture spatial similarity, focusing only on the spatial region of the image, while ignoring temporal information. However, both spatial and temporal information are important for target tracking. The former contains target appearance information for positioning, which can help overcome challenges such as occlusion and noise. The latter contains the state changes of the target across frames (such as motion mode or speed), which can be used to combat the problems of rapid motion, deformation, and partial occlusion of the target. This deficiency may reduce the performance of the model in complex scenes, such as objects that undergo large-scale changes or frequently enter and exit the view.

[0093] Finally, in the template update module, the introduction of dynamic templates is intended to cope with various changes that may occur in the target appearance during the tracking process, such as illumination changes, occlusions, deformations, and motion blur. However, directly updating the template or using a simple fusion strategy may lead to the accumulation of tracking errors and the ultimate loss of the target. To address these issues, researchers have explored strategies that only retain reliable dynamic templates, which involves training a classification network to evaluate the confidence of the template. However, this approach increases the amount of computation, reduces real-time performance, and the granularity of judging the reliability of the template is large and depends on the selection of hyperparameter thresholds. Therefore, how to reduce the amount of computation, improve real-time performance, and find an adaptive threshold setting method while ensuring tracking robustness and accuracy is still a problem that needs to be further studied in the field of target tracking.

[0094] Based on the above introduction to the related technology, it can be seen that the related technology of this application mainly has the following problems.

[0095] (1) The maximum pooling method used in the feature extraction module of the existing target tracking network will cause the loss of some non-maximum feature information, while the average pooling method can retain the overall information of the image but will lose the edge texture information that is useful for target tracking, making the image blurred, which will affect the accuracy and stability of target tracking.

[0096] (2) Existing trackers only focus on the spatial area of ​​the image during the feature fusion stage, and fail to fully utilize spatial and temporal information for feature fusion, resulting in the problem of losing the target in complex scenes. Some trackers that use the Transformer's Encoder-Decoder architecture to fuse spatiotemporal information not only have a large amount of computation and are difficult to track in real time, but also have problems matching the Encoder-Decoder architecture with the target tracking network.

[0097] (3) In the template update module of target tracking, some current dynamic template update methods are unreliable in cropping templates when the target is completely occluded or moves out of the field of view, or when the tracker drifts, which will cause pollution to the template image. Some dynamic template update methods take the reliability of the template into consideration, but this solution only checks whether the entire template is reliable and has a large granularity. This "reliable" template still has unreliable backgrounds, which will result in loss of tracking accuracy.

[0098] In view of the problems existing in the above related technologies, the embodiments of the present application provide the following solutions.

[0099] (1) An edge pixel ratio adaptive pooling method is proposed. The edge information of the feature map is obtained by edge extraction, and then different pooling calculations are performed according to different regions of the target (pixel ratio calculation). Maximum pooling can be performed on the edge of the target to highlight the edge and texture features. Secondly, average pooling is performed inside the target to retain the overall features. Finally, at the mixture of the edge and the inside, maximum pooling and average pooling are mixed to retain various feature information to the maximum extent.

[0100] (2) A parallel spatiotemporal dual-channel feature fusion strategy is proposed. The spatial information brought by the initial template and the temporal information of the dynamic template are simultaneously utilized to deeply fuse the search area features with the initial template features and the dynamic template features, and the feature fusion network is reconstructed. This reduces the computational complexity of the network without losing accuracy, thereby improving the real-time performance of target tracking.

[0101] (3) A template update network based on the correlation calculation between the dynamic template and the initial template is proposed. The multi-channel feature map of the dynamic template is correlated with the feature map of the initial template to obtain a similarity weight map between the dynamic template and the initial template. The similarity map is then used as a penalty operator to calculate on the dynamic template, so that the pixels belonging to the target on the dynamic template are given a higher weight to participate in the calculation, while the pixels that are not the target are suppressed, ensuring that the dynamic template is updated instantly while avoiding contamination.

[0102] It is difficult to find a suitable pooling solution to perform different pooling methods on different areas of the image in the existing technical solutions. Or a single pooling method is selected, although this method can meet the needs of most scenes. However, once the tracking scene becomes complicated, this method becomes unreliable.

[0103] The technical difficulty in template updating lies in how to select a reliable dynamic template. Some current solutions only consider whether the target tracked in a certain frame is reliable. Although this can solve the template contamination problem to a certain extent, this solution will make it difficult to update the dynamic template in a timely manner when encountering complex scenarios such as occlusion because the tracked target is unreliable.

[0104] The following is a detailed description of the technical solutions such as the object recognition method, object recognition device, computer-readable medium, electronic device, and computer program product provided by the present application in conjunction with specific implementation methods.

[0105] Figure 5 FIG. 1 is a flow chart showing an object recognition method in one embodiment of the present application. Figure 5 As shown, the object recognition method includes the following steps S510 to S540.

[0106] S510: Acquire an image to be identified, and perform edge pixel detection on the image to be identified to obtain edge pixel points and non-edge pixel points in the image to be identified; edge pixel points are pixel points located in the edge area of ​​the content object in the image to be identified, and non-edge pixel points are pixel points located in the non-edge area of ​​the content object in the image to be identified;

[0107] S520: selecting a target pooling algorithm that matches the pooling area from a plurality of candidate pooling algorithms according to the distribution ratio of edge pixels in the pooling area; the pooling area is a local image area collected by moving a pooling kernel with a preset size on the image to be recognized;

[0108] S530: Performing pooling processing on the pooling area according to the target pooling algorithm to obtain a feature map corresponding to the image to be recognized;

[0109] S540: Locate the distribution position of the target object to be identified in the image to be identified according to the feature map.

[0110] In the object recognition method provided in the embodiment of the present application, the accuracy and efficiency of image recognition are effectively improved through fine edge pixel detection and intelligent pooling algorithm selection. First, the edge and non-edge areas of the content object in the image are distinguished through edge pixel detection, which provides accurate area division for subsequent processing. Then, according to the distribution ratio of edge pixels in the pooling area, the most appropriate pooling algorithm is intelligently selected to ensure the pertinence and effectiveness of feature extraction. Finally, the feature map generated by the pooling process can accurately locate the position of the target object in the image, thereby achieving efficient and accurate image recognition. The object recognition method provided in the embodiment of the present application is particularly suitable for target detection in complex scenes, and can significantly improve the recognition accuracy and processing speed.

[0111] In one embodiment of the present application, according to the distribution ratio of edge pixels in the pooling area, selecting a target pooling algorithm that matches the pooling area from a plurality of candidate pooling algorithms may further include: counting the distribution ratio of edge pixels in the image to be identified to obtain the overall area ratio of edge pixels; counting the distribution ratio of edge pixels in the pooling area to obtain the local area ratio of edge pixels; according to the numerical relationship between the overall area ratio and the local area ratio, selecting a target pooling algorithm that matches the pooling area from a plurality of candidate pooling algorithms.

[0112] The embodiment of the present application provides a variety of candidate pooling algorithms to choose from, and flexibly selects the target pooling algorithm according to the distribution ratio of edge pixels. Among them, the candidate pooling algorithms include the maximum pooling algorithm, the average pooling algorithm and the hybrid pooling algorithm, and the hybrid pooling algorithm is a pooling algorithm formed by combining the maximum pooling algorithm and the average pooling algorithm.

[0113] Max Pooling divides the input image into several rectangular regions and outputs the maximum value for each sub-region. Its definition is as follows:

[0114]

[0115] Among them, y kij Represents the rectangular area R related to the kth feature map ij The maximum pooling output value, x kpq Represents the rectangular area R ij The element at (p,q) in .

[0116] Figure 6 Figure 2 shows a schematic diagram of the principle of maximum pooling. Figure 6 As shown in the figure, taking a 6×6 feature map as an example, a 2×2 pooling kernel is used with a step size of 2 (moving 2 grids) to "scan" and select the maximum value to output to the next layer. This is called maximum pooling.

[0117] For the maximum pooling operation, only the maximum value in each rectangular area is selected to enter the next layer, and other elements will not enter the next layer. Therefore, the maximum pooling extracts the part with the strongest response in the feature map and enters the next layer. This method discards a lot of redundant information in the network and retains more texture information of the image, making the network easier to optimize. Maximum pooling only selects the maximum value in each pooling window, so it discards non-maximum information, which may lead to the loss of some detail information. Secondly, if the maximum value happens to be generated by noise, then the maximum pooling may mistakenly retain this noise information.

[0118] Average Pooling divides the input image into several rectangular regions and outputs the average value of all elements in each sub-region. Its definition is as follows:

[0119]

[0120] Among them, y kij Represents the rectangular area R related to the kth feature map ij The average pooling output value, x kpq Represents the rectangular area R ij The element at (p,q) in |R ij | represents a rectangular area R ij The number of elements in .

[0121] Figure 7 Figure 2 shows a schematic diagram of the principle of uniform pooling. Figure 7 As shown in the figure, taking a 4×4 feature map as an example, a 2×2 pooling kernel is used with a step size of 2 to "scan", average the elements in the area, and output the average value to the next layer. This is called average pooling.

[0122] Average pooling can retain the overall information of an image or feature map, has a certain effect on image smoothing, and pays more attention to the overall information. However, it is not suitable for emphasizing significant features, and because average pooling averages the feature points in the neighborhood, it may cause the image or feature map to become blurred.

[0123] The mixed pooling method mainly combines average pooling and maximum pooling to achieve a more optimized pooling effect. This method obtains the final pooling output by randomly selecting or linearly combining the results of average pooling and maximum pooling.

[0124] Random selection is to randomly select the result of mean pooling or maximum pooling as the final pooling output according to a certain probability (such as 0.5). This method combines the advantages of the two pooling methods, but may introduce some randomness.

[0125] Linear combination is to linearly combine the results of mean pooling and maximum pooling by setting a weight parameter λ(λ∈[0,1]). The formula is as follows:

[0126]

[0127] in, represents the pooled output of the k-th position of the l-th layer, represents the result of mean pooling, Represents the result of maximum pooling. The value of λ can be adjusted according to specific tasks and data sets. Although hybrid pooling combines the advantages of mean pooling and maximum pooling to a certain extent, it not only retains the overall data characteristics of the image, but also extracts the texture characteristics of the image, thereby retaining the image information more comprehensively. However, additional parameters are introduced. In hybrid pooling, a weight parameter λ is usually required to control the weights of mean pooling and maximum pooling, which may introduce additional parameters, increase the complexity of the model and the difficulty of adjusting parameters. Secondly, randomness is introduced: If random selection is used to implement hybrid pooling, some randomness will be introduced, resulting in unstable output results of the model.

[0128] In summary, the current maximum pooling, average pooling and hybrid pooling all have the above-mentioned inevitable defects. Therefore, it is very important to find a solution that can retain edge texture information while not losing detail information. Therefore, the embodiment of the present application proposes a solution for selecting a pooling algorithm based on the edge pixel distribution ratio, performing maximum pooling in the target edge area to highlight the edge texture information, and performing average pooling in the non-edge area to retain detail information.

[0129] Figure 8 The following is a schematic diagram of the grayscale value of an image in an application scenario of the present invention. In the distribution of the grayscale value of pixels in an image, the pixel difference between the edge of the target object and the surrounding background is generally large, while the pixel values ​​of other non-edge parts are relatively smooth and will not have large mutations. Figure 8 As shown, the central area of ​​the image belongs to the target object that needs to be identified and tracked.

[0130] Fig. 9 The schematic diagram of the operator of Robert edge calculation used in an application scenario of the embodiment of the present application is shown. By performing edge extraction on this image (calculating the difference between the gray value of the pixel point and the surrounding gray value), the Robert operator can be used to extract the edge information of the horizontal and vertical directions of the image.

[0131] like Fig. 9 As shown, the gradient magnitudes in the x and y directions and the final gradient magnitudes at pixel point P5 are:

[0132]

[0133] g=|g x |+|g y |

[0134] After gradient calculation, the gradient value of the image is obtained. The larger the gradient value, the greater the possibility that the pixel point is an edge pixel point of the image.

[0135] In order to compare gradient values ​​at the same scale, the gradient values ​​obtained by gradient calculation can be normalized.

[0136] Fig.10 A schematic diagram of pixel distribution after normalizing the gradient value in an application scenario of an embodiment of the present application is shown. The gradient value can be normalized to between 0 and 100 through normalization.

[0137] In order to determine which pixels are edge pixels, the gradient value can be binarized, and if it is greater than the average value, it is set to 1, otherwise it is set to 0. In this way, it can be determined which pixels in the image can be used as edge pixels, which is convenient for subsequent calculations.

[0138] Fig.11 A binary image of an image edge obtained in an application scenario according to an embodiment of the present application is shown.

[0139] based on Fig.11 For the binary image shown, the area ratio of edge pixels can be calculated according to the following formula.

[0140]

[0141] In one embodiment of the present application, based on the numerical relationship between the overall area ratio and the local area ratio, a target pooling algorithm matching the pooling area is selected from a plurality of candidate pooling algorithms, which may include: when the local area ratio is greater than the overall area ratio, selecting the maximum pooling algorithm as the target pooling algorithm matching the pooling area; when the local area ratio is less than the overall area ratio and greater than zero, selecting the hybrid pooling algorithm as the target pooling algorithm matching the pooling area; when the local area ratio is equal to zero, selecting the average pooling algorithm as the target pooling algorithm matching the pooling area.

[0142] Fig.12 FIG. 1 shows a schematic diagram of a local area in which a maximum pooling algorithm is selected in an application scenario in an embodiment of the present application. Fig.12 As shown in , when the local area ratio is greater than the overall area ratio, the maximum pooling algorithm is selected as the target pooling algorithm that matches the pooling area. Specifically, when the edge pixel ratio Px in the pooling kernel is greater than the total edge pixel ratio P 11*11 When , it means that there are many edge pixels in the pooling core area, and maximum pooling should be adopted to focus on retaining key edge texture information.

[0143] Fig.13 FIG. 1 shows a schematic diagram of a local area in which a hybrid pooling algorithm is selected in an application scenario according to an embodiment of the present application. Fig.13As shown in , when the local area ratio is less than the overall area ratio and greater than zero, the hybrid pooling algorithm is selected as the target pooling algorithm that matches the pooling area. Specifically, when the edge pixel ratio in the pooling kernel is less than the total edge pixel ratio P 11*11 When , it means that the pixel is at the junction of the edge and the background, and a hybrid pooling algorithm including maximum pooling and average pooling can be adopted at the same time.

[0144] In one embodiment of the present application, when a hybrid pooling algorithm is selected as the target pooling algorithm that matches the pooling area, the pooling area is pooled according to the target pooling algorithm, including: pooling the pooling area according to the maximum pooling algorithm to obtain a first pooling eigenvalue; pooling the pooling area according to the uniform pooling algorithm to obtain a second pooling eigenvalue; using the local area ratio as a weighting coefficient, performing a weighted operation on the first pooling eigenvalue and the second pooling eigenvalue to obtain a hybrid pooling eigenvalue of the pooling area.

[0145] For example, the embodiment of the present application can calculate the mixed pooling feature value result according to the following formula.

[0146] Result = P x *Max_Pool+(1-P x )Average_Pool

[0147] Because P x It represents the local area ratio of edge pixels, that is, the ratio of edge pixels in the current area, so it is used as the coefficient of maximum pooling. (1-P x ) represents the ratio of non-edge pixels as the coefficient of average pooling, and finally the values ​​of the two are added as the result, so as to avoid the loss of information at the junction of the edge and background by a single pooling method.

[0148] In one embodiment of the present application, when a hybrid pooling algorithm is selected as the target pooling algorithm that matches the pooling area, the pooling area is pooled according to the target pooling algorithm, including: pooling the pooling area according to the maximum pooling algorithm to obtain a first pooling eigenvalue; pooling the pooling area according to the uniform pooling algorithm to obtain a second pooling eigenvalue; and randomly selecting a eigenvalue from the first pooling eigenvalue and the second pooling eigenvalue as the hybrid pooling eigenvalue of the pooling area using the local area ratio as the selection probability.

[0149] The embodiment of the present application realizes multi-angle feature extraction of the pooled area by combining the maximum pooling and uniform pooling algorithms. The maximum pooling algorithm can capture the maximum response value of the local area, while the uniform pooling algorithm can average all the response values ​​of the local area. The combination of the two makes the feature extraction more comprehensive. By taking the local area proportion as the selection probability and randomly selecting a feature value as the mixed pooling feature value, the flexibility and randomness of feature selection are increased, which helps to improve the adaptability and generalization ability of the model to different types of images. The embodiment of the present application can effectively balance the locality and globality of features, enhance the expressiveness of features, and thus achieve better performance in image recognition and processing tasks.

[0150] Fig.14 FIG. 1 shows a schematic diagram of a local area in which an average pooling algorithm is selected in an application scenario in an embodiment of the present application. Fig.14 As shown in the figure, when the local area ratio is equal to zero, the average pooling algorithm is selected as the target pooling algorithm that matches the pooling area. Specifically, in the area where the edge pixel ratio is 0, it is almost all background information. Here, the overall information needs to be retained, so the average pooling is the best, and it can also reduce the impact of abnormal noise peaks.

[0151] In one embodiment of the present application, the distribution position of the target object to be identified in the image to be identified is located according to the feature map, including: obtaining a feature template corresponding to the target object to be identified; performing feature fusion processing on the feature template and the candidate area in the feature map to obtain a response map of the candidate area, the response map is used to represent the similarity between the candidate area and the feature template; and locating the candidate area with the highest similarity to the feature template as the distribution position of the target object in the image to be identified according to the response map.

[0152] The embodiment of the present application significantly improves the accuracy and efficiency of target object recognition through intelligent matching of feature templates and efficient generation of response graphs. First, the embodiment of the present application provides an accurate reference for the recognition process by obtaining a feature template corresponding to the target object. Then, a response graph is generated by performing feature fusion processing on the feature template and the candidate area in the feature graph, which intuitively represents the similarity between the candidate area and the feature template. Finally, based on the analysis of the response graph, the candidate area with the highest similarity to the feature template is quickly located, thereby accurately determining the position of the target object in the image to be identified. The embodiment of the present application not only reduces the possibility of misidentification, but also improves the speed and automation of the recognition process, and is particularly suitable for scenarios that require fast and accurate recognition, such as security monitoring, autonomous driving and other fields.

[0153] At present, trackers based on CNN, CNN-Transformer, or completely Transformer only focus on the spatial area of ​​the image in the feature fusion stage, and fail to fully utilize spatial and temporal information for feature fusion. However, both spatial and temporal information are important for target tracking. The former contains the target appearance information used for positioning, which can help overcome challenges such as occlusion and noise. The latter contains the state changes of the target across frames (such as motion mode or speed), which can be used to combat the problems of rapid target movement, deformation, and partial occlusion.

[0154] Transformer is a deep learning model based on the self-attention mechanism. This model has caused a revolution in the field of natural language processing (NLP) because its design overcomes the limitations of traditional models in dealing with long-distance dependencies and parallel computing. In recent years, Transformer has also been applied in the field of computer vision, such as image classification, object detection and other tasks.

[0155] In order to make full use of the temporal information provided by the dynamic template, Transformer-based target tracking networks have emerged one after another, and have begun to use the Transformer's self-attention mechanism to simultaneously model the initial template, dynamic template, and search area, such as Stark.

[0156] Stark and other Transformer-based target tracking networks also face some problems. First, the self-attention mechanism in the Transformer model is very computationally expensive, especially when processing high-resolution images or video frames. This will cause the model to be too slow when processing real-time target tracking tasks and it is difficult to meet real-time requirements.

[0157] The second is the problem of matching data sequences: in target tracking tasks, the model usually needs to find the position of the same target between consecutive video frames. However, the Transformer's Encoder-Decoder architecture is designed to handle sequence-to-sequence mapping tasks, such as machine translation or text summarization, where the input and output sequences usually have similar lengths and structures. In target tracking, the input is usually the image features of the current frame, and the output is the prediction of the target position, and there is no direct correspondence between the two.

[0158] Finally, when the initial template, dynamic template and search area are concatenated and sent to the Transformer for feature extraction, the Transformer's self-attention will calculate the self-attention score of each pixel with other pixels. Therefore, when calculating the self-attention of pixels between the search area and the initial template, more attention can be paid to the pixels belonging to the target in the search area. When calculating the self-attention of pixels between the search area and the dynamic template, the time information provided by the dynamic template can be fully utilized to extract the time features of the target. However, this network framework will also calculate the self-attention of the dynamic template and the initial template. Once the dynamic template is unreliable, it will pollute the most robust initial template features and increase unnecessary computational consumption.

[0159] In view of the problems existing in the above related technologies, the embodiment of the present application provides a parallel spatiotemporal dual-channel feature fusion strategy. Combining the advantages of static templates (static templates are generally manually framed in the first frame of the video) and dynamic templates (dynamic templates are cut by the center point of the tracked results, and the cut size is consistent with the static template size), the spatial and temporal information is integrated into a whole instead of splitting them. In order to solve the matching problem of data sequences, the embodiment of the present application does not adopt an architecture based on Encoder-Decoder that is not suitable for target tracking, but only uses the self-attention calculation module of Encoder to fuse the features of the template and the search area, which can solve the adaptation problem and reduce the calculation amount of the model. Secondly, in order to solve the problem of contaminated initial templates caused by fusion similar to Stark series, the embodiment of the present application adopts parallel dual-branch fusion, avoiding the direct operation of dynamic templates and initial templates, and reducing the problem of initial template contamination. Finally, distinctive spatiotemporal features are generated for target positioning to increase feature robustness.

[0160] In one embodiment of the present application, the feature template includes a pre-configured static template and a dynamic template dynamically updated during the object recognition process; the feature template is subjected to feature fusion processing with the candidate area in the feature map, including: performing feature fusion processing on the static template and the candidate area in the feature map to obtain a first fusion feature with spatial information; performing feature fusion processing on the dynamic template and the candidate area in the feature map to obtain a second fusion feature with temporal information; performing convolution operation on the first fusion feature and the second fusion feature to obtain a response map of the candidate area.

[0161] Fig.15 A diagram of a parallel spatiotemporal dual-channel feature fusion architecture implemented in an application scenario of an embodiment of the present application is shown.

[0162] like Fig.15 As shown, the candidate region f x and the static template f z0The spatial channel feature modeling branch, the candidate region f x and dynamic template f z1 The temporal channel feature modeling branch is formed. Given the computational properties of Transformer self-attention, it is necessary to perform encoder calculation by concatenating candidate regions and templates. Two feature vectors are obtained by concatenating them in pairs.

[0163] f mix1 =Concat(f z1 , f x )

[0164] f mix2 =Concat(f z0 , f x )

[0165] Since static templates and dynamic templates lack contextual relevance, and in order to avoid the static template being contaminated by inaccurate dynamic templates, we choose not to directly calculate the attention score between the static template and the dynamic template through the self-attention module. Instead, we perform self-attention calculations on the candidate region features separately, and the concatenated feature sequence is sent to the multi-head attention module to calculate the feature sequence self-attention score.

[0166] f′ mix1 =f mix1 +MSA(f mix1 )

[0167] f′ mix2 =f mix2 +MSA(f mix2 )

[0168] The FFN module is used to enhance the fitting ability of the model. It is a fully connected feedforward network consisting of two linear transformations with a ReLU in the middle. It then extracts the fused candidate region features and restores them to a multi-dimensional feature map.

[0169] Assume that the original candidate region f x Dimensions are H*W*C, static template f z0 H1*W1*C1 is stretched into one dimension and then spliced ​​f mix2 =Concat(f z0 ,f x ), the size is 1*1*(HWC+H1W1C1), and after network fusion, it is still 1*1*(HWC+H1W1C1). The embodiment of the present application needs to track the target in the candidate area, so it is necessary to crop the features of the candidate area. The clipping method is to remove the features of the template and obtain the candidate area features of 1*1*HWC.

[0170] The vector is restored to a multi-dimensional feature map, that is, the 1*1*HWC vector is converted into a three-dimensional matrix of H*W*C).

[0171] The overall calculation process is as follows:

[0172]

[0173] Based on the above formula, the feature based on time information is obtained and spatial information Then fuse them through convolution-based CNN layers and In this way, we can obtain the candidate region feature map with both temporal and spatial information.

[0174]

[0175] Through the above-mentioned feature fusion network module, the embodiment of the present application can model the global relationship between all elements in the spatial and temporal dimensions to extract discriminative spatiotemporal features, providing an accurate and robust feature map for the subsequent target positioning network module.

[0176] In the field of target trackers, dynamic templates are introduced to solve the problem of changes in target appearance during tracking. Since the target's appearance may be affected by many factors such as lighting changes, occlusion, deformation, motion blur, etc., static templates often have difficulty adapting to these changes, resulting in tracking failure. Dynamic templates can be dynamically updated according to real-time changes in the target's appearance, thereby improving the robustness and accuracy of tracking.

[0177] Fig.16 FIG. 2 shows a target tracking solution based on initial template update adopted in a related technology of the present application. Fig.16 As shown in the figure, this scheme directly uses the tracking result as the dynamic template to replace the initial template for tracking the next frame. If the target is deformed or blocked, the target tracked by this scheme is often unreliable. If the unreliable result is used as the new dynamic template for calculation of subsequent frames, it will lead to error accumulation and eventually cause the loss of the tracked target.

[0178] In order to solve this problem, some target tracking networks use the tracked target as a dynamic template and merge it with the initial template into a new dynamic template. Although this method retains the information of the initial template and can achieve good tracking effects in short-term tracking, in long-distance target tracking scenarios, once the target is lost, the dynamic template becomes unreliable. This unreliable information will also gradually accumulate in long-term tracking scenarios, eventually causing the template to be contaminated and unable to effectively track the target.

[0179] In order to solve these problems of the above template update scheme, some studies have begun to study how to retain only reliable dynamic templates to participate in subsequent target tracking. These methods study by training a classification network, with dynamic templates as input and confidence scores as output. Once the confidence is higher than a certain threshold, the template is considered reliable and can participate in target tracking in subsequent frames. However, this method also has some problems. One is that the calculation through an additional deep learning network will increase the amount of calculation during the tracking process and reduce the real-time performance of tracking. Secondly, the granularity of judging whether the dynamic template is reliable is too large, and the granularity at the image level will inevitably lead to some areas of the dynamic template being unreliable. Finally, this template update method is very dependent on the selection of hyperparameter thresholds. How to select a suitable threshold is a difficult problem because the threshold values ​​of different scenes will show great changes.

[0180] Based on the above-mentioned problems of current template update networks, an embodiment of the present application proposes a template update network based on the correlation calculation between the tracking results and the initial template. The multi-channel feature map of the tracking result is correlated with the feature map of the initial template to obtain a similarity weight map between the tracking result and the initial template. The similarity map is then used as a penalty operator to calculate the latest dynamic template with the tracking result, so that the pixel points belonging to the target on the dynamic template are given a higher weight to participate in the calculation, while the non-target pixel points are suppressed, so that pixel-level dynamic template selection can be achieved.

[0181] In one embodiment of the present application, target features are cropped from a feature map according to the distribution position of the target object in the image to be identified; feature similarity between the target features and a static template is obtained; and the dynamic template is updated according to the feature similarity.

[0182] In one embodiment of the present application, the feature similarity includes similarity values ​​at each pixel point; updating the dynamic template according to the feature similarity includes: using the similarity values ​​at each similar point as weights, performing weighted operations on each pixel value of the dynamic template, and obtaining an updated dynamic template.

[0183] Fig.17 A diagram showing an architecture of a pixel-level template update based on tracking results and an initial template in an application scenario according to an embodiment of the present application.

[0184] like Fig.17 As shown, it is assumed that there are two feature maps, represented as the tracked target A feature map and the initial template B feature map, and they have the same spatial dimension (for example, both are H×W two-dimensional feature maps). Each feature map may also have C channels (for example, 64 or 128).

[0185] Fig.18A schematic diagram of calculating the similarity between tracking results and initial template features in an application scenario of an embodiment of the present application is shown.

[0186] like Fig.18 As shown, in order to calculate the correlation between the two feature maps, first take the first vector A of the tracking result feature map in the H and W directions 1,1 Each vector B in the H and W directions of the initial template i,j Calculate the dot product and divide it by the total number of calculations to get the average value, and finally get the value similarity 1,1 The similarity value can also be called a weight.

[0187]

[0188] Next, the similarity values ​​of other vectors in the target feature A and the initial template B are calculated in the same way until a complete H×W similarity graph is obtained. This similarity graph intuitively shows the similarity between each pixel on the tracked target and the corresponding pixel on the initial template. Fig.18 As shown, by observing the similarity graph, we can clearly see which pixels are highly correlated with the target and which pixels are irrelevant to the target or have a low correlation.

[0189] After obtaining the similarity graph, it can be used as a penalty operator to calculate with the tracked results. Specifically, the embodiment of the present application uses each value on the similarity graph as a weight, multiplies it with the pixel value of the corresponding position on the tracked result, and the final result is used as a new dynamic template. In this way, the pixel points belonging to the target will get a higher weight because of their higher similarity, thereby playing a greater role in subsequent calculations; and the non-target pixel points are suppressed because of their lower similarity, reducing the impact on the results.

[0190] In order to verify the technical effect of the technical solution provided by the embodiment of the present application, the embodiment of the present application measured the tracking results on the LaSOT public test set. LaSOT contains a total of 280 videos with an average length of 2448 frames. The most important evaluation indicators on the LaSOT tracking data set are success rate and accuracy. During the tracking process, if the IoU (Intersection over Union) overlap rate between the bounding box output by the tracker and the true bounding box exceeds a certain threshold τ, the tracking is considered successful, and the success rate is the proportion of the number of frames successfully tracked to the total number of frames. This study also compares some trackers and indicates the sources of these trackers. These include CVPR (IEEE International Conference on Computer Vision and Pattern Recognition), ECCV (European International Conference on Computer Vision), ICCV (International Conference on Computer Vision) and IJCAI (International Joint Conference on Artificial Intelligence).

[0191] As shown in Table 1, the success rate of the tracker in the embodiment of the present application has been significantly improved, reaching 68.2%, ranking first among the following algorithms, exceeding SparseTT (66.0%) by 2.2 absolute percentages, and exceeding TrDiMP which also uses Transformer for feature fusion and Stark_ST50 which uses dynamic templates. In terms of normalized accuracy, it exceeds TrDiMP by 4.1 absolute percentages. Although the template update network is added, since the template update network calculation of this solution does not require a large number of parameters, it can still achieve more than 30 fps on the machine (GPU3060Ti, 32G memory), meeting the real-time requirements. The tracking results and robustness of the tracker have been improved to a certain extent.

[0192] Table 1 Tracking results on the LaSOT test set

[0193]

[0194] Fig.19 The accuracy curves of the present application and several related technologies on LaSOT are shown. Fig. 20 The success rate curve of the present application and multiple related technologies on LaSOT is shown. As shown in the figure, the larger the area under the entire curve, the better the tracking effect. As can be seen from the figure, in the LaSOT test set, compared with the tracking algorithms in recent years, the tracker of the embodiment of the present application, referred to as MixAT, ranks first in AUC and normalized accuracy, higher than SparseTT which also uses Transformer feature extraction and fusion. It can be seen that good results have been achieved after adding the spatiotemporal attention network.

[0195] It should be noted that although the steps of the method in the present application are described in a specific order in the drawings, this does not require or imply that the steps must be performed in this specific order, or that all the steps shown must be performed to achieve the desired results. Additionally or alternatively, some steps may be omitted, multiple steps may be combined into one step, and / or one step may be decomposed into multiple steps, etc.

[0196] The following introduces an apparatus embodiment of the present application, which can be used to execute the object recognition method in the above-mentioned embodiment of the present application. Fig.21 The structure block diagram of the object recognition device provided by the embodiment of the present application is schematically shown. Fig.21 As shown, the object recognition device 2100 includes:

[0197] The detection module 2110 is configured to obtain an image to be identified, and perform edge pixel detection on the image to be identified to obtain edge pixel points and non-edge pixel points in the image to be identified; the edge pixel points are pixel points located in the edge area of ​​the content object in the image to be identified, and the non-edge pixel points are pixel points located in the non-edge area of ​​the content object in the image to be identified;

[0198] The selection module 2120 is configured to select a target pooling algorithm that matches the pooling area from a plurality of candidate pooling algorithms according to the distribution ratio of the edge pixels in the pooling area; the pooling area is a local image area collected by moving a pooling kernel with a preset size on the image to be identified;

[0199] The pooling module 2130 is configured to perform pooling processing on the pooling area according to the target pooling algorithm to obtain a feature map corresponding to the image to be recognized;

[0200] The positioning module 2140 is configured to locate the distribution position of the target object to be identified in the image to be identified according to the feature map.

[0201] In some embodiments of the present application, based on the above technical solution, the selection module 2120 includes:

[0202] A first statistical module is configured to count the distribution ratio of the edge pixels in the image to be identified to obtain the overall area ratio of the edge pixels;

[0203] A second statistical module is configured to count the distribution ratio of the edge pixels in the pooling area to obtain the local area ratio of the edge pixels;

[0204] The algorithm selection module is configured to select a target pooling algorithm that matches the pooling area from a plurality of candidate pooling algorithms based on a numerical relationship between the overall area ratio and the local area ratio.

[0205] In some embodiments of the present application, based on the above technical solution, the candidate pooling algorithms include a maximum pooling algorithm, an average pooling algorithm and a hybrid pooling algorithm, and the hybrid pooling algorithm is a pooling algorithm formed by combining the maximum pooling algorithm and the average pooling algorithm; the algorithm selection module is further configured as: when the local area proportion is greater than the overall area proportion, the maximum pooling algorithm is selected as the target pooling algorithm matching the pooling area; when the local area proportion is less than the overall area proportion and greater than zero, the hybrid pooling algorithm is selected as the target pooling algorithm matching the pooling area; when the local area proportion is equal to zero, the average pooling algorithm is selected as the target pooling algorithm matching the pooling area.

[0206] In some embodiments of the present application, based on the above technical solution, the pooling module 2130 is further configured to: perform pooling processing on the pooling area according to the maximum pooling algorithm to obtain a first pooling eigenvalue; perform pooling processing on the pooling area according to the uniform pooling algorithm to obtain a second pooling eigenvalue; use the local area proportion as a weighting coefficient, perform a weighted operation on the first pooling eigenvalue and the second pooling eigenvalue, and obtain a mixed pooling eigenvalue of the pooling area.

[0207] In some embodiments of the present application, based on the above technical solution, the pooling module 2130 is further configured to: perform pooling processing on the pooling area according to the maximum pooling algorithm to obtain a first pooling eigenvalue; perform pooling processing on the pooling area according to the uniform pooling algorithm to obtain a second pooling eigenvalue; and take the local area proportion as the selection probability, and randomly select a eigenvalue from the first pooling eigenvalue and the second pooling eigenvalue as the mixed pooling eigenvalue of the pooling area.

[0208] In some embodiments of the present application, based on the above technical solution, the positioning module 2140 includes:

[0209] A template acquisition module is configured to acquire a feature template corresponding to a target object to be identified;

[0210] a feature fusion module configured to perform feature fusion processing on the feature template and the candidate region in the feature map to obtain a response map of the candidate region, wherein the response map is used to represent the similarity between the candidate region and the feature template;

[0211] The object positioning module is configured to locate the candidate area with the highest similarity to the feature template as the distribution position of the target object in the image to be identified according to the response map.

[0212] In some embodiments of the present application, based on the above technical solution, the feature template includes a pre-configured static template and a dynamic template dynamically updated during the object recognition process; the feature fusion module is further configured to: perform feature fusion processing on the static template and the candidate area in the feature map to obtain a first fusion feature with spatial information; perform feature fusion processing on the dynamic template and the candidate area in the feature map to obtain a second fusion feature with time information; perform convolution operation on the first fusion feature and the second fusion feature to obtain a response map of the candidate area.

[0213] In some embodiments of the present application, based on the above technical solution, the object recognition device 2100 also includes an update module, which is configured to: crop target features from the feature map according to the distribution position of the target object in the image to be recognized; obtain the feature similarity between the target feature and the static template; and update the dynamic template according to the feature similarity.

[0214] In some embodiments of the present application, based on the above technical solution, the feature similarity includes the similarity value at each pixel point; the dynamic template is updated according to the feature similarity, and the update module is further configured to: use the similarity value at each similar point as a weight, perform weighted operation on each pixel value of the dynamic template, and obtain an updated dynamic template.

[0215] The specific details of the object recognition device provided in each embodiment of the present application have been described in detail in the corresponding method embodiment and will not be repeated here.

[0216] Fig. 22 The structure block diagram of a computer system for implementing an electronic device according to an embodiment of the present application is schematically shown.

[0217] It should be noted that Fig. 22 The computer system 2200 of the electronic device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0218] like Fig. 22 As shown, the computer system 2200 includes a central processing unit 2201 (CPU), which can perform various appropriate actions and processes according to the program stored in the read-only memory 2202 (ROM) or the program loaded from the storage part 2208 to the random access memory 2203 (RAM). Various programs and data required for system operation are also stored in the random access memory 2203. The central processing unit 2201, the read-only memory 2202 and the random access memory 2203 are connected to each other through a bus 2204. An input / output interface 2205 (Input / Output interface, i.e., I / O interface) is also connected to the bus 2204.

[0219] The following components are connected to the input / output interface 2205: an input section 2206 including a keyboard, a mouse, etc.; an output section 2207 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker; a storage section 2208 including a hard disk, etc.; and a communication section 2209 including a network interface card such as a LAN card, a modem, etc. The communication section 2209 performs communication processing via a network such as the Internet. A drive 2210 is also connected to the input / output interface 2205 as needed. A removable medium 2211, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 2210 as needed so that a computer program read therefrom is installed into the storage section 2208 as needed.

[0220] In particular, according to an embodiment of the present application, the process described in each method flow chart can be implemented as a computer software program. For example, an embodiment of the present application includes a computer program product, which includes a computer program carried on a computer readable medium, and the computer program contains a program code for executing the method shown in the flow chart. In such an embodiment, the computer program can be downloaded and installed from the network through the communication part 2209, and / or installed from the removable medium 2211. When the computer program is executed by the central processor 2201, various functions defined in the system of the present application are executed.

[0221] It should be noted that the computer-readable medium shown in the embodiment of the present application may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, a system, device or device of electricity, magnetism, light, electromagnetic, infrared, or semiconductor, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM), a flash memory, an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present application, a computer-readable storage medium may be any tangible medium containing or storing a program, which may be used by an instruction execution system, device or device or used in combination with it. In the present application, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, wherein a computer-readable program code is carried. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. Computer readable signal media may also be any computer readable medium other than computer readable storage media, which may send, propagate, or transmit programs for use by or in conjunction with an instruction execution system, apparatus, or device. The program code contained on the computer readable medium may be transmitted using any suitable medium, including but not limited to: wireless, wired, etc., or any suitable combination of the above.

[0222] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present application. In this regard, each box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the above-mentioned module, program segment or a part of a code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flow chart, and the combination of the boxes in the block diagram or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0223] It should be noted that, although several modules or units of the equipment for action execution are mentioned in the above detailed description, this division is not mandatory. In fact, according to the embodiments of the present application, the features and functions of two or more modules or units described above can be embodied in one module or unit. On the contrary, the features and functions of one module or unit described above can be further divided into being embodied by multiple modules or units.

[0224] Through the description of the above implementation methods, it is easy for those skilled in the art to understand that the example implementation methods described here can be implemented by software or by combining software with necessary hardware. Therefore, the technical solution according to the implementation methods of the present application can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash drive, a mobile hard disk, etc.) or on a network, and includes several instructions to enable a computing device (which can be a personal computer, a server, a touch terminal, or a network device, etc.) to execute the method according to the implementation methods of the present application.

[0225] Those skilled in the art will readily appreciate other embodiments of the present application after considering the specification and practicing the invention disclosed herein. The present application is intended to cover any modification, use or adaptation of the present application, which follows the general principles of the present application and includes common knowledge or customary technical means in the art that are not disclosed in the present application.

[0226] It should be understood that the present application is not limited to the precise structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present application is limited only by the appended claims.

Claims

1. An object recognition method, characterized in that: include: Acquire an image to be identified, and perform edge pixel detection on the image to be identified to obtain edge pixel points and non-edge pixel points in the image to be identified; The edge pixel points are pixel points located in the edge area of ​​the content object in the image to be identified, and the non-edge pixel points are pixel points located in the non-edge area of ​​the content object in the image to be identified; According to the distribution ratio of the edge pixels in the pooling area, a target pooling algorithm matching the pooling area is selected from a plurality of candidate pooling algorithms; the pooling area is a local image area collected by moving a pooling kernel with a preset size on the image to be identified; Performing pooling processing on the pooling area according to the target pooling algorithm to obtain a feature map corresponding to the image to be identified; The distribution position of the target object to be identified in the image to be identified is located according to the feature map.

2. The object recognition method according to claim 1, characterized in that: According to the distribution ratio of the edge pixels in the pooling area, a target pooling algorithm matching the pooling area is selected from a plurality of candidate pooling algorithms, including: Counting the distribution ratio of the edge pixels in the image to be identified to obtain the overall area ratio of the edge pixels; Counting the distribution ratio of the edge pixels in the pooling area to obtain the local area ratio of the edge pixels; According to the numerical relationship between the overall area ratio and the local area ratio, a target pooling algorithm matching the pooling area is selected from a plurality of candidate pooling algorithms.

3. The object recognition method according to claim 2, characterized in that: The candidate pooling algorithms include a maximum pooling algorithm, an average pooling algorithm and a hybrid pooling algorithm, wherein the hybrid pooling algorithm is a pooling algorithm formed by combining the maximum pooling algorithm and the average pooling algorithm; according to the numerical relationship between the overall area proportion and the local area proportion, a target pooling algorithm matching the pooling area is selected from a plurality of candidate pooling algorithms, including: When the proportion of the local area is greater than the proportion of the overall area, selecting the maximum pooling algorithm as the target pooling algorithm matching the pooling area; When the proportion of the local area is less than the proportion of the overall area and greater than zero, the hybrid pooling algorithm is selected as the target pooling algorithm matching the pooling area; When the local area ratio is equal to zero, the average pooling algorithm is selected as the target pooling algorithm matching the pooling area.

4. The object recognition method according to claim 3, characterized in that: When the hybrid pooling algorithm is selected as the target pooling algorithm matching the pooling region, the pooling region is pooled according to the target pooling algorithm, including: Performing pooling processing on the pooling area according to the maximum pooling algorithm to obtain a first pooling eigenvalue; Performing pooling processing on the pooling area according to the uniform pooling algorithm to obtain a second pooling eigenvalue; The proportion of the local area is used as a weighting coefficient to perform a weighted operation on the first pooling eigenvalue and the second pooling eigenvalue to obtain a mixed pooling eigenvalue of the pooling area.

5. The object recognition method according to claim 3, characterized in that: When the hybrid pooling algorithm is selected as the target pooling algorithm matching the pooling region, the pooling region is pooled according to the target pooling algorithm, including: Performing pooling processing on the pooling area according to the maximum pooling algorithm to obtain a first pooling eigenvalue; Performing pooling processing on the pooling area according to the uniform pooling algorithm to obtain a second pooling eigenvalue; Taking the local area proportion as the selection probability, a eigenvalue is randomly selected from the first pooling eigenvalue and the second pooling eigenvalue as the mixed pooling eigenvalue of the pooling area.

6. The object recognition method according to any one of claims 1 to 5, characterized in that: Locating the distribution position of the target object to be identified in the image to be identified according to the feature map includes: Acquire a feature template corresponding to the target object to be identified; Performing feature fusion processing on the feature template and the candidate region in the feature map to obtain a response map of the candidate region, wherein the response map is used to represent the similarity between the candidate region and the feature template; According to the response map, the candidate region with the highest similarity to the feature template is located as the distribution position of the target object in the image to be identified.

7. The object recognition method according to claim 6, characterized in that: The feature templates include pre-configured static templates and dynamic templates dynamically updated during the object recognition process; Performing feature fusion processing on the feature template and the candidate area in the feature map, including: Performing feature fusion processing on the static template and the candidate area in the feature map to obtain a first fusion feature with spatial information; Performing feature fusion processing on the dynamic template and the candidate area in the feature map to obtain a second fusion feature with time information; A convolution operation is performed on the first fused feature and the second fused feature to obtain a response map of the candidate region.

8. The object recognition method according to claim 7, characterized in that: After locating the distribution position of the target object to be identified in the image to be identified according to the feature map, the method further includes: According to the distribution position of the target object in the image to be identified, clipping the target feature from the feature map; Obtaining feature similarity between the target feature and the static template; The dynamic template is updated according to the feature similarity.

9. The object recognition method according to claim 8, characterized in that: The feature similarity includes the similarity value at each pixel point; Updating the dynamic template according to the feature similarity includes: The similarity value at each similar point is used as a weight to perform a weighted operation on each pixel value of the dynamic template to obtain an updated dynamic template.

10. An electronic device, characterized in that: include: processor; as well as A memory, configured to store executable instructions of the processor; The processor is configured to execute the executable instructions to implement the object recognition method as described in any one of claims 1 to 9.

Images