Object recognition methods, apparatus, computer equipment, and storage media

Abstract

This application provides an object recognition method, apparatus, computer device, and storage medium. The method includes: acquiring a target object image containing an object to be identified; performing masking processing on the target object image using an object detection bounding box of the object to be identified to obtain a pseudo-mask image of the target object image; performing feature extraction processing on the pseudo-mask image to obtain a pseudo-mask feature map; the feature value of any pixel in the pseudo-mask feature map is used to indicate the importance of the pixel in the target object image; generating a target fusion feature vector based on the pseudo-mask feature map and the target object image; and performing category recognition on the object to be identified based on the target fusion feature vector to obtain a category recognition result. By using the pseudo-mask image of the object image, the region where the object is located in the object image can be focused on, thereby more accurately identifying the category of the object in the object image.

Description

Technical Field

[0001] This application relates to the field of Internet technology, and more particularly to the field of computer technology, and more particularly to an object recognition method, apparatus, computer device, and storage medium. Background Technology

[0002] With the continuous development of internet technology, the application fields of object recognition are becoming increasingly widespread. Object recognition typically refers to using computers to process, analyze, and understand images in order to identify the object categories contained within them. Currently, object recognition is usually performed directly using the input image to obtain the corresponding object category. However, the accuracy of the object recognition results is closely related to the information contained in the input image. Therefore, how to better utilize the input image to ensure the accuracy of object recognition has become a hot research topic. Summary of the Invention

[0003] This application provides an object recognition method, apparatus, computer device, and storage medium, which can focus on the area where the object is located in the object image by using a pseudo-mask image of the object image, so as to more accurately identify the category of the object in the object image.

[0004] On one hand, embodiments of this application provide an object recognition method, the method comprising:

[0005] Obtain an image of the target object containing the object to be identified;

[0006] The target object image is masked using the object detection bounding box of the object to be identified to obtain a pseudo mask image of the target object image;

[0007] The pseudo-mask image is subjected to feature extraction processing to obtain a pseudo-mask feature map; the feature value of any pixel in the pseudo-mask feature map is used to indicate the importance of any pixel in the target object image;

[0008] Based on the pseudo-mask feature map and the target object image, a target fusion feature vector is generated; and the object to be identified is classified according to the target fusion feature vector to obtain the classification result.

[0009] On the other hand, embodiments of this application provide an object recognition device, the device comprising:

[0010] The acquisition unit is used to acquire an image of a target object containing the object to be identified.

[0011] The masking unit is used to perform masking processing on the target object image using the object detection box of the object to be identified, so as to obtain a pseudo mask image of the target object image;

[0012] An extraction unit is used to perform feature extraction processing on the pseudo-mask image to obtain a pseudo-mask feature map; the feature value of any pixel in the pseudo-mask feature map is used to indicate the importance of any pixel in the target object image;

[0013] The recognition unit is used to generate a target fusion feature vector based on the pseudo mask feature map and the target object image; and to perform category recognition on the object to be recognized according to the target fusion feature vector to obtain the category recognition result.

[0014] In another aspect, embodiments of this application provide a computer device, the computer device including an input device and an output device, and the computer device further includes:

[0015] A processor, adapted to implement one or more instructions; and,

[0016] A computer storage medium storing one or more instructions adapted for loading by the processor and executing the following steps:

[0017] Obtain an image of the target object containing the object to be identified;

[0018] The target object image is masked using the object detection bounding box of the object to be identified to obtain a pseudo mask image of the target object image;

[0019] The pseudo-mask image is subjected to feature extraction processing to obtain a pseudo-mask feature map; the feature value of any pixel in the pseudo-mask feature map is used to indicate the importance of any pixel in the target object image;

[0020] Based on the pseudo-mask feature map and the target object image, a target fusion feature vector is generated; and the object to be identified is classified according to the target fusion feature vector to obtain the classification result.

[0021] In another aspect, embodiments of this application provide a computer storage medium storing one or more instructions, which are adapted to be loaded by the processor and executed as follows:

[0022] Obtain an image of the target object containing the object to be identified;

[0023] The target object image is masked using the object detection bounding box of the object to be identified to obtain a pseudo mask image of the target object image;

[0024] The pseudo-mask image is subjected to feature extraction processing to obtain a pseudo-mask feature map; the feature value of any pixel in the pseudo-mask feature map is used to indicate the importance of any pixel in the target object image;

[0025] Based on the pseudo-mask feature map and the target object image, a target fusion feature vector is generated; and the object to be identified is classified according to the target fusion feature vector to obtain the classification result.

[0026] In another aspect, embodiments of this application provide a computer program product, which includes a computer program; the computer program is stored in a computer-readable storage medium, and when the computer program is executed by the processor of a computer device, it performs the above-described object identification method.

[0027] In this embodiment, a computer device can acquire a target object image containing the object to be identified, and perform masking processing on the target object image using the object detection bounding box of the object to be identified to obtain a pseudo-mask image of the target object image. Then, feature extraction processing can be performed on the pseudo-mask image to obtain a pseudo-mask feature map, wherein the feature value of any pixel in the pseudo-mask feature map can be used to indicate the importance of any pixel in the target object image. Further, based on the pseudo-mask feature map and the target object image, a target fusion feature vector is generated, and the object to be identified is classified according to the target fusion feature vector to obtain the classification result. By implementing the above method, a corresponding pseudo-mask image can be generated using the object detection bounding box of the object in the object image, and by combining the object image and the pseudo-mask image of the object image, the region of interest (i.e., the area where the object is located) in the object image can be focused on, while suppressing the interference of background information in the object image, thereby effectively improving the accuracy of object classification. Attached Figure Description

[0028] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0029] To more clearly illustrate the technical solutions of the embodiments of this application, the drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0030] Figure 1This is a schematic diagram of the architecture of an object recognition system provided in an embodiment of this application;

[0031] Figure 2 This is a flowchart illustrating an object recognition method provided in an embodiment of this application;

[0032] Figure 3a This is a schematic diagram of a structure for determining a target object image provided in an embodiment of this application;

[0033] Figure 3b This is a schematic diagram of a structure for determining a target object image provided in an embodiment of this application;

[0034] Figure 4a This is a coordinate diagram illustrating how to determine the coordinate position of an object detection box in an image of a target object, as provided in an embodiment of this application.

[0035] Figure 4b This is a schematic diagram of the structure of a pseudo-mask image provided in an embodiment of this application;

[0036] Figure 4c This is a schematic diagram of the structure of a pseudo-mask image provided in an embodiment of this application;

[0037] Figure 5a This is a schematic diagram of the structure of an object recognition model provided in an embodiment of this application;

[0038] Figure 5b This is a schematic diagram of the structure of an object recognition model provided in an embodiment of this application;

[0039] Figure 5c This is a schematic diagram of the structure of an object recognition model provided in an embodiment of this application;

[0040] Figure 5d This is a schematic diagram of the structure of an object recognition model provided in an embodiment of this application;

[0041] Figure 6a This is a schematic diagram of the structure of an object recognition model provided in an embodiment of this application;

[0042] Figure 6b This is a schematic diagram of the structure of an object recognition model provided in an embodiment of this application;

[0043] Figure 6c This is a flowchart illustrating an object recognition method provided in an embodiment of this application;

[0044] Figure 6d This is a schematic diagram of a structure for determining a cross-correlation matrix provided in an embodiment of this application;

[0045] Figure 6eThis is a schematic diagram of a structure for determining a cross-correlation matrix provided in an embodiment of this application;

[0046] Figure 7a This is a schematic diagram of the structure of an object recognition model provided in an embodiment of this application;

[0047] Figure 7b This is a flowchart illustrating an object recognition method provided in an embodiment of this application;

[0048] Figure 8 This is a schematic diagram of the structure of an object recognition device provided in an embodiment of this application;

[0049] Figure 9 This is a schematic diagram of the structure of a computer device provided in an embodiment of this application. Detailed Implementation

[0050] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0051] Artificial intelligence (AI) is the theory, methods, technology, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to achieve optimal results. In other words, AI is a comprehensive technology within computer science that attempts to understand the essence of intelligence and produce a new kind of intelligent machine that can react in a way similar to human intelligence. AI studies the design principles and implementation methods of various intelligent machines, enabling them to possess the functions of perception, reasoning, and decision-making.

[0052] Artificial intelligence (AI) is a comprehensive discipline encompassing a wide range of fields, including both hardware and software technologies. Fundamental AI technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing, operating / interactive systems, and mechatronics. AI software technologies primarily include computer vision, speech processing, natural language processing, and machine learning / deep learning.

[0053] Computer vision (CV) is a science that studies how to enable machines to "see." More specifically, it refers to machine vision, which uses cameras and computers to replace human eyes in recognizing, tracking, and measuring targets, and then performs image processing to create images more suitable for human observation or transmission to instruments. As a scientific discipline, computer vision studies related theories and technologies, attempting to build artificial intelligence systems capable of extracting information from images or multidimensional data. Computer vision technologies typically include image processing, image recognition, image semantic understanding, image retrieval, OCR, video processing, video semantic understanding, video content / behavior recognition, 3D object reconstruction, 3D technology, virtual reality, augmented reality, simultaneous localization and mapping (SLAM), and common biometric recognition technologies such as facial recognition and fingerprint recognition.

[0054] Machine Learning (ML) is a multidisciplinary field involving probability theory, statistics, approximation theory, convex analysis, and algorithm complexity theory. It specifically studies how computers can simulate or implement human learning behavior to acquire new knowledge or skills and reorganize existing knowledge structures to continuously improve their performance. Machine learning is the core of artificial intelligence and the fundamental way to endow computers with intelligence; its applications span all areas of artificial intelligence. Machine learning / deep learning typically includes techniques such as artificial neural networks, belief networks, reinforcement learning, transfer learning, inductive learning, and instructional learning.

[0055] Based on the computer vision and machine learning technologies mentioned in the aforementioned artificial intelligence technologies, this application proposes an object recognition model (i.e., a model for identifying the category of objects in an image). This object recognition model can learn the image features of the region containing the object in the object image by using a fake mask, thus accurately identifying the category of the object in the image; where the fake mask refers to a mask generated based on the object detection bounding box of the object in the object image. Furthermore, this application also proposes an object recognition scheme based on this object recognition model; specifically, the general principle of the scheme is as follows: For a target object image containing the object to be identified, a fake mask image of the target object image can be obtained first; the fake mask image refers to the image obtained by masking the target object image using the object detection bounding box of the object to be identified. Then, the object recognition model can be called to perform feature extraction processing on the fake mask image to obtain a fake mask feature map; and then, the object recognition model can be called to perform category identification on the object to be identified based on the target fusion feature vector generated from the fake mask feature map and the target object image, obtaining the category identification result of the object to be identified. Specifically, the object recognition model can perform category recognition on the object to be recognized based solely on the target fusion feature vector to obtain the category recognition result of the object to be recognized; or, it can first call the object recognition model to extract features from the target object image to obtain the target image feature vector of the target object image, so that the object recognition model can perform category recognition on the object to be recognized based on the target fusion feature vector and the target image feature vector to obtain the category recognition result of the object to be recognized.

[0056] It should be noted that: ① When the object recognition model needs to obtain the category recognition result based on the target fusion feature vector and the target image feature vector, the object recognition model can include two branch networks; one branch network can be an image branch network for correlation processing of the target object image, and the other branch network can be a mask branch network for correlation processing of the pseudo-mask image. By setting two branch networks in the object recognition model, the target object image and the pseudo-mask image can be processed in parallel through these two branch networks, thereby improving the efficiency of object recognition. ② The target object image mentioned in the above scheme can be the actual object image involved in the model inference process (i.e., the model application process) of the object recognition model, or it can be a sample object image used to optimize the object recognition model; there is no limitation on this. When the target object image is an actual object image, the processing flow of the above object recognition scheme can be understood as the model inference process of the object recognition model; then, in this case, after obtaining the category recognition result of the object to be recognized, the category recognition result can be output. When the target object image is a sample object image, the processing flow of the above object recognition scheme can be understood as the forward computation process involved in the model optimization process of the object recognition model (i.e., the computation process before calculating the model loss value). In this case, after obtaining the category recognition result of the object to be recognized, the model loss value of the object recognition model can be calculated based on the category recognition result. The object recognition model can then be optimized based on the model loss value. This allows the optimized object recognition model to be called to obtain the category to which the object in the object image belongs when it is actually needed to perform category recognition on an object in an object image.

[0057] In specific implementations, the executing entity of the object recognition scheme mentioned above can be a computer device, which can be a terminal or a server. The terminal mentioned here can be a smartphone, tablet, laptop, desktop computer, or other devices, or peripheral devices such as a gamepad or touchscreen; the server can be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms, etc. It should be noted that when the computer device is a server, this application embodiment provides an object recognition system, such as... Figure 1As shown, the object recognition system includes at least one terminal and at least one server; the terminal can acquire a target object image containing the object to be identified, and upload the acquired target object image containing the object to be identified to the server (i.e., computer device), so that the computer device can obtain the target object image and determine the pseudo mask image of the target object image based on the target object image for subsequent category recognition.

[0058] Alternatively, the object recognition scheme mentioned above can also be executed jointly by the terminal and the server. For example, after acquiring a target object image containing the object to be identified, the terminal can determine a pseudo-mask image of the target object image; then, the acquired target object image and its pseudo-mask image are uploaded to the server. This allows the server to directly perform feature extraction processing on the pseudo-mask image, fusion processing of the mask feature map and the target object image, and category recognition processing of the object to be identified. It should be noted that when the object recognition scheme is executed jointly by the terminal and the server, the terminal and the server can be directly or indirectly connected via wired or wireless communication, and this embodiment of the application does not impose any limitations on this.

[0059] Practice has shown that the object recognition scheme proposed in this application can have the following beneficial effects: it can generate a corresponding pseudo-mask image by using the object detection box of the object in the object image, and combine the object image and the pseudo-mask image of the object image to guide the object recognition model to focus on the region of interest (i.e. the region where the object is located) in the object image, while suppressing the interference of background information in the object image, so as to improve the object recognition ability and thus improve the accuracy of object category recognition.

[0060] Based on the object recognition scheme provided above, this application provides an object recognition method, which can be executed by the aforementioned computer device (i.e., a terminal or a server); or, the object recognition method can be executed jointly by a terminal and a server. For ease of explanation, this application uses the example of a computer device executing the object recognition method, and the target object image being an actual object image involved in the object recognition model's inference process (i.e., the model application process). Please refer to... Figure 2 The object identification method may include the following steps S201-S205:

[0061] S201, Obtain the target object image containing the object to be identified.

[0062] The object to be identified refers to any object that needs to be categorized. Specifically, the object to be identified can be a defect in a component produced during the manufacturing process; in this case, the target object image can be an image containing the component defect. For example, the product could be a computer, tablet, mobile phone, or digital audio player, and the object to be identified could be scratches, wear, blemishes, etc., on these products. Alternatively, the object to be identified can be any body part of any living organism; in this case, the target object image can be an image of that body part. For example, if any body part is a face, the target object image could be a facial image containing the face, such as a human face image or a cat face image; similarly, if any living organism is a human and any body part is a hand, the target object image could be an image of a hand gesture, and so on. Alternatively, the object to be identified can be a vehicle's license plate number, vehicle model, or other objects; in this case, the target object image could be a vehicle image containing a license plate number or a vehicle image containing a vehicle model. It should be noted that this is only an example of several objects to be identified, and is not an exhaustive list.

[0063] In the specific implementation process, the original image containing the object to be identified can be acquired first. The original image refers to an image obtained by photographing the object. For example, in the manufacturing process of a product, a photograph can be taken of the product produced at a certain production station, and the resulting photograph can be the original image. Here, a production station can be understood as a location with multiple production steps in the product manufacturing process, and each production step corresponds to a production station. After acquiring the original image, the computer device can use any of the following specific implementation methods to obtain an image of the target object containing the object to be identified based on the original image:

[0064] Specific implementation 1: The original image can be directly used as the target object image containing the object to be identified.

[0065] Implementation Method 2: Considering that subsequent object recognition will be performed using an object recognition model, and that this model typically has requirements regarding the shape of the input image, the image shape adapted by the object recognition model can be referred to as the target image shape. The method then determines whether the shape of the original image matches the target image shape. If they match, the original image can be used as the target object image; otherwise, the original image can be reshaped based on the target image shape to ensure that the reshaped original image matches the target image shape, thus making the reshaped original image the target object image.

[0066] Specific Implementation 3: Considering that the original image may include background images in addition to the object to be identified, directly obtaining the target object image based on the original image for subsequent category recognition may affect the accuracy of category recognition due to interference from the background image. Therefore, the computer device can first perform object detection on the original image to obtain the object detection result, which may include the object detection box of the object to be identified. The object detection box can indicate the location of the object to be identified; it can be a rectangular bounding box or an irregular detection box, such as an irregular polygonal detection box. This application mainly uses a rectangular object detection box as an example for illustration. Then, the original image is cropped using the object detection box to relatively reduce the background image in the original image, thereby reducing background interference caused by excessive background information when using the object recognition model for category recognition, thus improving the object recognition capability. It should be noted that the embodiments of this application do not limit the object detection method used for the original image; for example, the object detection can be implemented using artificial intelligence, that is, the object detection model can be called to perform object detection on the original image to obtain the object detection result of the original image. The method of cropping the original image using object detection boxes can include any of the following:

[0067] Cropping Method 1: Use all images in the original image except for the object detection box as the background image. You can randomly crop part of the background image from the original image, and then adjust the shape of the cropped original image to match the target image shape that the object recognition model can use. The adjusted original image is then used as the target object image.

[0068] Cropping Method Two: Based on the target image shape and object detection bounding box adapted by the object recognition model, a target detection bounding box containing the object to be recognized can be determined in the original image. The shape of this target detection bounding box is the same as the target image shape, and its size is larger than that of the object detection bounding box. For example, if the target image shape adapted by the object recognition model is square, then the target detection bounding box shape is also square. Similarly, if the target image shape adapted by the object recognition model is rectangular, then the target detection bounding box shape is also rectangular. Then, the region image contained within the target detection bounding box can be cropped from the original image, and this cropped region image can be used as the target object image.

[0069] One specific implementation for determining the target detection box containing the object to be identified in the original image based on the target image shape adapted by the object recognition model and the object detection box can be as follows: First, according to the target image shape adapted by the object recognition model, the object detection box in the original image is shape-transformed to obtain an initial detection box containing the object to be identified. In one possible implementation, this initial detection box can be directly used as the target detection box. For example, assuming the object detection box in the original image is a rectangle and the target image shape is a square, the rectangle can be transformed into a square based on its longest side. Figure 3a As shown, the rectangle marked by 301 can be an object detection box containing the object to be identified. After the object detection box 301 is transformed, an initial detection box marked by 302 can be obtained, and the initial detection box marked by 302 can be used as the target detection box.

[0070] Furthermore, research has shown that in object recognition processing, a moderate proportion of background area helps in object identification, while too much or too little background area reduces the object recognition ability. Therefore, after obtaining the initial detection box, it can be expanded to increase the background area, and the expanded initial detection box can then be used as the target detection box. The expansion process can be performed by expanding the initial detection box according to a preset expansion ratio. This expansion ratio can be preset; for example, practice has shown that an expansion ratio of 0.5 results in better recognition performance of the final target object image. Figure 3a As shown, after obtaining the initial detection box 302, the initial detection box 302 is further expanded to obtain the target detection box marked by 303.

[0071] For example, see Figure 3b As shown: Image 304 represents the object detection box for a specific product defect in a real-world scenario, and Image 305 is the image region with its shape adjusted and boundaries expanded based on this object detection box, i.e., the target detection box. It can be seen that the final target detection box (i.e., the target object image containing the object to be identified) ensures that the object to be identified (the product defect) has a certain background area and that the object to be identified does not undergo geometric distortion. This effectively guarantees the quality of the input image for the object recognition model, and a higher quality input image also ensures better recognition of the object to be identified in the input image.

[0072] It should be noted that the above is merely an illustrative example of how to determine the target detection box based on the shape of the target image and the object detection box, and is not an exhaustive list. For example, in other embodiments, the aspect ratio of the object detection box can be kept unchanged, and the bounding box of the object detection box can be expanded proportionally to obtain the expanded object detection box. Then, the shape of the expanded object detection box can be adjusted to the shape of the target image, thereby determining the object detection box containing the object to be identified as the target detection box.

[0073] S202, the object detection bounding box of the object to be identified is used to perform masking processing on the target object image to obtain a pseudo mask image of the target object image.

[0074] Image masking can be understood as binarizing an image, making the resulting image appear black and white. In other words, the pseudo-mask image of the target object can be a black and white image. Image binarization typically involves setting the pixel value of each pixel in an image to two values, such as 0 or 255, where 255 can be replaced by 1. In an image, the pixel value of the image region identified as the object can be represented by 255 or 1; while the pixel value of the image region other than the object or the background image region can be represented by 0. Image binarization significantly reduces the amount of data in the image, thus clearly highlighting the outline of objects in the image.

[0075] In one specific implementation, the object detection result involved in step S201 may further include the original position coordinates of the object detection box in the original image. In this application, a specific implementation for determining the pseudo-mask image of the target object image can be: First, the original position coordinates of the object detection box in the original image can be remapped to the target object image to obtain the mapped position coordinates of the object detection box. For example, assuming the original position coordinates of the object detection box in the original image are... To indicate, among which, Represents the minimum x-coordinate of the object detection box. This represents the maximum x-coordinate of the object detection box. Represents the minimum ordinate of the object detection box. This represents the maximum ordinate of the object detection bounding box; the subsequent understanding of position coordinates can be found here. Then, the object detection bounding box in the original image is transformed into a square and expanded proportionally. The resulting target position coordinates of the target detection bounding box are... This is represented by the term. It is understood that this application crops the region of interest according to the object detection box, which is the image region of the object detection box described above. After cropping, the target object image can be obtained. However, the position coordinates of the object detection box in the target object image need to be remapped. In specific implementation, the top-left coordinate of the object detection box can be used as the starting coordinate point (i.e., the origin) for remapping, and the corresponding mapping relationship can be shown in Formula 1:

[0076]

[0077] Where h and w are the height and width of the object detection box, respectively.

[0078] For example, see Figure 4a As shown, assuming rectangle 401 is the original image, rectangle 402 is the object detection box, and rectangle 403 is the target detection box, the image region corresponding to the target detection box 403 is the target object image. Taking the top-left corner of the original image 401 as the starting coordinate point (i.e., the origin), the coordinate system based on the original image 401 can be the x0-y0 coordinate system shown in the figure, which can be called the first coordinate system. Similarly, the coordinate system based on the target object image 403 can be the xy coordinate system shown in the figure, which can be called the second coordinate system, which also takes the top-left corner of the target object image 403 as the origin. The original position coordinates of the object detection box 402 are... and the target position coordinates of the target detection box 403 The relevant coordinate information in the first coordinate system can be as follows: Figure 4a As shown, please refer to the markings in the first coordinate system for details. The original position coordinates of the object detection box 402 in the original image 401 are remapped to the target object image. In other words, the original position coordinates of the object detection box 402 in the original image 401 are remapped to the corresponding second coordinate system in the target object image.

[0079] from Figure 4a As can be seen, when the object detection box 402 is in the second coordinate system, the horizontal distance from the top corner of the object detection box 402 to the origin is length 404, and length 404 is... The minimum x-coordinate of the object detection box 402 in the second coordinate system is The vertical distance from the top corner of the object detection box 402 to the origin is 405, and the length 405 is... The minimum ordinate of the object detection box 402 in the second coordinate system is Since the width and height of the object detection box are w and h respectively, the maximum x-coordinate of the object detection box 402 in the second coordinate system is x. max =x min +w-1, the maximum y-coordinate is y max =y min +h-1. The subtraction of 1 in the maximum x-coordinate and maximum y-coordinate takes into account that an image can be composed of a large number of image sub-blocks. Based on the position of the image sub-block, the coordinates of the front end of the image sub-block can be used as the position of the image sub-block. In this case, a subtraction operation is required when calculating the position coordinates.

[0080] After obtaining the mapped coordinates of the object detection box using the method described above, the positional relationship between the object detection box and each pixel in the target object image can be determined based on these coordinates and the positional coordinates of each pixel. This positional relationship indicates whether a pixel is within the object detection box. Based on this determined positional relationship, the updated pixel values ​​of each pixel in the target object image can be determined, allowing the creation of a pseudo-mask image. Specifically, the pixel values ​​of pixels within the object detection box are updated to valid values, while the pixel values ​​of pixels not within the object detection box are updated to invalid values. Valid values ​​can be represented by 255 or 1, and invalid values ​​by 0, effectively binarizing the target object image. The updated target object image can then serve as the pseudo-mask image. Since the pseudo-mask image is based on the updated pixel values ​​of the target object image, it has the same size as the target object image.

[0081] For example, for a pixel with coordinates (i, j) in a pseudo-mask image, its corresponding pixel value M ij This can be determined by whether the coordinates of the location are within the target detection bounding box. This can be explained mathematically, as shown in Formula 2 for the pixel values ​​of each pixel in the pseudo-mask image:

[0082]

[0083] Among them, M ij Let x be the pixel value of the pixel at coordinates (i, j) in the pseudo-mask image, which is the pixel value after updating the pixel value in the target object image; in x min ≤i<x max ,y min ≤j<y max In the case of M ij=1, in other cases, that is, when the pixel is not located in the object detection box, M ij =0. For example, see Figure 4b As shown: Figure 4b The image marked by 41 is the target object image, and the image marked by 42 is the pseudo-mask image of the target object image. The pseudo-mask image exhibits a clear black and white effect.

[0084] It should be noted that, theoretically, the more accurate the pseudomask image, the better the object recognition model's performance. When constructing the corresponding pseudomask image based on the target object image, the target detection box can be defined using the standard top-left and bottom-right coordinates, or it can be defined using other methods. For example, see... Figure 4c As shown: Image 43 is the target object image, and Image 44 is the target detection box defined using two-point coordinates. This method is also the one described above for determining the target detection box. Other methods may include constructing the target detection box coordinates based on four points in arbitrary directions. This construction method is more refined, as shown in Image 45, which is a target detection box defined using four-point coordinates. Alternatively, a more refined pseudo-mask image can be constructed using more refined polygons. This processing method can further improve the recognition performance of the object recognition model, as shown in Image 46, which is a target detection box defined using polygons. Generally, the more refined the pseudo-mask image, the easier it is for the object recognition model to focus on the region where the object to be recognized is located, and the less interference from background information. Therefore, the category recognition ability of the object recognition model is higher. However, in practice, considering that the cost of labeling a finely detailed mask is too high, and that using a target detection box defined based on two-point coordinates can also effectively improve the recognition performance of the object recognition model, this application mainly uses a target detection box defined by two-point coordinates for explanation.

[0085] S203, perform feature extraction processing on the pseudo-mask image to obtain the pseudo-mask feature map.

[0086] In this context, the feature value of any pixel in the pseudo-mask feature map can be used to indicate the importance of any pixel in the target object image.

[0087] It should be noted that common processing methods for multiple input data of neural network models include direct concatenation and fusion methods based on feature extraction of each input data. In this application, in order to utilize the pseudo-mask image to determine the effective background around the object to be identified and suppress the ineffective background around the object, direct concatenation may not fully utilize the pseudo-mask information in the pseudo-mask image. Therefore, feature extraction processing of the pseudo-mask image can be considered to obtain a pseudo-mask feature map, which can be used for subsequent fusion processing. Here, the pseudo-mask feature map, as well as the target fusion feature vector and category recognition results involved in the subsequent process, are all obtained by calling the object recognition model.

[0088] In summary, the object recognition model may include a mask branch network, which can be used to extract features from a pseudo-mask image to obtain a pseudo-mask feature map. In one possible implementation, the mask branch network may include a second convolutional sub-network, which can then be invoked to perform convolution processing on the pseudo-mask image to soften the pixel values ​​of each pixel in the pseudo-mask image, thereby obtaining the pseudo-mask feature map. This second convolutional sub-network can be a multi-layer convolutional joint nonlinear mapping network. For example, the second convolutional sub-network may specifically consist of the first two residual modules of a ResNet18 network, or it may be composed of other network structures, which are not limited in this application. The second convolutional sub-network can convert the pseudo-mask hard values ​​into weighted floating-point numbers associated with the position coordinates, thus "softening" the hard truncation of the pseudo-mask hard values ​​and facilitating subsequent feature map fusion. The aforementioned pseudo-mask hard values ​​can be understood as the two pixel values ​​included in the pseudo-mask image described above, such as pixel values ​​0 and 1, or pixel values ​​0 and 255.

[0089] S204, fuse the pseudo-mask feature map and the target object image to obtain the target fused feature vector.

[0090] In one specific implementation, the object recognition model may further include an image branch network, which can be used to process the target object image. This image branch network may include a first convolutional sub-network, which performs convolution processing on the target object image to obtain a first feature map of the target object image. After obtaining the first feature map of the target object image, the pseudo-mask feature map and the corresponding first feature map of the target object image can be fused to obtain an intermediate fused feature map. This fusion process can be a dot product operation, that is, using a matrix dot product operation between the first feature map of the target object image and the pseudo-mask feature map to obtain the intermediate fused feature map. After obtaining the intermediate fused feature map, features can be further extracted and vector transformed from it to obtain the target fused feature vector. For example, a feature map can generally be understood as a two-dimensional feature map, and a feature vector as a one-dimensional feature map; therefore, the intermediate fused feature map can be further extracted and its dimensions transformed to obtain a one-dimensional target fused feature vector.

[0091] This application considers that the pseudo-mask feature map has a clear physical meaning, that is, pixels with a valid value (such as 1 or 255) in the pseudo-mask image represent objects located in the object detection box in the target object image. The feature values ​​in the pseudo-mask feature map obtained after "softening" the pseudo-mask image through the second convolutional sub-network can be used as weight coefficients. Therefore, when performing matrix multiplication on the first feature map and the pseudo-mask feature map corresponding to the target object image, the feature values ​​in the pseudo-mask feature map can be used as weights to reflect the importance of the corresponding pixel position of the object to be identified in the target object image. This method can bring the following beneficial effects: ① When optimizing the object recognition model, values ​​with larger weights in the pseudo-mask feature map will be primarily allocated to the object detection box and regions related to the object detection box, while information unrelated to the object detection box can be assigned smaller weights; ② The "softening" operation does not completely exclude background information outside the object detection box in the target object image, but rather assigns certain weights to the background information outside the object detection box, so as to adaptively utilize background information through learning.

[0092] As described above, the mask branch network may include a second convolutional sub-network, and the image branch network may include a first convolutional sub-network. The second convolutional sub-network and the first convolutional network may have the same network structure or different network structures.

[0093] (i) If the second convolutional subnetwork has the same network structure as the first convolutional subnetwork, the second convolutional subnetwork can be directly called to perform convolution processing on the pseudo-mask image to obtain the pseudo-mask feature map. For example, the model structure of the object recognition model mentioned here can be found in [reference needed]. Figure 5a As shown, Figure 5aAs shown, the object recognition model can include an image branch network and a mask branch network. The image branch network can include a first convolutional sub-network, and the mask branch network can include a second convolutional sub-network. After the computer device acquires a target object image containing the object to be recognized and a pseudo-mask image of the target object image, the target object image can be input into the first convolutional sub-network to obtain a first feature map of the target object image; the pseudo-mask image can be input into the second convolutional network to obtain a pseudo-mask feature map. After the first feature map and the pseudo-mask feature map, a matrix multiplication operation can be performed on the first feature map and the pseudo-mask feature map to obtain an intermediate fused feature map.

[0094] (II) If the second convolutional subnetwork and the first convolutional subnetwork have different network structures, the mask branch network may include not only the second convolutional subnetwork but also a first channel adjustment subnetwork. The first channel adjustment subnetwork can be used to adjust the number of channels in the feature map output by the second convolutional subnetwork to ensure that the feature map output by the mask branch network has the same number of channels as the feature map output by the image branch network (i.e., the first feature map). Because the same number of channels ensures that the feature map output by the mask branch network and the feature map output by the image branch network can be directly multiplied, meaning that the pseudo-mask feature map output by the mask branch network and the first feature map output by the image branch network can be fused. Specifically, to ensure that the feature map output by the mask branch network has the same number of channels as the feature map output by the image branch network, the number of convolutional kernels used by the first channel adjustment subnetwork can be equal to the number of channels in the feature map output by the first convolutional subnetwork. The convolutional kernels used can be 1×1 kernels. In this case, the specific implementation process of obtaining the pseudo-mask feature map may include: First, calling the second convolutional sub-network to perform convolution processing on the pseudo-mask image to soften the pixel values ​​of each pixel in the pseudo-mask image and obtain the second feature map; then, using each convolution kernel in the first channel adjustment sub-network to perform convolution operation on the second feature map to adjust the number of channels of the second feature map and obtain the pseudo-mask feature map.

[0095] For example, the model structure of the object recognition model mentioned here can be found in [reference needed]. Figure 5b As shown, Figure 5bAs shown, the object recognition model can include an image branch network and a mask branch network. The image branch network can include a first convolutional sub-network, and the mask branch network can include a second convolutional sub-network and a first channel adjustment sub-network. After the computer device acquires a target object image containing the object to be recognized and a pseudo-mask image of the target object image, the target object image can be input into the first convolutional sub-network to obtain a first feature map of the target object image. The pseudo-mask image can first be input into the second convolutional sub-network to obtain a second feature map, and then the second feature map is input into the first channel adjustment sub-network to obtain a pseudo-mask feature map. After the first feature map and the pseudo-mask feature map, a matrix multiplication operation can be performed on the first feature map and the pseudo-mask feature map to obtain an intermediate fused feature map.

[0096] It is important to note that while the second cropping method mentioned above ensures that the object has a certain background area and prevents geometric distortion, it still presents several problems: First, the outward expansion ratio is difficult to control in order to achieve optimal performance of the object recognition model, requiring continuous experimentation. Second, it may not guarantee that the activated region of the object recognition model is the object of interest; for example, elongated objects are easily interfered with by horizontal or vertical structures in the background area. Therefore, this application proposes an object recognition method based on pseudo-mask images and model optimization based on cross-correlation matrices. At the feature extraction level, this method first extracts features from the pseudo-mask image and the target object image respectively to obtain a pseudo-mask feature map and a first feature map. Then, the feature maps from the two network branches are fused by dot product to further extract features and improve the model's recognition ability. This feature fusion method can bring the following beneficial effects: First, it is insensitive to the outward expansion ratio; second, it can automatically focus on the object to be recognized, reducing interference from background information; and third, it can utilize background information to improve recognition performance.

[0097] S205, perform category identification on the object to be identified based on the target fusion feature vector to obtain the category identification result.

[0098] In one specific implementation, category identification can be performed based on the target fusion feature vector, or it can be based on the target fusion feature vector and the first feature map output by the image branching network. Therefore, the category identification results will differ depending on the data used. For example, when performing category identification based on the target fusion feature vector, see [reference needed]. Figure 5a or Figure 5bAs shown, the intermediate fused feature map can be directly passed through the recognition network, and further features can be extracted and vectors transformed within this network to obtain the target fused feature vector. The corresponding category recognition result can then be obtained based on this target fused feature vector. In the case of category recognition of the object to be identified based on the target fused feature vector and the first feature map, see [reference needed]. Figure 5c or Figure 5d As shown, the intermediate fused feature map and the first feature map can be jointly passed through the recognition network. In this recognition network, the intermediate fused feature map can be further extracted and vector transformed to obtain the target fused feature vector. Then, the corresponding category recognition result can be obtained based on the target fused feature vector and the first feature map.

[0099] The following section will elaborate on the category recognition results obtained when using the target fusion feature vector to perform category recognition on the object to be recognized, and when using the first feature map and the target fusion feature vector to perform category recognition on the object to be recognized.

[0100] (1) When performing category recognition processing on the object to be identified based on the target fusion feature vector, in specific implementation, the object to be identified can be category recognized based on the target fusion feature vector to obtain the second category score of the object to be identified in each of the multiple preset categories. It should be noted that any category score mentioned in the embodiments of this application (such as the second category score mentioned here, the first category score mentioned later) essentially refers to the real value output by the fully connected layer network after the object to be identified is category recognized through the fully connected layer network, and its value range can be -inf (real number) to +inf. The preset category can be a pre-set category for the object to be identified. For example, assuming the object to be identified is a defect in a product, the multiple preset categories can refer to the categories of defects, such as defect category 1, defect category 2, defect category 3, etc. After obtaining the second category score of the object to be identified in each preset category, the target probability of the object to be identified belonging to each preset category can be calculated. The preset category corresponding to the highest target probability is added to the category recognition result of the object to be identified, and the category recognition result is the category of the object to be identified.

[0101] For example, if the above description were expressed using a mathematical formula, see Formula 3 below:

[0102]

[0103] Here, label represents the preset category corresponding to the highest target probability obtained in this case. represents the second category score under the k-th preset category. softmax(·) means that after the softmax function is applied, the second category score can be normalized. After the function is applied, the target probability of each preset category can be obtained. argmax(·) means that after the argmax function is applied, the preset category corresponding to the maximum target probability can be obtained.

[0104] (2) When classifying the object to be identified based on the target fusion feature vector and the first feature map output by the image branch network, in specific implementation, the object to be identified can be classified according to the feature vector corresponding to the first feature map to obtain the first category score of the object to be identified in each of the multiple preset categories. The feature vector corresponding to the first feature map can be obtained by further extracting features and transforming vectors from the first feature map in the recognition network. Furthermore, the object to be identified can be classified according to the target fusion feature vector to obtain the second category score of the object to be identified in each preset category. Then, the first category score and the corresponding second category score of the object to be identified in each preset category are fused to obtain the target category score of the object to be identified in each preset category. By combining the two feature maps, the robustness and accuracy of the object recognition model can be effectively improved.

[0105] The method for fusing the first category score and the corresponding second category score of the object to be identified under each preset category can include any of the following:

[0106] Score fusion processing method one: Score fusion processing can refer to averaging the first category score and the corresponding second category score under each preset category. For example, suppose the first category score of the object to be identified under a certain preset category is logit. a The corresponding second category score is logit b The target category score obtained after score fusion is (logit a +logit b ) / 2.

[0107] Score fusion processing method one: If the importance of the image branch network and the mask branch network is considered, the first category score corresponding to the image branch network and the second category score corresponding to the mask branch network can be weighted separately, and the weighted sum can be the target category score. The weights corresponding to the first category score and the second category score can be preset. For example, if the image branch network is more important than the mask branch network, the weight corresponding to the first category score is greater than the weight corresponding to the second category score.

[0108] Therefore, after obtaining the target category score of the object to be identified under each preset category through any of the above scoring fusion processing methods, the target probability of the object belonging to each preset category can be calculated based on the target category score of the object under each preset category. The preset category corresponding to the highest target probability is then added to the category identification result of the object to be identified, which is the category of the object to be identified. For example, if the above description is expressed mathematically, see Formula 4 below:

[0109]

[0110] in, This represents the score of the first category under the k-th preset category. This represents the second category score under the k-th preset category, where A and B represent the weights corresponding to the first and second category scores, respectively. Both A and B can be 1 / 2. k This represents the target category score under the k-th preset category, and the label represents the preset category corresponding to the maximum target probability obtained in this case.

[0111] In one possible implementation, besides directly fusing the first category score and the corresponding second category score under each preset category as described above, another approach is to first determine the first probability of the object belonging to each preset category based on the first category score, and then determine the second probability based on the second category score. Then, the first probability and the corresponding second probability of the object belonging to each preset category are fused to obtain the target probability of the object in each preset category. Finally, the preset category corresponding to the highest target probability is added to the category recognition result of the object, which then represents the category of the object.

[0112] Similarly, referring to the score fusion processing method described above, the probability fusion processing here can also be based on the weights corresponding to the first probability and the corresponding weights corresponding to the second probability, performing probability fusion processing on the first probability and the corresponding second probability. If the importance of the image branch network is higher than that of the mask branch network, then the weight corresponding to the first probability is greater than the weight corresponding to the second probability. For example, if the above description is expressed mathematically, see Formula 5 below:

[0113]

[0114] Where a_out k b_out represents the first probability under the k-th preset category. k This represents the second probability under the k-th preset category, where A and B represent the weights corresponding to the first and second probabilities, respectively, and both A and B can be 1 / 2; out k This represents the target probability under the k-th preset category, and label represents the preset category corresponding to the maximum target probability obtained in this case.

[0115] In this embodiment, a target object image containing the object to be identified can be acquired, and the target object image can be masked using the object detection bounding box of the object to be identified to obtain a pseudo-mask image of the target object image. Then, feature extraction processing can be performed on the pseudo-mask image to obtain a pseudo-mask feature map. Further, based on the pseudo-mask feature map and the target object image, a target fusion feature vector is generated, and the object to be identified is classified according to the target fusion feature vector to obtain the classification result. By implementing the above method, the corresponding pseudo-mask image can be generated using the object detection bounding box of the object. Combined with the input image and the pseudo-mask image corresponding to the input image, the object recognition model can be guided to focus on the region of interest in the input image, suppressing the interference of background information in the input image, thereby improving the object classification capability. For example, the object recognition method in this application can be applied to various object recognition scenarios for effective object recognition. For example, it can be applied to industrial quality inspection projects, such as the detection of defects in mobile phone camera module mounting brackets, the detection of defects in charging heads, and the classification of Type-C (Type C mainly refers to computer, communication, and consumer electronics products) defects, etc. For example, it can be applied to facial recognition, etc.

[0116] Based on the above description, this application further proposes another object recognition method. In this embodiment, the object recognition method is mainly implemented using a computer device, and the target object image is a sample object image used for model optimization of the object recognition model. Furthermore, the object recognition model mentioned in this embodiment may include an image branch network and a mask branch network; the image branch network may include a first convolutional subnetwork and a first enhancement subnetwork, and the mask branch network may include a second convolutional subnetwork and a second enhancement subnetwork. Wherein:

[0117] For the image branching network: the first convolutional subnetwork can be used to perform convolution processing on the target object image to obtain a first feature map of the target object image; the first enhancement subnetwork can be used to perform feature enhancement processing on the first feature map to obtain a target image feature vector of the target object image. The first enhancement subnetwork can be composed of the last three levels of residual modules of ResNet18, or it can include other network structures, which are not limited in this application. After the first enhancement subnetwork, a fully connected layer network can also be included. The fully connected layer network can obtain the first category score of the object to be identified under each preset category based on the output of the first enhancement subnetwork. The dimension of the fully connected layer network is equal to the number of preset categories.

[0118] Regarding masked branch networks: the second convolutional subnetwork can have the same network structure as the first convolutional subnetwork, or it can have a different network structure. The corresponding object recognition model will also differ depending on the specific situation.

[0119] (i) When the second convolutional subnetwork has a different network structure than the first convolutional subnetwork, the mask branch network can also include a first channel adjustment subnetwork. In this case, the object recognition model can be as follows: Figure 6a As shown. In the mask branch network: the second convolutional subnetwork can be used to extract features from the pseudo-mask image to obtain a second feature map; the first channel adjustment subnetwork can be used to perform convolution operations on the second feature map to obtain a pseudo-mask feature map. The second enhancement subnetwork can be used to perform feature enhancement processing on the intermediate fused feature map obtained from the feature fusion processing of the pseudo-mask feature map and the first feature map to obtain the target fused feature vector.

[0120] The first channel adjustment subnetwork can be used to adjust the number of channels in the second feature map output by the second convolutional subnetwork. This ensures that the adjusted second feature map (i.e., the pseudo-mask feature map) has the same number of channels as the first feature map output by the image branch network. Because of this same number of channels, the pseudo-mask feature map and the first feature map can be directly multiplied, allowing for feature fusion processing to obtain an intermediate fused feature map. To ensure the pseudo-mask feature map and the first feature map have the same number of channels, the number of convolutional kernels used by the first channel adjustment subnetwork can be equal to the number of channels in the feature map output by the first convolutional subnetwork.

[0121] (ii) When the second convolutional subnetwork has the same network structure as the first convolutional subnetwork, the mask branch network may not include the first channel adjustment subnetwork. In this case, the object recognition model can be as follows: Figure 6bAs shown. In the mask branch network: the second convolutional sub-network can be used to extract features from the pseudo-mask image to obtain the pseudo-mask feature map; the second enhancement sub-network can be used to perform feature enhancement processing on the intermediate fused feature map obtained by feature fusion processing of the pseudo-mask feature map and the first feature map to obtain the target fused feature vector.

[0122] It should be noted that the feature extraction or feature enhancement processes performed using various convolutional sub-networks described above can specifically be convolutional processing. Furthermore, the second enhancement sub-network can have the same network structure as the first enhancement sub-network to ensure that the number of channels in the target image feature vector and the target fused feature vector are the same. This same number of channels also ensures that the target image feature vector and the target fused feature vector can be multiplied subsequently to obtain a cross-correlation matrix. Similarly, a fully connected layer network can be included after the second enhancement sub-network. The fully connected layer network can obtain the second category score of the object to be identified under each preset category based on the output of the second enhancement sub-network. The dimension of the fully connected layer network is equal to the number of preset categories.

[0123] based on Figure 6a and Figure 6b The object recognition model shown below, combined with Figure 6c The specific process of the object recognition method proposed in the embodiments of this application is described below. See also... Figure 6c As shown, the object recognition method may include the following steps S601-S608:

[0124] S601, Obtain the target object image containing the object to be identified.

[0125] S602, the object detection bounding box of the object to be identified is used to perform masking processing on the target object image to obtain a pseudo mask image of the target object image.

[0126] S603 performs feature extraction processing on the pseudo-mask image to obtain the pseudo-mask feature map.

[0127] It should be noted that the specific implementation of steps S601-S603 can be found in the relevant descriptions of steps S201-S203 mentioned in the above embodiments, and will not be repeated here.

[0128] S604 generates a target fusion feature vector based on the pseudo-mask feature map and the target object image.

[0129] In one specific implementation, see as follows: Figure 6a and Figure 6bAs shown, feature fusion processing can be performed on the pseudo-mask feature map and the target object image in the mask branch network of the object recognition model to obtain an intermediate fused feature map. After obtaining the intermediate fused feature map, further feature enhancement processing is performed on it in the mask branch network to obtain the target fused feature vector. For example, the second enhancement sub-network in the mask branch network can be called to perform feature enhancement processing on the intermediate fused feature map to obtain the target fused feature vector. In one implementation, when performing feature fusion processing on the pseudo-mask feature map and the target object image, the first convolutional sub-network in the image branch network can be called to perform convolution processing on the target object image to obtain a first feature map of the target object image. This first feature map is then passed from the image branch network to the mask branch network, where the first feature map and the pseudo-mask feature map are fused to obtain the intermediate fused feature map.

[0130] In the mask branch network, when feature enhancement processing is performed on the intermediate fused feature map to obtain the target fused feature vector, feature enhancement processing can also be performed on the first feature map of the target image in the image branch network to obtain the target image feature vector. For example, the first enhancement subnetwork in the image branch network can be called to perform feature enhancement processing on the first feature map to obtain the target image feature vector.

[0131] S605, perform category identification on the object to be identified based on the target fusion feature vector, and obtain the category identification result.

[0132] As mentioned above, when classifying an object, the object can be classified based on the target fusion feature vector, or it can be classified based on both the target fusion feature vector and the target image feature vector. The classification results will also differ depending on the feature vectors used.

[0133] (1) When performing category recognition processing on the object to be identified based on the target fusion feature vector, in specific implementation, the object to be identified can be processed for category recognition based on the target fusion feature vector to obtain the second category score of the object to be identified under each preset category. Then, the category recognition result of the object to be identified is constructed using the second category score of the object to be identified under each preset category.

[0134] (2) When performing category recognition processing on the object to be identified based on the target fusion feature vector and the target image feature vector, in specific implementation, the object to be identified can be classified according to the target image features to obtain the first category score of the object to be identified in each of the multiple preset categories, and the object to be identified can be classified according to the target fusion feature vector to obtain the second category score of the object to be identified in each of the preset categories. After obtaining the first category score and the corresponding second category score in each preset category, based on the different processing of the first category score and the second category score, the constructed category recognition result of the object to be identified can also include the following two cases.

[0135] ① After obtaining the first category score and the corresponding second category score under each preset category, the first category score and the second category score of the object to be identified under each preset category are directly used to construct the category identification result of the object to be identified.

[0136] ② After obtaining the first category score and the corresponding second category score under each preset category, the first category score and the corresponding second category score under each preset category are first fused to obtain the target category score under each preset category. The target category score under each preset category is then used to construct the category recognition result of the object to be identified. The score fusion process of the first category score and the corresponding second category score under each preset category can be referred to the description in step S205, and will not be repeated here.

[0137] After obtaining the category recognition results, the object recognition model can be optimized based on the constructed category recognition results.

[0138] S606, Obtain the label category of the object to be identified.

[0139] In one specific implementation, the label category of the object to be identified can be preset, and this label category can refer to the actual category of the object to be identified. It is understood that this label category is one of the multiple preset categories mentioned above.

[0140] S607, Calculate the model loss value of the object recognition model based on the label category and the category recognition result.

[0141] In one specific implementation, the model loss value of the object recognition model can be calculated based on the label category and the category recognition result. The model parameters of the object recognition model can then be optimized based on the model loss value to obtain an optimized object recognition model. However, the model loss value of the object recognition model will vary depending on the category recognition result. The following section explains how to calculate the model loss value for different categories of category recognition results.

[0142] (1) When the category recognition result includes the second category score of the object to be recognized under each preset category, the first feature loss value of the object recognition model can be calculated based on the label category and the category recognition result. This first feature loss value is the model loss value. The loss function used in calculating the model loss value can be the cross-entropy loss function, as shown in Formula 6:

[0143]

[0144] Among them, L B p represents the model loss value calculated for the target object image, where M represents the number of preset categories. m Let p represent the label category of the sample object image. If the label category of the sample object image is the m-th preset category, indicating that the m-th preset category is the true label, then p m =1, if the label category of the sample object image is not the m-th preset category, that is, if the m-th preset category is not the true label, then p m =0; The score represents the second category score of the sample object image belonging to the m-th preset category.

[0145] In practical model optimization, to improve optimization efficiency, a batch of sample object images can be used to optimize the object recognition model at a time. In this case, for a batch of sample object images, the loss function used when calculating the model loss value can be as shown in Equation 7. Similarly, this loss function is also a cross-entropy loss function.

[0146]

[0147] Among them, L B p represents the model loss value calculated for a batch of sample object images, b represents the number of sample object images in a batch, and p represents the number of sample object images in a batch. im Let p represent the label category of the i-th sample object image, and if the label category of the i-th sample object image is the m-th preset category, then p im =1, if the label category of the i-th sample object image is not the m-th preset category, then p im =0; This represents the second category score of the i-th sample object image belonging to the m-th preset category.

[0148] (2) When the category recognition result includes the first category score and the corresponding second category score of the object to be recognized under each preset category, the feature loss value for the target image feature vector can be calculated based on the first category score under each preset category, and the feature loss value for the target fused feature vector can be calculated based on the second category score under each preset category. Then, the feature loss value for the target image feature vector and the feature loss value for the target fused feature vector are summed to obtain the model loss value of the object recognition model. The calculation of the model loss value can be as shown in Formula 8:

[0149]

[0150] Among them, L A L represents the feature loss value calculated for the feature vector of the target image, given a single sample image; B L represents the feature loss value calculated for the target fused feature vector in the case of a single sample object image; L represents the model loss value in the case of a single sample object image.

[0151] Similarly, in this case, when calculating the model loss value of the object recognition model for a batch of sample object images, the specific calculation method will not be elaborated here, but can be referred to Formula 7 above. It should be noted that the following explanation of calculating the model loss value uses the target object image as an example. The target object image can refer to one of the batch of sample images. However, in the actual model optimization process, it may also include calculating the model loss value for a batch of sample object images. In the description of the corresponding loss value, this will not be described in detail.

[0152] (3) When the category recognition result includes the target category score of the object to be identified under each preset category, the model loss value can be calculated as shown in Formula 9.

[0153]

[0154] Where L represents the model loss value for a single sample image, logit m This represents the target category score of the sample object image belonging to the m-th preset category; A and B represent the weights used in the score fusion process of the first category score and the second category score, respectively.

[0155] In another specific implementation, besides calculating the model loss value of the object recognition model based on the label category and category recognition results described above, a second feature loss value of the object recognition model can also be calculated using the cross-correlation matrix constructed from the target image feature vector and the target fused feature vector, as well as the optimization objective of the cross-correlation matrix. This second feature loss value can also be used as the model loss value of the object recognition model. The cross-correlation matrix can be used to indicate the similarity between the target image feature vector and the target fused feature vector.

[0156] Assume the feature vector of the target image is z A The target fusion feature vector is z B Let the feature dimension of the target image be d. Then the cross-correlation matrix constructed by the target image feature vector and the target fused feature vector is shown in Formula 10.

[0157]

[0158] Where C represents the cross-correlation matrix, and ||·||2 represents the L2 norm; the corresponding calculation diagram can be seen as follows. Figure 6d As shown in Figure 6d, a cross-correlation matrix can be constructed based on the target image feature vector and the target fusion feature vector corresponding to the target object image, and the dimension of the cross-correlation matrix is ​​only related to the feature dimension d.

[0159] In the actual model optimization process, in order to improve optimization efficiency, a batch of sample object images can be used to optimize the object recognition model at one time. In this case, the target object image or the original image can be one of the batch of sample object images, and the computer device can obtain the target image feature vector and the target fusion feature vector of each sample object image based on the aforementioned steps S601 to S604, so that a cross-correlation matrix can be constructed based on the target image feature vector and the target fusion feature vector of each sample object image.

[0160] The following explanation uses a batch of sample object images as an example to illustrate the construction of the cross-correlation matrix. Let the batch size be b and the feature dimension be d; let Z be the target image feature vector matrix extracted from a batch of sample object images. A The target fusion feature vector matrix is ​​Z. B Then the feature vector matrix Z of the target image A The feature vector matrix Z is fused with the target. B The constructed cross-correlation matrix is ​​shown in Formula 11 below.

[0161]

[0162] The corresponding calculation diagram can be found as follows: Figure 6eAs shown in Figure 6e, the dimension of the cross-correlation matrix depends only on the feature dimension d and is independent of the batch size b. The target image feature vector matrix Z... A Each row in the matrix can represent the feature vector of the target image corresponding to a sample object image; similarly, the target fusion feature vector matrix Z... B Each row in the vector can represent the target fusion feature vector corresponding to a sample object image.

[0163] The optimization objective of the cross-correlation matrix is ​​to make the target image feature vector and the target fused feature vector sufficiently similar, meaning that the features of the pseudo-mask feature map should be invariant before and after fusion. Therefore, the features at corresponding positions in the target image feature vector and the target fused feature vector should be sufficiently similar, while the features at non-corresponding positions should have sufficiently large differences to improve feature recognition capability. If we translate the above characteristics into the cross-correlation matrix, the element located diagonally opposite to the target image feature vector is the product of the corresponding elements of the target image feature vector and the target fused feature vector. As mentioned above, the features at corresponding positions in the target image feature vector and the target fused feature vector should be sufficiently similar, meaning the diagonally opposite elements in the cross-correlation matrix should tend to 1; the features at non-corresponding positions should have sufficiently large differences, meaning the off-diagonal elements in the cross-correlation matrix should be sufficiently close to 0. Based on the above description, the loss function used to calculate the second feature loss value of the object recognition model can be shown in Equation 12.

[0164]

[0165] Among them, L Corr C represents the second feature loss value. ii C represents the diagonal elements in the cross-correlation matrix. ij λ represents the off-diagonal elements in the cross-correlation matrix; λ is the adjustment factor. Practice has shown that λ = 1 results in better model performance for object recognition.

[0166] Therefore, after obtaining the second feature loss value using the above method, this second feature loss value can be used as the model loss value of the object recognition model to optimize the model parameters. By constructing a cross-correlation matrix between the target image feature vector and the target fused feature vector, this cross-correlation matrix can be optimized to approach an identity matrix, thereby improving the object feature discrimination ability and thus enhancing the model's recognition capability. Figure 6a or Figure 6b The image corresponding to the cross-correlation matrix shown can be a symbolic representation, and the image corresponding to the optimization objective can also be a symbolic representation. The image corresponding to the optimization objective can be represented by optimizing the cross-correlation matrix so that the cross-correlation matrix eventually converges to the identity matrix.

[0167] In another specific implementation, for ease of subsequent description, the feature loss value calculated based on the label category and category recognition result is referred to as the first feature loss value of the object recognition model, and the feature loss value corresponding to the cross-correlation matrix constructed based on the target image feature vector and the target fused feature vector is referred to as the second feature loss value of the object recognition model. In this application, the model loss value of the object recognition model can be determined by combining the first and second feature loss values. For example, the first and second feature loss values ​​can be summed, and the summation result can be used as the model loss value of the object recognition model. The calculation of the first feature loss value can include any of the cases described above. In this case, by constructing the cross-correlation matrix, the loss function corresponding to the cross-correlation matrix, the loss function corresponding to the image features, and the loss function corresponding to the fused features are jointly optimized to perform multi-dimensional feature learning, thereby improving the model's recognition ability.

[0168] It should be noted that when calculating the first feature loss value of the object recognition model based on the label category and the recognition result of that category, and when calculating the second feature loss value of the object recognition model based on the cross-correlation matrix, it is possible to base it on... Figure 6a and Figure 6b The object recognition model shown is used to calculate, in Figure 6a and Figure 6b In the object recognition model shown, both the image branch network and the fusion branch network have an enhancement subnetwork that enhances the feature maps. If we... Figure 6a and Figure 6b The enhanced subnetworks (first enhanced subnetwork and second enhanced subnetwork) in the object recognition model shown are removed. The model obtained after removal can also calculate the model loss value based on the above-described method for calculating the loss value, and the corresponding object recognition model can be optimized using the model loss value. That is, the optimization of the object recognition model described in the embodiments of this application can also be applied to various object recognition models involved in steps S201-S205, only requiring corresponding parameter adjustments when calculating the model loss value.

[0169] S608, optimize the model parameters of the object recognition model in order to reduce the model loss value.

[0170] In one specific implementation, the model parameters of the object recognition model can be optimized in the direction of reducing the model loss value to obtain an optimized object recognition model. After obtaining the optimized object recognition model, when the computer device detects an object recognition request for the image to be processed, the computer device calls the optimized object recognition model to perform object recognition on the image to obtain the object category of the objects contained in the image.

[0171] In this embodiment, a target object image containing the object to be identified can be acquired, and the target object image can be masked using the object detection bounding box of the object to be identified to obtain a pseudo-mask image of the target object image. Then, feature extraction processing is performed on the pseudo-mask image to obtain a pseudo-mask feature map. Further, a target fusion feature vector can be generated based on the pseudo-mask feature map and the target object image, and the category of the object to be identified can be identified according to the target fusion feature vector to obtain the category identification result. After obtaining the category identification result, the object recognition model can be further optimized according to the category identification result. The label category of the object to be identified can be obtained, and the model loss value of the object recognition model can be calculated based on the label category and the category identification result. The model parameters of the object recognition model can be optimized in the direction of reducing the model loss value. By implementing the above method, a cross-correlation matrix between the target image feature vector and the target fused feature vector can be constructed. This cross-correlation matrix can be optimized to approach an identity matrix, thereby improving the object feature discrimination capability. Furthermore, by jointly optimizing the loss function corresponding to the cross-correlation matrix, the loss function corresponding to the image features, and the loss function corresponding to the fused features through the constructed cross-correlation matrix, multi-dimensional feature learning can be performed to optimize the object recognition model. This effectively improves the accuracy of the object recognition model's category identification and ensures the accuracy of subsequent object category identification using the optimized object recognition model.

[0172] Based on the above description, this application further proposes another object recognition method; in this application embodiment, the target object image is mainly used as a sample object image for model optimization of the object recognition model for illustration; and the model structure of the object recognition model mentioned in this application embodiment can be found in [reference needed]. Figure 7a As shown.

[0173] Based on the idea of ​​dot-multiplication weighting of pseudo-mask feature maps, it is possible to... Figure 6a The model structure was slightly modified to design such as Figure 7a The object recognition model shown may include an image branch network and a mask branch network. The image branch network may include a first enhancement subnetwork, and the mask branch network may include a forward convolution subnetwork, a deconvolution subnetwork, a second channel adjustment subnetwork, and a second enhancement subnetwork.

[0174] Will Figure 7a The object recognition model shown is Figure 6a Compare the object recognition models shown below: Figure 7aThe object recognition model shown omits a first convolutional subnetwork in its image branch network. Instead, a deconvolutional subnetwork is added to the mask branch network. This deconvolutional subnetwork ensures that the width and height of the feature map output by the mask branch network match the width and height of the target object image. Additionally, a second channel adjustment subnetwork is added. The number of convolutional kernels used in this second channel adjustment subnetwork is equal to the number of channels in the target object image. These kernels can be 1×1. The second channel adjustment subnetwork adjusts the number of channels in the output of the deconvolutional subnetwork to ensure that the feature map output by the mask branch network has the same number of channels as the target object image. This same number of channels also allows the feature map output by the mask branch network to be directly multiplied with the target object image, enabling fusion processing between the two. However, if the number of channels in the feature map output by the deconvolutional subnetwork is the same as the number of channels in the target object image, the second channel adjustment subnetwork for channel adjustment may not be included in the mask branch network. Figure 7a The positive convolutional subnetwork in the object recognition model shown can be combined with... Figure 6a The network structure of the second convolutional subnetwork in the object recognition model shown may be the same or different, and this application does not impose any restrictions.

[0175] based on Figure 7a The object recognition model shown below, combined with Figure 7b The specific process of the object recognition method proposed in the embodiments of this application is described below. See also... Figure 7b As shown, the object recognition method may include the following steps S701-S710:

[0176] S701, Obtain the target object image containing the object to be identified.

[0177] S702, the object detection bounding box of the object to be identified is used to perform masking processing on the target object image to obtain a pseudo mask image of the target object image.

[0178] S703, in the mask branch network of the object recognition model, calls the positive convolution sub-network to perform a positive convolution operation on the pseudo mask image to obtain a positive convolution feature map.

[0179] The forward convolution operation on the pseudo-mask image using the forward convolution sub-network is used to soften the pixel values ​​of each pixel in the pseudo-mask image, which is the same as the function of the second convolution sub-network mentioned above.

[0180] S704 calls the deconvolution sub-network to perform a reverse convolution operation on the forward convolution feature map to obtain the deconvolution feature map.

[0181] In this context, the image size of the deconvolution feature map is equal to the image size of the target object image.

[0182] S705 calls the second channel to adjust each convolution kernel in the sub-network, performs convolution processing on the deconvolution feature map, and obtains the pseudo-mask feature map.

[0183] S706 generates a target fusion feature vector based on the pseudo-mask feature map and the target object image.

[0184] In one specific implementation, see as follows: Figure 7a As shown, feature fusion processing can be performed on the pseudo-mask feature map and the target object image in the mask branch network of the object recognition model to obtain an intermediate fused feature map. After obtaining the intermediate fused feature map, further feature enhancement processing is performed on the intermediate fused feature map in the mask branch network to obtain the target fused feature vector.

[0185] In the mask branch network, while performing feature enhancement processing on the intermediate fused feature map to obtain the target fused feature vector, the image branch network can also perform feature enhancement processing on the target object image to obtain the target image feature vector. For example, the first enhancement subnetwork in the image branch network can be called to perform feature enhancement processing on the target object image to obtain the target image feature vector.

[0186] S707: Based on the target fusion feature vector, classify the object to be identified and obtain the class identification result.

[0187] S708, Obtain the label category of the object to be identified.

[0188] S709, based on the label category and the category recognition result, calculate the model loss value of the object recognition model.

[0189] S710 optimizes the model parameters of the object recognition model in order to reduce the model loss value.

[0190] The specific implementation of steps S701 and S702 can be found in the detailed description of steps S201 and S202 in the above embodiment, and the specific implementation of steps S707-S710 can be found in the detailed description of steps S605-S608 in the above embodiment, which will not be repeated here.

[0191] In one specific implementation, the model parameters of the object recognition model can be optimized to reduce the model loss value, resulting in an optimized object recognition model. After obtaining the optimized object recognition model, it can be used for object category identification. Practical experience has shown that using... Figure 7aThe object recognition model shown directly fuses the target object image using a pseudo-mask feature map during category recognition, and then feeds the intermediate fused feature map into the subsequent feature enhancement network and recognition network. Its recognition performance can reach [percentage missing]. Figure 6a or Figure 6b The object recognition model shown exhibits similar recognition performance.

[0192] In this embodiment, the object recognition model based on the variant can also optimize the object recognition model by jointly optimizing the loss function corresponding to the cross-correlation matrix, the loss function corresponding to the image features, and the loss function corresponding to the fused features through the constructed cross-correlation matrix, thereby performing multi-dimensional feature learning. This effectively improves the accuracy of the object recognition model in class identification and ensures the accuracy of subsequent object class identification using the optimized object recognition model.

[0193] Based on the description of the above-described object recognition method embodiments, this application also discloses an object recognition device, which can be a computer program (including program code) running on the aforementioned computer device. The object recognition device can execute... Figure 2 or Figure 6c or Figure 7b The method shown. Please refer to [link / reference]. Figure 8 The object recognition device can operate the following units:

[0194] The acquisition unit 801 is used to acquire an image of a target object containing the object to be identified;

[0195] The masking unit 802 is used to perform masking processing on the target object image using the object detection box of the object to be identified, so as to obtain a pseudo mask image of the target object image;

[0196] The extraction unit 803 is used to perform feature extraction processing on the pseudo-mask image to obtain a pseudo-mask feature map; the feature value of any pixel in the pseudo-mask feature map is used to indicate the importance of any pixel in the target object image;

[0197] The recognition unit 804 is used to generate a target fusion feature vector based on the pseudo mask feature map and the target object image; and to perform category recognition on the object to be recognized according to the target fusion feature vector to obtain the category recognition result.

[0198] In another implementation, the pseudo-mask feature map, the target fusion feature vector, and the category recognition result are all obtained by calling an object recognition model; the acquisition unit 801, when used to acquire a target object image containing the object to be recognized, can specifically be used for:

[0199] Obtain the original image containing the object to be identified;

[0200] Object detection is performed on the original image to obtain the object detection result of the original image, and the object detection result includes the object detection box of the object to be identified;

[0201] Based on the target image shape adapted by the object recognition model and the object detection box, a target detection box containing the object to be recognized is determined in the original image; the shape of the target detection box is the target image shape, and the size of the target detection box is larger than the size of the object detection box.

[0202] The region image contained in the target detection box is cropped from the original image, and the cropped region image is used as the image to be identified.

[0203] In another implementation, the acquisition unit 801, when determining a target detection box containing the object to be identified in the original image based on the target image shape adapted by the object recognition model and the object detection box, may specifically be used for:

[0204] According to the target image shape adapted by the object recognition model, the object detection box is transformed in the original image to obtain an initial detection box containing the object to be recognized.

[0205] The initial detection box is expanded to obtain a target detection box containing the image to be identified.

[0206] In another implementation, the object detection result further includes the original position coordinates of the object detection box in the original image; the mask unit 802, when used to perform masking processing on the target object image using the object detection box of the object to be identified to obtain a pseudo mask image of the target object image, can be specifically used for:

[0207] The original position coordinates of the object detection box in the original image are remapped to the target object image to obtain the mapped position coordinates of the object detection box;

[0208] Based on the mapped position coordinates of the object detection box and the position coordinates of each pixel in the target object image, the positional relationship between the object detection box and each pixel is determined; the positional relationship is used to indicate whether a pixel is located within the object detection box.

[0209] Based on the determined positional relationship, the pixel values ​​of the pixels in the target object image that are located in the object detection box are updated to valid values, and the pixel values ​​of the pixels in the target object image that are not located in the object detection box are updated to invalid values.

[0210] The target object image after pixel value updates is used as the pseudo-mask image of the target object image.

[0211] In another implementation, the recognition unit 804, when generating a target fusion feature vector based on the pseudo-mask feature map and the target object image, may specifically be used for:

[0212] In the mask branch network of the object recognition model, feature fusion processing is performed on the pseudo-mask feature map and the target object image to obtain an intermediate fused feature map;

[0213] In the mask branch network, the intermediate fused feature map is subjected to feature enhancement processing to obtain the target fused feature vector.

[0214] In another implementation, the object recognition model further includes an image branch network, which includes a first convolutional sub-network; the recognition unit 804, when performing feature fusion processing on the pseudo-mask feature map and the target object image in the mask branch network of the object recognition model to obtain an intermediate fused feature map, can specifically be used for:

[0215] The first convolutional sub-network is invoked to perform convolution processing on the target object image to obtain the first feature map of the target object image;

[0216] The first feature map is passed from the image branch network to the mask branch network, and the first feature map and the pseudo mask feature map are fused in the mask branch network to obtain an intermediate fused feature map.

[0217] In another implementation, the mask branch network includes a second convolutional sub-network, and the second convolutional sub-network has the same network structure as the first convolutional network; the recognition unit 804, when performing feature extraction processing on the pseudo-mask image to obtain a pseudo-mask feature map, can specifically be used for:

[0218] The second convolutional sub-network is invoked to perform convolution processing on the pseudo-mask image to soften the pixel values ​​of each pixel in the pseudo-mask image, thereby obtaining a pseudo-mask feature map.

[0219] In another implementation, the mask branch network includes a second convolutional sub-network and a first channel adjustment sub-network; wherein the second convolutional sub-network has a different network structure from the first convolutional sub-network, and the number of convolutional kernels used by the first channel adjustment sub-network is equal to the number of channels of the feature map output by the first convolutional network; the extraction unit 803, when used to perform feature extraction processing on the pseudo-mask image to obtain a pseudo-mask feature map, can specifically be used for:

[0220] The second convolutional sub-network is invoked to perform convolution processing on the pseudo-mask image to soften the pixel values ​​of each pixel in the pseudo-mask image, thereby obtaining a second feature map;

[0221] The first channel adjustment subnetwork uses each convolution kernel to perform a convolution operation on the second feature map to obtain a pseudo-mask feature map.

[0222] In another implementation, the mask branch network includes: a forward convolutional subnetwork, a deconvolutional subnetwork, and a second channel adjustment subnetwork; wherein the number of convolutional kernels used by the second channel adjustment subnetwork is equal to the number of channels of the target object image; the extraction unit 803, when performing feature extraction processing on the pseudo-mask image to obtain a pseudo-mask feature map, can specifically be used for:

[0223] The positive convolutional sub-network is invoked to perform a positive convolution operation on the pseudo-mask image to soften the pixel values ​​of each pixel in the pseudo-mask image and obtain a positive convolutional feature map.

[0224] The deconvolutional subnetwork is invoked to perform a reverse convolution operation on the forward convolutional feature map to obtain a deconvolutional feature map, the image size of which is equal to the image size of the target object image;

[0225] The convolution kernels in the second channel adjustment sub-network are invoked to perform convolution processing on the deconvolution feature map to obtain the pseudo-mask feature map.

[0226] In another implementation, the image branch network in the object recognition model is used to output the target image feature vector of the target object image, and the number of channels of the target image feature vector is the same as the number of channels of the target fused feature vector; the recognition unit 804, when used to perform category recognition on the object to be recognized based on the target fused feature vector to obtain the category recognition result, can be specifically used for:

[0227] Based on the feature vector of the target image, the object to be identified is classified to obtain the first category score of the object to be identified in each of the multiple preset categories;

[0228] Based on the target fusion feature vector, the object to be identified is subjected to category identification processing to obtain the second category score of the object to be identified under each preset category;

[0229] The first category score and the corresponding second category score of the object to be identified under each preset category are fused to obtain the target category score of the object to be identified under each preset category.

[0230] Based on the target category score of the object to be identified under each preset category, calculate the target probability of the object to be identified belonging to each preset category; and add the preset category corresponding to the highest target probability to the category identification result of the object to be identified.

[0231] In another implementation, when the image branch network includes the first convolutional sub-network, the target image feature vector is obtained by performing feature enhancement processing on the first feature map of the target object image;

[0232] When the image branch network does not include the first convolutional sub-network, the target image feature vector is obtained by performing feature enhancement processing on the target object image.

[0233] In another implementation, the image branch network in the object recognition model is used to output the target image feature vector of the target object image, and the number of channels of the target image feature vector is the same as the number of channels of the target fused feature vector; the recognition unit 804, when used to perform category recognition on the object to be recognized based on the target fused feature vector to obtain the category recognition result, can be specifically used for:

[0234] Based on the target image features, the object to be identified is classified to obtain the first category score of the object to be identified in each of the multiple preset categories;

[0235] Based on the target fusion feature vector, the object to be identified is subjected to category identification processing to obtain the second category score of the object to be identified under each preset category;

[0236] The category recognition result of the object to be identified is constructed by using the first category score of the object to be identified under each preset category and the second category score of the object to be identified under each preset category.

[0237] In another implementation, the identification unit 804 may further be used for:

[0238] Obtain the label category of the object to be identified;

[0239] Based on the label category and the category recognition result, calculate the model loss value of the object recognition model;

[0240] The model parameters of the object recognition model are optimized in the direction of reducing the model loss value.

[0241] In another implementation, the recognition unit 804, when calculating the model loss value of the object recognition model based on the label category and the category recognition result, may specifically be used for:

[0242] Based on the label category and the category recognition result, the first feature loss value of the object recognition model is calculated;

[0243] A cross-correlation matrix is ​​constructed using the target image feature vector and the target fusion feature vector; the cross-correlation matrix is ​​used to indicate the similarity between the target image feature vector and the target fusion feature vector.

[0244] Based on the cross-correlation matrix and the optimization objective of the cross-correlation matrix, the second feature loss value of the object recognition model is calculated;

[0245] The model loss value of the object recognition model is obtained by summing the first feature loss value and the second feature loss value.

[0246] According to one embodiment of this application, Figure 2 or Figure 6c or Figure 7b Each step involved in the method shown can be performed by... Figure 8 This is performed by each unit in the object recognition device shown. For example, Figure 2 The step S201 shown is by Figure 8 The acquisition unit 801 shown is used to execute step S202. Figure 8 The mask unit 802 shown is used to perform step S203. Figure 8 The extraction unit 803 shown is used to perform the steps S204 and S205. Figure 8 The identification unit 804 shown is used to perform this function. For example, Figure 6c The step S601 shown is by Figure 8 The acquisition unit 801 shown is used to execute step S602. Figure 8 The mask unit 802 shown is used to perform step S603. Figure 8 The extraction unit 803 shown is used to perform the steps S604 to S608. Figure 8 The identification unit 804 shown is used to perform this function. For example, Figure 7b The step S701 shown is by Figure 8 The acquisition unit 801 shown is used to perform the acquisition, and step S702 is performed by... Figure 8 The mask unit 802 shown is used to perform the steps S703 to S705. Figure 8 The extraction unit 803 shown is used to perform the steps S706 to S710. Figure 8 The identification unit 804 shown is used to perform this function.

[0247] According to another embodiment of this application, Figure 8The various units in the object recognition device shown can be individually or entirely merged into one or more other units, or some of the units can be further divided into multiple functionally smaller units. This achieves the same operation without affecting the technical effects of the embodiments of this application. The above-mentioned units are divided based on logical functions. In practical applications, the function of one unit can also be implemented by multiple units, or the function of multiple units can be implemented by one unit. In other embodiments of this application, the object recognition device may also include other units. In practical applications, these functions can also be implemented with the assistance of other units, and can be implemented collaboratively by multiple units.

[0248] According to another embodiment of this application, processing elements and storage elements, such as a central processing unit (CPU), random access storage medium (RAM), and read-only storage medium (ROM), can be used. For example, a general-purpose computing device such as a computer can run on a device capable of performing tasks such as... Figure 2 or Figure 6c or Figure 7b The computer program (including program code) for each step involved in the corresponding method shown, to construct such... Figure 8 The object recognition device shown, and the object recognition method for implementing the embodiments of this application, are described. The computer program may be recorded on, for example, a computer-readable recording medium, loaded onto the aforementioned computer device via the computer-readable recording medium, and run therein.

[0249] In this embodiment, a computer device can acquire a target object image containing the object to be identified, and perform masking processing on the target object image using the object detection bounding box of the object to be identified to obtain a pseudo-mask image of the target object image. Then, feature extraction processing can be performed on the pseudo-mask image to obtain a pseudo-mask feature map. Further, based on the pseudo-mask feature map and the target object image, a target fusion feature vector is generated, and the object to be identified is classified according to the target fusion feature vector to obtain the classification result. By implementing the above method, a corresponding pseudo-mask image can be generated using the object detection bounding box of the object in the object image. By combining the object image and the pseudo-mask image of the object image, the region of interest (i.e., the area where the object is located) in the object image can be focused on, while suppressing the interference of background information in the object image, thereby effectively improving the accuracy of object classification.

[0250] Based on the description of the above object recognition method embodiments, this application also discloses a computer device; please refer to [link to relevant documentation]. Figure 9The computer device may include at least a processor 901, an input device 902, an output device 903, and a computer storage medium 904. The processor 901, input device 902, output device 903, and computer storage medium 904 within the computer device may be connected via a bus or other means.

[0251] The computer storage medium 904 is a memory device in a computer device used to store programs and data. It is understood that the computer storage medium 904 can include the computer device's built-in storage medium, or it can include extended storage media supported by the computer device. The computer storage medium 904 provides storage space that stores the computer device's operating system. Furthermore, this storage space also stores one or more instructions suitable for loading and execution by the processor 901. These instructions can be one or more computer programs (including program code). It should be noted that the computer storage medium can be a high-speed RAM memory; optionally, it can also be at least one computer storage medium located remotely from the aforementioned processor. The processor can be called a Central Processing Unit (CPU), which is the core and control center of the computer device, suitable for implementing one or more instructions, specifically loading and executing one or more instructions to achieve corresponding method flows or functions.

[0252] In one feasible embodiment, the processor 901 can load and execute one or more first instructions stored in the computer storage medium to implement the corresponding steps of the method in the above-described embodiment of the object recognition method; specifically, the processor 901 loads and executes one or more first instructions in the computer storage medium as follows:

[0253] Obtain an image of the target object containing the object to be identified;

[0254] The target object image is masked using the object detection bounding box of the object to be identified to obtain a pseudo mask image of the target object image;

[0255] The pseudo-mask image is subjected to feature extraction processing to obtain a pseudo-mask feature map; the feature value of any pixel in the pseudo-mask feature map is used to indicate the importance of any pixel in the target object image;

[0256] Based on the pseudo-mask feature map and the target object image, a target fusion feature vector is generated; and the object to be identified is classified according to the target fusion feature vector to obtain the classification result.

[0257] In another implementation, when determining a target detection box containing the object to be identified in the original image based on the target image shape adapted by the object recognition model and the object detection box, the one or more instructions are loaded and executed by the processor 901:

[0258] According to the target image shape adapted by the object recognition model, the object detection box is transformed in the original image to obtain an initial detection box containing the object to be recognized.

[0259] The initial detection box is expanded to obtain a target detection box containing the image to be identified.

[0260] In another implementation, the object detection result further includes the original position coordinates of the object detection box in the original image; when performing masking processing on the target object image using the object detection box of the object to be identified to obtain a pseudo-mask image of the target object image, the one or more instructions are loaded and specifically executed by the processor 901:

[0261] The original position coordinates of the object detection box in the original image are remapped to the target object image to obtain the mapped position coordinates of the object detection box;

[0262] Based on the mapped position coordinates of the object detection box and the position coordinates of each pixel in the target object image, the positional relationship between the object detection box and each pixel is determined; the positional relationship is used to indicate whether a pixel is located within the object detection box.

[0263] Based on the determined positional relationship, the pixel values ​​of the pixels in the target object image that are located in the object detection box are updated to valid values, and the pixel values ​​of the pixels in the target object image that are not located in the object detection box are updated to invalid values.

[0264] The target object image after pixel value updates is used as the pseudo-mask image of the target object image.

[0265] In another implementation, when generating a target fusion feature vector based on the pseudo-mask feature map and the target object image, the one or more instructions are loaded and executed by the processor 901:

[0266] In the mask branch network of the object recognition model, feature fusion processing is performed on the pseudo-mask feature map and the target object image to obtain an intermediate fused feature map;

[0267] In the mask branch network, the intermediate fused feature map is subjected to feature enhancement processing to obtain the target fused feature vector.

[0268] In another implementation, the object recognition model further includes an image branch network, which includes a first convolutional sub-network; when performing feature fusion processing on the pseudo-mask feature map and the target object image in the mask branch network of the object recognition model to obtain an intermediate fused feature map, the one or more instructions are loaded and executed by the processor 901:

[0269] The first convolutional sub-network is invoked to perform convolution processing on the target object image to obtain the first feature map of the target object image;

[0270] The first feature map is passed from the image branch network to the mask branch network, and the first feature map and the pseudo mask feature map are fused in the mask branch network to obtain an intermediate fused feature map.

[0271] In another implementation, the mask branch network includes a second convolutional sub-network, and the second convolutional sub-network has the same network structure as the first convolutional network; when performing feature extraction processing on the pseudo-mask image to obtain a pseudo-mask feature map, the one or more instructions are loaded and specifically executed by the processor 901:

[0272] The second convolutional sub-network is invoked to perform convolution processing on the pseudo-mask image to soften the pixel values ​​of each pixel in the pseudo-mask image, thereby obtaining a pseudo-mask feature map.

[0273] In another implementation, the mask branch network includes a second convolutional sub-network and a first channel adjustment sub-network; wherein the second convolutional sub-network has a different network structure from the first convolutional sub-network, and the number of convolutional kernels used by the first channel adjustment sub-network is equal to the number of channels of the feature map output by the first convolutional sub-network; when performing feature extraction processing on the pseudo-mask image to obtain a pseudo-mask feature map, the one or more instructions are loaded and specifically executed by the processor 901:

[0274] The second convolutional sub-network is invoked to perform convolution processing on the pseudo-mask image to soften the pixel values ​​of each pixel in the pseudo-mask image, thereby obtaining a second feature map;

[0275] The first channel adjustment subnetwork uses each convolution kernel to perform a convolution operation on the second feature map to obtain a pseudo-mask feature map.

[0276] In another implementation, the mask branch network includes: a forward convolutional subnetwork, a deconvolutional subnetwork, and a second channel adjustment subnetwork; wherein the number of convolutional kernels used by the second channel adjustment subnetwork is equal to the number of channels of the target object image; when performing feature extraction processing on the pseudo-mask image to obtain a pseudo-mask feature map, one or more instructions are loaded and specifically executed by the processor 901:

[0277] The positive convolutional sub-network is invoked to perform a positive convolution operation on the pseudo-mask image to soften the pixel values ​​of each pixel in the pseudo-mask image and obtain a positive convolutional feature map.

[0278] The deconvolutional subnetwork is invoked to perform a reverse convolution operation on the forward convolutional feature map to obtain a deconvolutional feature map, the image size of which is equal to the image size of the target object image;

[0279] The convolution kernels in the second channel adjustment sub-network are invoked to perform convolution processing on the deconvolution feature map to obtain the pseudo-mask feature map.

[0280] In another implementation, the image branch network in the object recognition model is used to output the target image feature vector of the target object image, and the number of channels of the target image feature vector is the same as the number of channels of the target fused feature vector; when performing category recognition on the object to be recognized based on the target fused feature vector to obtain the category recognition result, the one or more instructions are loaded and specifically executed by the processor 901:

[0281] Based on the feature vector of the target image, the object to be identified is classified to obtain the first category score of the object to be identified in each of the multiple preset categories;

[0282] Based on the target fusion feature vector, the object to be identified is subjected to category identification processing to obtain the second category score of the object to be identified under each preset category;

[0283] The first category score and the corresponding second category score of the object to be identified under each preset category are fused to obtain the target category score of the object to be identified under each preset category.

[0284] Based on the target category score of the object to be identified under each preset category, calculate the target probability of the object to be identified belonging to each preset category; and add the preset category corresponding to the highest target probability to the category identification result of the object to be identified.

[0285] In another implementation, when the image branch network includes the first convolutional sub-network, the target image feature vector is obtained by performing feature enhancement processing on the first feature map of the target object image;

[0286] When the image branch network does not include the first convolutional sub-network, the target image feature vector is obtained by performing feature enhancement processing on the target object image.

[0287] In another implementation, the image branch network in the object recognition model is used to output the target image feature vector of the target object image, and the number of channels of the target image feature vector is the same as the number of channels of the target fused feature vector; when performing category recognition on the object to be recognized based on the target fused feature vector to obtain the category recognition result, the one or more instructions are loaded and specifically executed by the processor 901:

[0288] Based on the target image features, the object to be identified is classified to obtain the first category score of the object to be identified in each of the multiple preset categories;

[0289] Based on the target fusion feature vector, the object to be identified is subjected to category identification processing to obtain the second category score of the object to be identified under each preset category;

[0290] The category recognition result of the object to be identified is constructed by using the first category score of the object to be identified under each preset category and the second category score of the object to be identified under each preset category.

[0291] In another implementation, the one or more instructions are loaded and executed by processor 901:

[0292] Obtain the label category of the object to be identified;

[0293] Based on the label category and the category recognition result, calculate the model loss value of the object recognition model;

[0294] The model parameters of the object recognition model are optimized in the direction of reducing the model loss value.

[0295] In another implementation, when calculating the model loss value of the object recognition model based on the label category and the category recognition result, the one or more instructions are loaded and executed by the processor 901:

[0296] Based on the label category and the category recognition result, the first feature loss value of the object recognition model is calculated;

[0297] A cross-correlation matrix is ​​constructed using the target image feature vector and the target fusion feature vector; the cross-correlation matrix is ​​used to indicate the similarity between the target image feature vector and the target fusion feature vector.

[0298] Based on the cross-correlation matrix and the optimization objective of the cross-correlation matrix, the second feature loss value of the object recognition model is calculated;

[0299] The model loss value of the object recognition model is obtained by summing the first feature loss value and the second feature loss value.

[0300] In this embodiment, a computer device can acquire a target object image containing the object to be identified, and perform masking processing on the target object image using the object detection bounding box of the object to be identified to obtain a pseudo-mask image of the target object image. Then, feature extraction processing can be performed on the pseudo-mask image to obtain a pseudo-mask feature map. Further, based on the pseudo-mask feature map and the target object image, a target fusion feature vector is generated, and the object to be identified is classified according to the target fusion feature vector to obtain the classification result. By implementing the above method, a corresponding pseudo-mask image can be generated using the object detection bounding box of the object in the object image. By combining the object image and the pseudo-mask image of the object image, the region of interest (i.e., the area where the object is located) in the object image can be focused on, while suppressing the interference of background information in the object image, thereby effectively improving the accuracy of object classification.

[0301] It should be noted that this application also provides a computer program product or computer program, which includes computer instructions stored in a computer-readable storage medium. The processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the object recognition method embodiment described above. Figure 2 or Figure 6c or Figure 7b The steps performed in the process.

[0302] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The program can be stored in a computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. The storage medium can be a magnetic disk, optical disk, read-only memory (ROM), or random access memory (RAM), etc.

[0303] The above-disclosed embodiments are merely preferred embodiments of this application and should not be construed as limiting the scope of this application. Those skilled in the art will understand that implementing all or part of the processes of the above embodiments and making equivalent changes in accordance with the claims of this application are still within the scope of the invention.

Claims

1. An object recognition method, characterized in that, include: Obtain an image of the target object containing the object to be identified; The method of masking the target object image using the object detection bounding box of the object to be identified to obtain a pseudo mask image of the target object image includes: generating a pseudo mask image corresponding to the size of the target object image based on the position of the object detection bounding box of the object to be identified in the target object image; wherein, the pixel value of the pixel in the pseudo mask image is used to indicate whether the pixel is located within the object detection bounding box; The pseudo-mask image is subjected to feature extraction processing to obtain a pseudo-mask feature map; the feature value of any pixel in the pseudo-mask feature map is used to indicate the importance of any pixel in the target object image; Based on the pseudo-mask feature map and the target object image, a target fusion feature vector is generated; and the object to be identified is classified according to the target fusion feature vector to obtain the classification result.

2. The method as described in claim 1, characterized in that, The pseudo-mask feature map, the target fusion feature vector, and the category recognition result are all obtained by calling an object recognition model; obtaining the target object image containing the object to be recognized includes: Obtain the original image containing the object to be identified; Object detection is performed on the original image to obtain the object detection result of the original image, and the object detection result includes the object detection box of the object to be identified; Based on the target image shape adapted by the object recognition model and the object detection box, a target detection box containing the object to be recognized is determined in the original image; the shape of the target detection box is the target image shape, and the size of the target detection box is larger than the size of the object detection box. The region image contained in the target detection box is cropped from the original image, and the cropped region image is used as the image to be identified.

3. The method as described in claim 2, characterized in that, The step of determining a target detection box containing the object to be identified in the original image based on the target image shape adapted by the object recognition model and the object detection box includes: According to the target image shape adapted by the object recognition model, the object detection box is transformed in the original image to obtain an initial detection box containing the object to be recognized. The initial detection box is expanded to obtain a target detection box containing the image to be identified.

4. The method as described in claim 2, characterized in that, The object detection result also includes the original position coordinates of the object detection box in the original image; the step of using the object detection box of the object to be identified to perform masking processing on the target object image to obtain a pseudo mask image of the target object image includes: The original position coordinates of the object detection box in the original image are remapped to the target object image to obtain the mapped position coordinates of the object detection box; Based on the mapped position coordinates of the object detection box and the position coordinates of each pixel in the target object image, the positional relationship between the object detection box and each pixel is determined; the positional relationship is used to indicate whether a pixel is located within the object detection box. Based on the determined positional relationship, the pixel values ​​of the pixels in the target object image that are located in the object detection box are updated to valid values, and the pixel values ​​of the pixels in the target object image that are not located in the object detection box are updated to invalid values. The target object image after pixel value updates is used as the pseudo-mask image of the target object image.

5. The method as described in claim 1, characterized in that, The step of generating a target fusion feature vector based on the pseudo-mask feature map and the target object image includes: In the mask branch network of the object recognition model, feature fusion processing is performed on the pseudo-mask feature map and the target object image to obtain an intermediate fused feature map; In the mask branch network, the intermediate fused feature map is subjected to feature enhancement processing to obtain the target fused feature vector.

6. The method as described in claim 5, characterized in that, The object recognition model further includes an image branching network, which includes a first convolutional subnetwork. In the mask branch network of the object recognition model, feature fusion processing is performed on the pseudo-mask feature map and the target object image to obtain an intermediate fused feature map, including: The first convolutional sub-network is invoked to perform convolution processing on the target object image to obtain the first feature map of the target object image; The first feature map is passed from the image branch network to the mask branch network, and the first feature map and the pseudo mask feature map are fused in the mask branch network to obtain an intermediate fused feature map.

7. The method as described in claim 6, characterized in that, The mask branch network includes a second convolutional subnetwork, and the second convolutional subnetwork has the same network structure as the first convolutional network; The step of performing feature extraction processing on the pseudo-mask image to obtain a pseudo-mask feature map includes: The second convolutional sub-network is invoked to perform convolution processing on the pseudo-mask image to soften the pixel values ​​of each pixel in the pseudo-mask image, thereby obtaining a pseudo-mask feature map.

8. The method as described in claim 6, characterized in that, The mask branch network includes a second convolutional subnetwork and a first channel adjustment subnetwork; wherein the second convolutional subnetwork has a different network structure from the first convolutional subnetwork, and the number of convolutional kernels used by the first channel adjustment subnetwork is equal to the number of channels of the feature map output by the first convolutional subnetwork. The step of performing feature extraction processing on the pseudo-mask image to obtain a pseudo-mask feature map includes: The second convolutional sub-network is invoked to perform convolution processing on the pseudo-mask image to soften the pixel values ​​of each pixel in the pseudo-mask image, thereby obtaining a second feature map; The first channel adjustment subnetwork uses each convolution kernel to perform a convolution operation on the second feature map to obtain a pseudo-mask feature map.

9. The method as described in claim 5, characterized in that, The mask branch network includes: a forward convolution subnetwork, a deconvolution subnetwork, and a second channel adjustment subnetwork; wherein, the number of convolution kernels used by the second channel adjustment subnetwork is equal to the number of channels of the target object image; The step of performing feature extraction processing on the pseudo-mask image to obtain a pseudo-mask feature map includes: The positive convolutional sub-network is invoked to perform a positive convolution operation on the pseudo-mask image to soften the pixel values ​​of each pixel in the pseudo-mask image and obtain a positive convolutional feature map. The deconvolutional subnetwork is invoked to perform a reverse convolution operation on the forward convolutional feature map to obtain a deconvolutional feature map, the image size of which is equal to the image size of the target object image; The convolution kernels in the second channel adjustment sub-network are invoked to perform convolution processing on the deconvolution feature map to obtain the pseudo-mask feature map.

10. The method according to any one of claims 5-9, characterized in that, The image branch network in the object recognition model is used to output the target image feature vector of the target object image, and the number of channels of the target image feature vector is the same as the number of channels of the target fusion feature vector; The step of classifying the object to be identified based on the target fusion feature vector to obtain the class identification result includes: Based on the feature vector of the target image, the object to be identified is classified to obtain the first category score of the object to be identified in each of the multiple preset categories; Based on the target fusion feature vector, the object to be identified is subjected to category identification processing to obtain the second category score of the object to be identified under each preset category; The first category score and the corresponding second category score of the object to be identified under each preset category are fused to obtain the target category score of the object to be identified under each preset category. Based on the target category score of the object to be identified under each preset category, calculate the target probability of the object to be identified belonging to each preset category; and add the preset category corresponding to the highest target probability to the category identification result of the object to be identified.

11. The method according to any one of claims 6-8, characterized in that, When the image branch network includes the first convolutional sub-network, the target image feature vector is obtained by performing feature enhancement processing on the first feature map of the target object image; When the image branch network does not include the first convolutional sub-network, the target image feature vector is obtained by performing feature enhancement processing on the target object image.

12. The method according to any one of claims 5-9, characterized in that, The image branch network in the object recognition model is used to output the target image feature vector of the target object image, and the number of channels of the target image feature vector is the same as the number of channels of the target fusion feature vector; The step of classifying the object to be identified based on the target fusion feature vector to obtain the class identification result includes: Based on the target image features, the object to be identified is classified to obtain the first category score of the object to be identified in each of the multiple preset categories; Based on the target fusion feature vector, the object to be identified is subjected to category identification processing to obtain the second category score of the object to be identified under each preset category; The category recognition result of the object to be identified is constructed by using the first category score of the object to be identified under each preset category and the second category score of the object to be identified under each preset category.

13. The method as described in claim 12, characterized in that, The method further includes: Obtain the label category of the object to be identified; Based on the label category and the category recognition result, calculate the model loss value of the object recognition model; The model parameters of the object recognition model are optimized in the direction of reducing the model loss value.

14. The method as described in claim 13, characterized in that, The step of calculating the model loss value of the object recognition model based on the label category and the category recognition result includes: Based on the label category and the category recognition result, the first feature loss value of the object recognition model is calculated; A cross-correlation matrix is ​​constructed using the target image feature vector and the target fusion feature vector; the cross-correlation matrix is ​​used to indicate the similarity between the target image feature vector and the target fusion feature vector. Based on the cross-correlation matrix and the optimization objective of the cross-correlation matrix, the second feature loss value of the object recognition model is calculated; The model loss value of the object recognition model is obtained by summing the first feature loss value and the second feature loss value.

15. A computer storage medium storing one or more instructions, which, when executed, perform the method as described in any one of claims 1-14.

Images