Image recognition method, device, equipment, storage medium and computer program product

Abstract

The application discloses an image recognition method and device, equipment, storage medium and computer program product, relates to the computer technical field, and can be applied to the traffic field. The method comprises the following steps: acquiring a target image, wherein the target image comprises an image element to be recognized; performing element recognition on the target image to obtain an image region corresponding to the image element to be recognized in the target image, wherein the image element to be recognized comprises at least one image sub-element; performing component recognition on the image element to be recognized to obtain a splitting result of the at least one image sub-element in the image element to be recognized, wherein the splitting result is used for indicating the position and category of the at least one image sub-element in the image element; and performing classification recognition on the image element to be recognized based on the splitting result of the at least one image sub-element to obtain a classification result corresponding to the image element to be recognized, so that the accuracy of image element type recognition is improved.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to an image recognition method, apparatus, device, storage medium, and computer program product. Background Technology

[0002] Traffic signs are facilities that use graphic symbols and text to convey specific information, manage traffic, indicate driving directions, and ensure smooth traffic flow and driving safety. Identifying sign categories is of great significance for applications such as intelligent transportation, assisted driving, and intelligent vehicle-road cooperation.

[0003] In related technologies, convolutional neural networks such as ResNet (Residual Network) are trained directly using a large amount of labeled data to extract and classify high-level semantic features of the signage as a whole, thus obtaining the final image recognition result.

[0004] However, due to the complex shape and diverse content of rectangular signs, and the significant interference between different categories of rectangular signs, the above method, which simply uses a deep learning classification network to classify signs based on their overall features, has low accuracy in image element recognition and low accuracy in sign type recognition. Summary of the Invention

[0005] This application provides an image recognition method, apparatus, device, storage medium, and computer program product capable of recognizing image types. The technical solution is as follows.

[0006] On the one hand, an image recognition method is provided, the method comprising:

[0007] Acquire a target image, wherein the target image includes image elements to be identified;

[0008] Element recognition is performed on the target image to obtain the image region in the target image that corresponds to the image element to be recognized, wherein the image element to be recognized includes at least one image sub-element;

[0009] Component recognition is performed on the image element to be identified to obtain the segmentation result of at least one image sub-element in the image element to be identified. The segmentation result is used to indicate the position and category of the at least one image sub-element in the image element.

[0010] Based on the splitting results of at least one image sub-element, the image element to be identified is classified and identified to obtain the classification result corresponding to the image element to be identified.

[0011] On the other hand, an image recognition device is provided, the device comprising:

[0012] A target image acquisition module is used to acquire a target image, wherein the target image includes image elements to be identified;

[0013] An image element recognition module is used to perform element recognition on the target image to obtain an image region in the target image that corresponds to the image element to be recognized, wherein the image element to be recognized includes at least one image sub-element.

[0014] The element component recognition module is used to perform component recognition on the image element to be recognized, and obtain the splitting result of at least one image sub-element in the image element to be recognized. The splitting result is used to indicate the position and category of the at least one image sub-element in the image element.

[0015] The image element classification module is used to classify and identify the image element to be identified based on the splitting result of the at least one image sub-element, and obtain the classification result corresponding to the image element to be identified.

[0016] On the other hand, a computer device is provided, the computer device including a processor and a memory, the memory storing at least one instruction, at least one program, code set or instruction set, the at least one instruction, the at least one program, the code set or instruction set being loaded and executed by the processor to implement the image recognition method as described in any of the embodiments of this application above.

[0017] On the other hand, a computer-readable storage medium is provided, wherein at least one instruction, at least one program, code set, or instruction set is stored therein, wherein the at least one instruction, the at least one program, the code set, or the instruction set is loaded and executed by a processor to implement the image recognition method as described in any of the embodiments of this application above.

[0018] On the other hand, a computer program product or computer program is provided, which includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform any of the image recognition methods described in the above embodiments.

[0019] The beneficial effects of the technical solutions provided in this application include at least the following:

[0020] By performing element recognition on image elements in a target image and component recognition on image sub-elements within those image elements, the feature of the image sub-elements is combined with the feature of the image elements—that is, the component features of the image elements are combined with the overall features of the image elements—to identify the category of the image elements, thereby improving the accuracy of image element type recognition.

[0021] In the field of transportation, vehicle-road cooperative applications involve recognizing traffic signs in road images captured by vehicle-mounted equipment or satellite navigation equipment, identifying the elements and components of the traffic signs, and combining the features of the elements and components with the overall features of the traffic signs to classify and identify them. This is used to assist in road navigation and improves the accuracy of traffic sign type recognition, thereby improving the efficiency of vehicle-road cooperation. Attached Figure Description

[0022] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0023] Figure 1 This is a schematic diagram of an implementation environment provided by an exemplary embodiment of this application;

[0024] Figure 2 This is a flowchart of an image recognition method provided in an exemplary embodiment of this application;

[0025] Figure 3 This is a schematic diagram of a target image provided in an exemplary embodiment of this application;

[0026] Figure 4 This is a schematic diagram of image region recognition provided in an exemplary embodiment of this application;

[0027] Figure 5 This is a schematic diagram of an image sub-element provided in an exemplary embodiment of this application;

[0028] Figure 6 This is a schematic diagram of the types of components provided in an exemplary embodiment of this application;

[0029] Figure 7 This is a schematic diagram of component detection provided in an exemplary embodiment of this application;

[0030] Figure 8 This is a schematic diagram of image element feature extraction provided in an exemplary embodiment of this application;

[0031] Figure 9 This is a schematic diagram of image element categories provided in an exemplary embodiment of this application;

[0032] Figure 10 This is a flowchart of an image element recognition process provided in an exemplary embodiment of this application;

[0033] Figure 11This is a schematic diagram of image element recognition provided in an exemplary embodiment of this application;

[0034] Figure 12 This is a schematic diagram of a candidate box provided in an exemplary embodiment of this application;

[0035] Figure 13 This is a flowchart illustrating the component identification process provided in an exemplary embodiment of this application;

[0036] Figure 14 This is a schematic diagram of image region feature extraction provided in an exemplary embodiment of this application;

[0037] Figure 15 This is a flowchart of a classification model training process provided in an exemplary embodiment of this application;

[0038] Figure 16 This is a schematic diagram of a sample image provided in an exemplary embodiment of this application;

[0039] Figure 17 This is a schematic diagram of image element classification provided in an exemplary embodiment of this application;

[0040] Figure 18 This is a structural block diagram of an image recognition device provided in an exemplary embodiment of this application;

[0041] Figure 19 This is a structural block diagram of an image recognition device module provided in an exemplary embodiment of this application;

[0042] Figure 20 This is a structural block diagram of a terminal provided in an exemplary embodiment of this application. Detailed Implementation

[0043] To make the objectives, technical solutions, and advantages of this application clearer, the embodiments of this application will be described in further detail below with reference to the accompanying drawings.

[0044] It should be understood that although the terms first, second, etc., may be used in this disclosure to describe various information, such information should not be limited to these terms. These terms are only used to distinguish information of the same type from one another. For example, a first parameter may also be referred to as a second parameter without departing from the scope of this disclosure, and similarly, a second parameter may also be referred to as a first parameter. Depending on the context, the word "if" as used herein may be interpreted as "when," "when," or "in response to determination."

[0045] Intelligent Vehicle Infrastructure Cooperative Systems (IVICS) are a development direction of Intelligent Transportation Systems (ITS). IVICS utilizes advanced wireless communication and next-generation Internet technologies to implement comprehensive, real-time dynamic information interaction between vehicles and infrastructure. Based on the collection and fusion of dynamic traffic information across all times and spaces, it conducts active vehicle safety control and cooperative road management, fully realizing effective collaboration between people, vehicles, and roads to ensure traffic safety, improve traffic efficiency, and thus form a safe, efficient, and environmentally friendly road traffic system. In the transportation field, traffic signs use graphic symbols and text to convey specific information, managing traffic, indicating driving directions, and ensuring smooth traffic flow and driving safety. Identifying sign categories is of great significance for applications such as intelligent transportation, assisted driving, and intelligent vehicle-infrastructure cooperation. In related technologies, by directly training convolutional neural networks such as ResNet (Residual Network) using a large amount of labeled data, high-level semantic features are extracted and classified from the overall signage to obtain the final image recognition result. However, due to the complex shape and diverse content of rectangular signs, and the significant interference between different categories of rectangular signs, the above method, which simply uses a deep learning classification network to classify signs based on their overall features, has low accuracy in image element recognition and low accuracy in sign type recognition.

[0046] This application provides an image recognition method that, while recognizing image elements in a target image, also identifies component elements within those image elements. By combining the features of the sub-elements with the features of the overall image elements—that is, combining the component features with the overall features—the method identifies the category of the image elements, thus improving the accuracy of image element type recognition. In the vehicle-road cooperative application scenario in the transportation field, this method recognizes traffic signs in road images captured by vehicle-mounted equipment or satellite navigation devices, and identifies the elements and components of the traffic signs. By combining the component features with the overall features of the traffic signs, the method classifies and identifies the traffic signs to assist in road navigation, thereby improving the accuracy of traffic sign type recognition and ultimately enhancing vehicle-road cooperative efficiency.

[0047] First, the implementation environment of this application will be introduced. Please refer to... Figure 1 It illustrates a schematic diagram of an implementation environment provided by an exemplary embodiment of this application, such as... Figure 1 As shown, the implementation environment includes: terminal 110, server 120 and communication network 130.

[0048] In some embodiments, terminal 110 is used to acquire target images. For example, in the fields of transportation, mapping, or autonomous driving, terminal 110 can be implemented as an in-vehicle central control device connected to a camera to take pictures of the road ahead of the vehicle and acquire the road image as the target image. Alternatively, in intelligent image recognition application scenarios, terminal 110 can be implemented as a mobile phone equipped with intelligent image recognition software, taking pictures of the plant to be identified through the phone's camera and acquiring the plant photo as the target image, or reading locally stored images containing the plant to be identified as the target image. Server 120 is used to perform image recognition on the target images acquired by terminal 110. For example, in the fields of transportation, mapping, and autonomous driving, server 120 is used to classify and recognize traffic signs in the acquired road images. Alternatively, in intelligent image recognition application scenarios, server 120 is used to classify and recognize plants in the acquired scene images.

[0049] In some embodiments, terminal 110 and server 120 are connected through communication network 130. Terminal 110 transmits the acquired target image to server 120 through communication network 130. After performing image recognition on the received target image, server 120 transmits the recognition result of the target image back to terminal 110 through communication network 130.

[0050] The aforementioned terminal is optional. The terminal can be a desktop computer, laptop computer, mobile phone, tablet computer, e-book reader, MP3 (Moving Picture Experts Group Audio Layer III) player, MP4 (Moving Picture Experts Group Audio Layer IV) player, smart TV, smart vehicle, smart voice interaction device, smart home appliance, aircraft and other terminal devices. This application embodiment does not limit the specific terminal device in this regard.

[0051] It is worth noting that the aforementioned servers can be independent physical servers, server clusters or distributed systems composed of multiple physical servers, or cloud servers that provide basic cloud computing services such as cloud services, cloud security, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDN), and big data and artificial intelligence platforms.

[0052] Cloud technology refers to a hosting technology that unifies a series of resources such as hardware, software, and networks within a wide area network or local area network to achieve data computing, storage, processing, and sharing.

[0053] In some embodiments, the server described above can also be implemented as a node in a blockchain system.

[0054] It should be noted that the information (including but not limited to user device information, user personal information, etc.), data (including but not limited to data used for analysis, data stored, data displayed, etc.) and signals involved in this application are all authorized by the user or fully authorized by all parties, and the collection, use and processing of related data must comply with the relevant laws, regulations and standards of the relevant countries and regions.

[0055] Optionally, embodiments of the present invention can be applied to various scenarios, including but not limited to cloud technology, artificial intelligence, smart transportation, and assisted driving.

[0056] This is illustrative; please refer to it. Figure 2 , Figure 2 This is a flowchart of an exemplary embodiment of an image recognition method provided in this application. This method can be applied to a terminal, a server, or both. This embodiment uses the application of this method to a terminal as an example for illustration. Figure 2 As shown, the method includes the following steps:

[0057] Step 210: Obtain the target image.

[0058] The target image includes image elements to be identified.

[0059] In the traffic field, as an illustration, vehicle-mounted cameras can photograph the road ahead of a vehicle to obtain road target images, including traffic signs. Please refer to [the example]. Figure 3 , Figure 3 This is a schematic diagram of a target image provided in an exemplary embodiment of this application, such as... Figure 3 As shown, target image 300 is a photograph of the road in front of the vehicle taken by the vehicle-mounted camera, which includes three image elements to be identified: traffic sign 301, traffic sign 302, and traffic sign 303.

[0060] Step 220: Perform element recognition on the target image to obtain the image region in the target image that corresponds to the image element to be recognized.

[0061] The image element to be identified includes at least one image sub-element. The image sub-element is a local image of the image element to be identified, serving as an independent component of the image element to be identified. Examples include arrows, text labels, vehicle logos, etc. in traffic signs, or leaves, petals, flower centers, roots, etc. of plants, or store trademarks, directional arrows, facility icons, etc. in shopping mall signs. These sub-elements are used to identify local features or local information of the image element to be identified.

[0062] This is illustrative; please refer to it. Figure 4 , Figure 4 This is a schematic diagram of image region recognition provided in an exemplary embodiment of this application, such as... Figure 4 As shown, element recognition is performed on the target image 400 to obtain the image region 410 in the target image 400 corresponding to the image element 401 to be recognized.

[0063] This is illustrative; please refer to it. Figure 5 , Figure 5 This is a schematic diagram of an image sub-element provided in an exemplary embodiment of this application, such as... Figure 5 As shown, the image element to be identified 500 includes 8 image sub-elements. The image element to be identified 500 serves as a traffic speed limit sign, including speed limit vehicle type signs 501 to 503, maximum speed limit signs 504 to 506 corresponding to the vehicle type, minimum speed limit text 507, and minimum speed limit sign 508.

[0064] In some optional embodiments, a convolutional neural network can be used to identify elements in the target image to obtain the image region in the target image that corresponds to the image element to be identified.

[0065] Step 230: Perform component recognition on the image element to be recognized to obtain the splitting result of at least one image sub-element in the image element to be recognized.

[0066] The splitting result is used to indicate the position and category of at least one image sub-element in the image element. The position information in the splitting result is a sub-region obtained by performing a secondary region division on the image region where the image element is located. It can be implemented as a region centered on the image sub-element and including the image sub-element and tangent to it, or it can be implemented as a region selected by a specified size and shape with the image sub-element as the region center.

[0067] This is illustrative; please refer to it. Figure 6 , Figure 6 This is a schematic diagram of the types of components provided in an exemplary embodiment of this application, such as... Figure 6 As shown, in the field of transportation, traffic signs can include various types of components such as directional arrows, vehicle types, and shape types, for example... Figure 6In Part Type Table 600, the image corresponding to item 1 belongs to the straight arrow part, and the image corresponding to item 36 belongs to the section speed measurement part. It is worth noting that... Figure 6 The 36 component categories listed are merely illustrative examples; component categories in traffic signs include, but are not limited to, those listed below. Figure 6 The categories listed herein are not limited in this application.

[0068] This is illustrative; please refer to it. Figure 7 , Figure 7 This is a schematic diagram of component detection provided in an exemplary embodiment of this application, such as... Figure 7 As shown, the component recognition module 720 performs component recognition on the image element 711 and the image element 712 to be recognized, and obtains the splitting result 731 of 9 image sub-elements in the image element 711 to be recognized, including 9 image sub-element regions marked by rectangular boxes, and obtains the splitting result 732 of 4 image sub-elements in the image element 712 to be recognized, including 4 image sub-elements marked by rectangular boxes.

[0069] Step 240: Based on the splitting result of at least one image sub-element, classify and identify the image element to be identified to obtain the classification result corresponding to the image element to be identified.

[0070] The classification result can be achieved in at least one of the following ways: image elements with category labels, the name of the image element category, and the number of image elements belonging to each category.

[0071] In some alternative embodiments, step 240 can be implemented as the following two steps:

[0072] first step Feature extraction is performed on the image elements to be identified to obtain the image feature representations corresponding to the image elements to be identified.

[0073] In some optional embodiments, a repeatedly weighted bidirectional feature pyramid network is used to extract features from the target image, obtaining the image feature representation corresponding to the target image. For illustrative examples, please refer to [reference needed]. Figure 8 , Figure 8 This is a schematic diagram of image element feature extraction provided in an exemplary embodiment of this application, such as... Figure 8 As shown, the image element 810 to be identified is input into a convolutional neural network for feature extraction. The image features are repeatedly weighted by a repetitive weighted bidirectional feature pyramid network 820, and finally the image feature representation 830 corresponding to the image element 810 to be identified is obtained.

[0074] Step 2 The image feature representation and the splitting result of at least one image sub-element are input into the classification prediction model, and the output is the classification result corresponding to the image element to be identified.

[0075] In some optional embodiments, the splitting result of at least one image sub-element can be used to adjust the weight parameters of the classification prediction model through an attention mechanism. The image feature representation is input into the classification prediction model, and the classification result corresponding to the image element to be identified is output.

[0076] This is illustrative; please refer to it. Figure 9 , Figure 9 This is a schematic diagram of image element categories provided in an exemplary embodiment of this application, such as... Figure 9 As shown, in the transportation field, traffic signs can include categories such as road directional signs and turn-in indicator signs, such as... Figure 9 The sign types listed in Table 900 include: a) signs of road name signs, h) signs of service area signs, and j) signs of yielding to pedestrians signs. It is important to note that the 13 sign types in Table 900 are merely illustrative examples; the types of traffic signs include, but are not limited to, those listed in Table 900, and this application does not impose any restrictions on them.

[0077] In summary, the method provided in this application improves the accuracy of image element type recognition by simultaneously identifying image elements in a target image and identifying sub-elements within those image elements. It combines sub-element features with overall image element features—that is, combining component features with overall image element features—to identify the category of image elements. In the context of vehicle-road cooperative applications in the transportation field, this method identifies traffic signs in road images captured by vehicle-mounted devices or satellite navigation equipment, and identifies the elements and components of the traffic signs. By combining the component features with the overall features of the traffic signs, it classifies and identifies the traffic signs to assist in road navigation, thereby improving the accuracy of traffic sign type identification and ultimately enhancing vehicle-road cooperative efficiency.

[0078] The method provided in this application clarifies the steps of the classification prediction method, combines image sub-element features with image element features to identify the category of image elements, and improves the accuracy of image element type identification.

[0079] Figure 10 This is a flowchart of an image element recognition process provided in an exemplary embodiment of this application, such as... Figure 10 As shown, in some optional embodiments, step 220 above can be implemented as steps 221 and 222.

[0080] Step 221: Extract features from the target image to obtain the image feature representation corresponding to the target image.

[0081] Image features are used to indicate the overall features of the target image.

[0082] In some optional embodiments, feature extraction of the target image to obtain the image feature representation corresponding to the target image can be achieved through the feature extraction module in a convolutional neural network structure. For illustrative examples, please refer to [reference needed]. Figure 11 , Figure 11 This is a schematic diagram of image element recognition provided in an exemplary embodiment of this application, such as... Figure 11 As shown, the feature extraction module 1100 in the convolutional neural network structure extracts features from the target image 1110 to obtain the image feature representation 1120 corresponding to the target image 1110.

[0083] Step 222: In the image feature representation, element detection is performed on at least one feature point using at least one preset candidate element box to obtain the region selected by the target element box corresponding to the image element to be identified in the target image as the image region.

[0084] The candidate element box is used to identify the region location of the image element to be identified in the target image. In some optional embodiments, the candidate element box is implemented as a rectangle with a specified aspect ratio and scale, with a specified position as the center of the rectangle, such as the center of the image element to be identified, and the size of the rectangle is the size of the image region to select the region where the image element to be identified is located.

[0085] This is illustrative; please refer to it. Figure 11 On the image feature representation 1120, element detection is performed on at least one feature point using at least one preset candidate element box 1130, and the region selected by the target element box 1131 corresponding to the image element 1121 to be identified in the target image 1140 is taken as the image region 1141.

[0086] In some alternative embodiments, step 222 can be implemented as the following two steps:

[0087] first step Obtain at least one candidate element box by combining at least one aspect ratio and at least one scale.

[0088] This is illustrative; please refer to it. Figure 12 , Figure 12 This is a schematic diagram of a candidate box provided in an exemplary embodiment of this application, such as... Figure 12 As shown, nine rectangular boxes of size 1200 with feature points as the center point, aspect ratios of {1:1, 2:1, 1:2}, and scales of 1, 2, and 3 feature points can be selected as candidate element boxes.

[0089] Step 2The feature points in the image feature representation are used as the center points of the candidate element boxes. Element detection is performed through at least one candidate element box to obtain the region selected by the target element box corresponding to the image element to be identified in the target image as the image region.

[0090] In some optional embodiments, the second step described above is implemented by using a rectangle with a specified aspect ratio and scale, taking the feature points in the image feature representation as the center point of the rectangle, and taking the size of the rectangle as the size of the image region, to select the region where the image sub-elements represented by the feature points are located.

[0091] Indicative, Figure 4 The feature points in the image feature representation obtained by feature extraction of the target image 400 are used as the center points of the candidate element box. Element detection is performed through a candidate element box with an aspect ratio of 1:2 and a scale of 2 feature points. The region 410 selected by the target box corresponding to the image element 401 to be identified in the target image 400 is used as the image region.

[0092] In summary, the method provided in this application clarifies the process of image element recognition, performs element detection on image elements by using candidate element boxes with various aspect ratios and scales, determines image regions, adapts to rectangular signs of various sizes, and improves the accuracy of element detection.

[0093] The method provided in this application provides a variety of candidate element box selection schemes with different aspect ratios and scales, which can adapt to rectangular signs of various sizes and improve the accuracy of element detection.

[0094] Figure 13 This is a flowchart illustrating the component identification process provided in an exemplary embodiment of this application, such as... Figure 13 As shown, in some optional embodiments, step 230 above can be implemented as steps 231 and 232.

[0095] Step 231: Extract features from the image regions corresponding to the image elements to be identified to obtain the region feature representations corresponding to the image regions.

[0096] In some optional embodiments, a repeatedly weighted bidirectional feature pyramid network can be used to extract features from the image regions corresponding to the image elements to be identified, thereby obtaining the region feature representations corresponding to the image regions. For illustration, please refer to [reference needed]. Figure 14 , Figure 14 This is a schematic diagram of image region feature extraction provided in an exemplary embodiment of this application, such as... Figure 14As shown, the image region 1410 corresponding to the image element to be identified is input into the convolutional neural network for feature extraction. The image features are repeatedly weighted by the repeated weighted bidirectional feature pyramid network 1420, and finally the region feature representation 1430 corresponding to the image region 1410 of the image element to be identified is obtained.

[0097] Step 232: For at least one feature point in the region feature representation, perform component detection using at least one preset candidate component box to obtain the splitting result of at least one image sub-element.

[0098] In this context, the candidate component bounding box is used to indicate the regional location of an image sub-element within an image element. In some optional embodiments, the candidate component bounding box can be implemented as a rectangle with a specified aspect ratio and scale. The center of the rectangle is the center of the feature point, and the size of the rectangle is the size of the component region. The region where the image sub-element corresponding to the feature point is located is selected. The segmentation result is the image element to be identified that is marked with the candidate component bounding box. The segmentation result includes the image sub-element marked by the candidate component bounding box with a component category label.

[0099] In some alternative embodiments, step 232 can be implemented as the following two steps:

[0100] first step Obtain at least one candidate component frame by combining at least one aspect ratio and at least one scale.

[0101] Indicative, such as Figure 12 As shown, nine rectangular boxes (1200) with feature points as the center point, aspect ratios of {1:1, 2:1, 1:2}, and scales of 1, 2, and 3 feature points can be selected as candidate component boxes.

[0102] Step 2 The feature points in the region feature representation are used as the center points of the candidate component boxes. Component segmentation prediction is performed through at least one candidate component box to obtain the segmentation result of at least one image sub-element in the image region.

[0103] In some optional embodiments, the second step described above is implemented by using a rectangle with a specified aspect ratio and scale, taking the feature points in the region feature representation as the center point of the rectangle, and taking the size of the rectangle as the size of the image region, to select the region where the image sub-elements represented by the feature points are located.

[0104] Indicative, Figure 14 The feature points in the middle region (1430) are used as the center points of the candidate component boxes. Component segmentation prediction is performed using candidate component boxes with an aspect ratio of 2:1 and a scale of 2 feature points. Figure 7 Image element 712 to be identified Figure 14The splitting result 732 of the four image sub-elements in the image region 1410 includes four image sub-elements marked by rectangular boxes.

[0105] In summary, the method provided in this application clarifies the component detection process, performs component detection on image sub-elements using candidate component boxes with various aspect ratios and scales, determines the component segmentation results, adapts to sign components of various sizes, and improves the accuracy of component detection.

[0106] The method provided in this application provides a variety of candidate component frame selection schemes with different aspect ratios and scales, which can adapt to sign components of various sizes and improve the accuracy of component detection.

[0107] In some optional embodiments, the element recognition of the target image in step 220 can be implemented by performing element recognition of the target image through a classification model, and the classification of the image elements to be recognized based on the splitting result of at least one image sub-element in step 240 can be implemented by performing classification of the image elements to be recognized based on the splitting result of at least one image sub-element through a classification model.

[0108] This is illustrative; please refer to it. Figure 15 , Figure 15 This is a flowchart of a classification model training process provided in an exemplary embodiment of this application, such as... Figure 15 As shown, the training process of the above classification model includes the following steps:

[0109] Step 1510: Obtain the sample image.

[0110] The sample image includes image elements labeled with classification labels and component region labels. The classification labels are used to indicate the classification result of the image element, and the component region labels are used to indicate the segmentation result of the image sub-elements in the image element.

[0111] This is illustrative; please refer to it. Figure 16 , Figure 16 This is a schematic diagram of a sample image provided in an exemplary embodiment of this application, such as... Figure 16 As shown, sample image 1600 is acquired through an in-vehicle camera. Sample image 1600 includes image element 1610 labeled with classification tags and component area tags. The classification tags of image element 1610 are a schematic diagram of image element categories. Figure 9 The category name of the label c, the component region label of image element 1610, indicates the segmentation result of image sub-elements 1601 to 1604 in image element 1610, including the position and component category of the image sub-elements, such as the component category of image sub-element 1604. Figure 6 The category corresponding to image number 14 is diagonally upwards.

[0112] Step 1520: Element recognition is performed on the sample image using a candidate classification model to obtain the predicted image region corresponding to the image element in the sample image.

[0113] Each image element includes at least one image sub-element.

[0114] In some optional embodiments, the candidate classification model includes an element recognition module. This module identifies elements in the sample image to obtain predicted image regions corresponding to the image elements. The element recognition module can be implemented as a convolutional neural network (CNN). This CNN structure includes convolutional layers, normalization layers, and activation layers. The convolutional layers extract basic features such as edge textures from the sample image. The normalization layers filter out noise features from the basic features extracted by the convolutional layers, such as normalizing the basic features extracted by the convolutional layers according to a normal distribution. The activation layers perform nonlinear mapping on the basic features extracted by the convolutional layers to enhance the generalization ability of the element recognition module in the candidate classification model.

[0115] Step 1530: Perform component recognition on the image elements to obtain the splitting result of at least one image sub-element in the image elements.

[0116] In some optional embodiments, the candidate classification model includes a component identification module, illustratively, such as... Figure 7 As shown, the component recognition module 720 performs component recognition on the image element 711 and the image element 712 to be recognized, and obtains the splitting result 731 of the 9 image sub-elements in the image element 711 to be recognized and the splitting result 732 of the 4 image sub-elements in the image element 712 to be recognized.

[0117] Step 1540: Classify and identify image elements based on the splitting results of at least one image sub-element to obtain the predicted classification results corresponding to the image elements.

[0118] In some optional embodiments, the candidate classification model includes an element classification module; for illustrative purposes, please refer to [reference needed]. Figure 17 , Figure 17 This is a schematic diagram of image element classification provided in an exemplary embodiment of this application, such as... Figure 17 As shown, based on the splitting results 1711 of the nine image sub-elements obtained by the component recognition module 1710 in the candidate classification model, the image elements are classified and identified by the candidate model element classification module 1720, and the predicted classification results 1721 corresponding to the image elements are obtained. This is a schematic diagram of the image element categories. Figure 9 The category to which sign b belongs is vehicle signage.

[0119] In some optional embodiments, the candidate classification model uses an attention mechanism to apply the splitting results of at least one image sub-element to classify and identify image elements. The attention mechanism can be implemented as a dot product model, which can be implemented as follows:

[0120]

[0121] Where i is used to identify the sample, x is the image feature of the element classification module, that is, the image feature representation of the target image obtained by feature extraction of the target image, and q is the feature value of the corresponding part recognition module, that is, the region feature representation of the image region obtained by feature extraction of the image region corresponding to the image element to be recognized in the target image. The part features of the part recognition module are passed to the element classification module in the form of a product, thereby guiding the learning of the element classification module and improving the accuracy of recognizing the category of image elements.

[0122] Step 1550: Based on the first difference between the splitting result and the component region label, and the second difference between the predicted classification result and the classification label, train the candidate classification model to obtain the classification model.

[0123] In some optional embodiments, the candidate classification model is trained by fusing the first and second differences, that is, by fusing regression loss and classification loss. The loss function of this method is as follows:

[0124]

[0125]

[0126] Among them, smoothL1(t i -t′ i ) represents the regression loss of the candidate classification model, used to accurately locate the position of the bounding boxes inside the rectangular sign, i.e., the position of the image sub-elements within the image elements, where c represents the class and t represents the category. i Let t' represent the true coordinates of sample i. i This represents the predicted coordinates of sample i. Let M be the classification loss of the candidate classification model, where M represents the number of rectangular sign categories, and y is the classification loss. ic The indicator variable takes a value of 0 or 1; it is 1 if the category is the same as that of sample i, and 0 otherwise. ic This represents the probability that sample i belongs to category c.

[0127] In summary, the method provided in this application clarifies how to classify and identify image elements in a target image using a classification model, thereby improving the accuracy of classification and identification.

[0128] The method provided in this application embodiment clarifies the training method of the classification model, trains the classification model based on the first difference and the second difference, enhances the generalization ability of the classification model, and improves the recognition accuracy of the classification model.

[0129] Figure 18 This is a structural block diagram of an image recognition device provided in an exemplary embodiment of this application, such as... Figure 18 As shown, the device includes the following parts:

[0130] The target image acquisition module 1810 is used to acquire a target image, which includes image elements to be identified.

[0131] The image element recognition module 1820 is used to perform element recognition on the target image to obtain the image region in the target image corresponding to the image element to be recognized, wherein the image element to be recognized includes at least one image sub-element.

[0132] The element component recognition module 1830 is used to perform component recognition on the image element to be recognized, and obtain the splitting result of at least one image sub-element in the image element to be recognized. The splitting result is used to indicate the position and category of the at least one image sub-element in the image element.

[0133] The image element classification module 1840 is used to classify and identify the image element to be identified based on the splitting result of the at least one image sub-element, and obtain the classification result corresponding to the image element to be identified.

[0134] In some alternative embodiments, please refer to Figure 19 , Figure 19 This is a structural block diagram of an image recognition device provided in an exemplary embodiment of this application, such as... Figure 19 As shown, the image element recognition module 1820 includes:

[0135] The image feature extraction unit 1821 is used to extract features from the target image to obtain the image feature representation corresponding to the target image;

[0136] The image region acquisition unit 1822 is used to perform element detection on the image feature representation for at least one feature point using at least one preset candidate element box, and obtain the region selected by the target element box corresponding to the image element to be identified in the target image as the image region.

[0137] In some optional embodiments, the image region acquisition unit 1822 is used to acquire at least one candidate element box obtained by combining at least one aspect ratio and at least one scale; use the feature points in the image feature representation as the center points of the candidate element boxes, perform element detection through the at least one candidate element box, and obtain the region selected by the target element box corresponding to the image element to be identified in the target image as the image region.

[0138] In some optional embodiments, the element component identification module 1830 includes:

[0139] The region feature extraction unit 1831 is used to extract features from the image region corresponding to the image element to be identified, and obtain the region feature representation corresponding to the image region.

[0140] The splitting result acquisition unit 1832 is used to perform component detection on at least one feature point in the region feature representation using at least one preset candidate component box to obtain the splitting result of the at least one image sub-element.

[0141] In some optional embodiments, the splitting result acquisition unit 1832 is used to acquire at least one candidate component box obtained by combining at least one aspect ratio and at least one scale; using the feature points in the region feature representation as the center points of the candidate component boxes, and performing component splitting prediction through the at least one candidate component box to obtain the splitting result of the at least one image sub-element in the image region.

[0142] In some optional embodiments, the image element classification module 1840 is used to extract features from the image element to be identified to obtain an image feature representation corresponding to the image element to be identified; input the image feature representation and the splitting result of the at least one image sub-element into the classification prediction model, and output the classification result corresponding to the image element to be identified.

[0143] In some optional embodiments, the image element recognition module 1820 is used to perform element recognition on the target image through a classification model to obtain the image region in the target image corresponding to the image element to be recognized, and the image element classification module 1840 is used to classify and recognize the image element to be recognized based on the splitting result of the at least one image sub-element through the classification model to obtain the classification result corresponding to the image element to be recognized.

[0144] In some optional embodiments, the apparatus further includes a classification model training module 1850, configured to acquire sample images, the sample images including image elements labeled with classification labels and component region labels, the classification labels indicating the classification result of the image elements, and the component region labels indicating the segmentation result of image sub-elements within the image elements; perform element recognition on the sample images using a candidate classification model to obtain predicted image regions in the sample images corresponding to the image elements, the image elements including at least one image sub-element; perform component recognition on the image elements to obtain the segmentation result of the at least one image sub-element within the image elements; classify and recognize the image elements based on the segmentation result of the at least one image sub-element to obtain a predicted classification result corresponding to the image elements; and train the candidate classification model based on a first difference between the segmentation result and the component region label, and a second difference between the predicted classification result and the classification label, to obtain the classification model.

[0145] In summary, the image recognition device provided in this application obtains image element recognition results and element component recognition results through image element recognition and element component recognition, respectively, and uses the element component recognition results for image element recognition. That is, it combines the overall features of the image element with the local features of the image element corresponding to the element component to achieve image element classification and recognition, thereby improving the accuracy of image element classification and recognition.

[0146] It should be noted that the image recognition device provided in the above embodiments is only an example of the division of the above functional modules. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above.

[0147] Figure 20 A structural block diagram of a terminal 2000 provided in an exemplary embodiment of this application is shown. The terminal 2000 may be a smartphone, tablet computer, MP3 player (Moving Picture Experts Group Audio Layer III), MP4 player (Moving Picture Experts Group Audio Layer IV), laptop computer, or desktop computer. The terminal 2000 may also be referred to as a user device, portable terminal, laptop terminal, desktop terminal, or other names.

[0148] Typically, terminal 2000 includes a processor 2001 and a memory 2002.

[0149] Processor 2001 may include one or more processing cores, such as a quad-core processor, an octa-core processor, etc. Processor 2001 may be implemented using at least one hardware form selected from DSP (Digital Signal Processing), FPGA (Field-Programmable Gate Array), and PLA (Programmable Logic Array). Processor 2001 may also include a main processor and a coprocessor. The main processor, also known as a CPU (Central Processing Unit), is used to process data in the wake-up state; the coprocessor is a low-power processor used to process data in the standby state. In some embodiments, processor 2001 may integrate a GPU (Graphics Processing Unit), which is responsible for rendering and drawing the content required to be displayed on the screen. In some embodiments, processor 2001 may also include an AI (Artificial Intelligence) processor, which is used to handle computational operations related to machine learning.

[0150] The memory 2002 may include one or more computer-readable storage media, which may be non-transitory. The memory 2002 may also include high-speed random access memory and non-volatile memory, such as one or more disk storage devices or flash memory devices. In some embodiments, the non-transitory computer-readable storage media in the memory 2002 are used to store at least one instruction, which is executed by the processor 2001 to implement the image recognition method provided in the method embodiments of this application.

[0151] In some embodiments, the terminal 2000 also includes other components, as those skilled in the art will understand. Figure 20 The structure shown does not constitute a limitation on the terminal 2000, and may include more or fewer components than shown, or combine certain components, or use different component arrangements.

[0152] Embodiments of this application also provide a computer device that can be implemented as follows: Figure 1 The terminal or server shown. The computer device includes a processor and a memory, the memory storing at least one instruction, at least one program, code set, or instruction set, the at least one instruction, at least one program, code set, or instruction set being loaded and executed by the processor to implement the image recognition method provided in the above-described method embodiments.

[0153] Embodiments of this application also provide a computer-readable storage medium storing at least one instruction, at least one program, code set, or instruction set, wherein the at least one instruction, at least one program, code set, or instruction set is loaded and executed by a processor to implement the image recognition method provided in the above-described method embodiments.

[0154] Embodiments of this application also provide a computer program product or computer program, which includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform any of the image recognition methods described in the above embodiments.

[0155] Optionally, the computer-readable storage medium may include: read-only memory (ROM), random access memory (RAM), solid-state drives (SSDs), or optical discs, etc. The random access memory may include resistive random access memory (ReRAM) and dynamic random access memory (DRAM). The sequence numbers of the embodiments in this application are merely descriptive and do not represent the superiority or inferiority of the embodiments.

[0156] Those skilled in the art will understand that all or part of the steps of the above embodiments can be implemented by hardware or by a program instructing related hardware. The program can be stored in a computer-readable storage medium, such as a read-only memory, a disk, or an optical disk.

[0157] The above description is merely an optional embodiment of this application and is not intended to limit this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the protection scope of this application.

Claims

1. An image recognition method, characterized in that, The method includes: Acquire a target image, wherein the target image includes image elements to be identified; Element recognition is performed on the target image to obtain the image region in the target image that corresponds to the image element to be recognized, wherein the image element to be recognized includes at least one image sub-element; Feature extraction is performed on the image region corresponding to the image element to be identified to obtain the region feature representation corresponding to the image region; In the region feature representation, for at least one feature point, component detection is performed using at least one preset candidate component box to obtain the splitting result of the at least one image sub-element. The splitting result is used to indicate the position and category of the at least one image sub-element in the image element. The candidate component box is used to select the region where the image sub-element corresponding to the feature point is located, with the feature point as the center. Based on the splitting results of at least one image sub-element, the image element to be identified is classified and identified to obtain the classification result corresponding to the image element to be identified.

2. The method according to claim 1, characterized in that, The step of performing element recognition on the target image to obtain the image region in the target image corresponding to the image element to be recognized includes: Feature extraction is performed on the target image to obtain the image feature representation corresponding to the target image; In the image feature representation, element detection is performed on at least one feature point using at least one preset candidate element box, and the region selected by the target element box corresponding to the image element to be identified in the target image is taken as the image region.

3. The method according to claim 2, characterized in that, The step of performing element detection on at least one feature point using at least one preset candidate bounding box on the image feature representation, and obtaining the region selected by the target bounding box corresponding to the image element to be identified in the target image as the image region, includes: Obtain at least one candidate element box by combining at least one aspect ratio and at least one scale. The feature points in the image feature representation are used as the center points of the candidate element boxes. Element detection is performed through the at least one candidate element box to obtain the region selected by the target element box corresponding to the image element to be identified in the target image as the image region.

4. The method according to any one of claims 1 to 3, characterized in that, The step of performing component detection on at least one feature point using at least one preset candidate component box on the region feature representation to obtain the segmentation result of the at least one image sub-element includes: Obtain at least one candidate component box by combining at least one aspect ratio and at least one scale. The feature points in the region feature representation are used as the center points of the candidate component boxes. Component segmentation prediction is performed using the at least one candidate component box to obtain the segmentation result of at least one image sub-element in the image region.

5. The method according to any one of claims 1 to 3, characterized in that, The process of classifying and identifying the image element to be identified based on the segmentation result of the at least one image sub-element to obtain the classification result corresponding to the image element to be identified includes: Feature extraction is performed on the image elements to be identified to obtain the image feature representations corresponding to the image elements to be identified; The image feature representation and the splitting result of at least one image sub-element are input into the classification prediction model, and the classification result corresponding to the image element to be identified is output.

6. The method according to any one of claims 1 to 3, characterized in that, The element recognition of the target image includes: The target image is identified using a classification model; The classification and identification of the image elements to be identified based on the segmentation results of the at least one image sub-element includes: The classification model classifies and identifies the image elements to be identified based on the splitting results of at least one image sub-element.

7. The method according to claim 6, characterized in that, The training process of the classification model includes: Obtain a sample image, which includes image elements labeled with classification labels and component region labels. The classification labels are used to indicate the classification result of the image elements, and the component region labels are used to indicate the segmentation result of the image sub-elements in the image elements. The sample image is identified by a candidate classification model to obtain a predicted image region in the sample image corresponding to the image element, wherein the image element includes at least one image sub-element. Component recognition is performed on the image element to obtain the splitting result of at least one image sub-element in the image element; Based on the splitting result of the at least one image sub-element, the image element is classified and identified to obtain the predicted classification result corresponding to the image element; Based on the first difference between the splitting result and the component region label, and the second difference between the predicted classification result and the classification label, the candidate classification model is trained to obtain the classification model.

8. An image recognition device, characterized in that, The device includes: A target image acquisition module is used to acquire a target image, wherein the target image includes image elements to be identified; An image element recognition module is used to perform element recognition on the target image to obtain an image region in the target image that corresponds to the image element to be recognized, wherein the image element to be recognized includes at least one image sub-element. The region feature extraction unit is used to extract features from the image region corresponding to the image element to be identified, and obtain the region feature representation corresponding to the image region. The splitting result acquisition unit is used to perform component detection on the region feature representation for at least one feature point using at least one preset candidate component box to obtain the splitting result of the at least one image sub-element. The splitting result is used to indicate the position and category of the at least one image sub-element in the image element. The candidate component box is used to select the region where the image sub-element corresponding to the feature point is located with the feature point as the center. The image element classification module is used to classify and identify the image element to be identified based on the splitting result of the at least one image sub-element, and obtain the classification result corresponding to the image element to be identified.

9. A computer device, characterized in that, The computer device includes a processor and a memory, the memory storing at least one program, which is loaded and executed by the processor to implement the image recognition method as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The storage medium stores at least one program segment, which is loaded and executed by a processor to implement the image recognition method as described in any one of claims 1 to 7.

11. A computer program product, characterized in that, It includes a computer program that, when executed by a processor, implements the image recognition method as described in any one of claims 1 to 7.

Images