Image recognition method and device, electronic device, and storage medium

Abstract

The disclosed embodiments relate to an image recognition method and apparatus, electronic device, and storage medium, and relate to the field of computer technology. The image recognition method includes: obtaining multiple semantic tags from a reference object and obtaining visual representations corresponding to each semantic tag; establishing a labeling system corresponding to an application system based on the semantic tags and visual representations; and annotating and identifying a pending image based on the labeling system to determine the location and category of objects in the pending image. The disclosed technical solution can improve the precision of the labeling system and the accuracy of image recognition.

Description

Technical Field

[0001] The present disclosure relates to the field of computer technology, and in particular to an image recognition method, an image recognition device, an electronic device, and a computer-readable storage medium. Background Art

[0002] Object detection is a fundamental recognition task in computer vision, used to identify objects and their locations within an image. When using network models for object detection, the categories and definitions of the objects labeled in the data directly impact the accuracy of object classification.

[0003] Related technologies primarily use semantic tags to determine object categories, or label a small number of objects. Because some objects are categorized based on real-life experience, using semantic tags to determine object categories can result in large intra-class differences and small inter-class differences, leading to limitations and poor accuracy.

[0004] It should be noted that the information disclosed in the above background technology section is only used to enhance the understanding of the background of the present disclosure, and therefore may include information that does not constitute prior art known to ordinary technicians in the field. Summary of the Invention

[0005] The purpose of the present disclosure is to provide an image recognition method and device, an electronic device, and a storage medium, thereby overcoming, at least to a certain extent, the problem of low classification accuracy caused by the limitations and defects of related technologies.

[0006] Other features and advantages of the present disclosure will become apparent from the following detailed description, or may be learned in part by practice of the present disclosure.

[0007] According to one aspect of the present disclosure, an image recognition method is provided, comprising: obtaining a plurality of semantic tags through a reference object, and obtaining a visual representation corresponding to each of the semantic tags; establishing a label system corresponding to an application system based on the semantic tags and the visual representation; and annotating and identifying an image to be processed based on the label system to determine an object position and an object category in the image to be processed.

[0008] According to one aspect of the present disclosure, an image recognition device is provided, comprising: a visual representation acquisition module for acquiring multiple semantic labels through a reference object, and acquiring a visual representation corresponding to each of the semantic labels; a label system construction module for establishing a label system corresponding to an application system based on the semantic labels and the visual representation; and a label recognition module for performing label recognition on an image to be processed based on the label system, and determining the position and category of objects in the image to be processed.

[0009] According to one aspect of the present disclosure, an electronic device is provided, comprising: a processor; and a memory for storing executable instructions of the processor; wherein the processor is configured to perform any one of the above-described image recognition methods by executing the executable instructions.

[0010] According to one aspect of the present disclosure, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the image recognition method described above is implemented.

[0011] In the image recognition method, image recognition device, electronic device, and computer-readable storage medium provided in the embodiments of the present disclosure, on the one hand, by constructing a label system through semantic labels and visual representations, the label system can be constructed from multiple different dimensions, avoiding the limitations caused by only being able to construct a label system from semantic labels, and improving the accuracy and comprehensiveness of the label system. On the other hand, by annotating and identifying the processed images based on the label system established based on semantic labels and visual representations, the situation where there are large differences within the class and small differences between classes is avoided, and because the label system can be constructed through a large number of reference objects for labeling, the accuracy and comprehensiveness are improved, and accurate object detection can be performed.

[0012] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] The accompanying drawings are incorporated into and constitute a part of the specification, illustrate embodiments consistent with the present disclosure, and together with the specification, are used to explain the principles of the present disclosure. Obviously, the drawings described below are only some embodiments of the present disclosure, and those skilled in the art can derive other drawings based on these drawings without inventive effort.

[0014] Figure 1 A schematic diagram showing a system architecture to which the image recognition method according to an embodiment of the present disclosure can be applied.

[0015] Figure 2 A schematic diagram schematically illustrates an image recognition method in an embodiment of the present disclosure.

[0016] Figure 3 The following schematically illustrates a method for determining a visual representation in an embodiment of the present disclosure.

[0017] Figure 4 The following schematically illustrates a flow chart of determining visual tags in an embodiment of the present disclosure.

[0018] Figure 5A schematic diagram schematically illustrates the construction of a labeling system corresponding to the annotation system in an embodiment of the present disclosure.

[0019] Figure 6 A schematic diagram schematically illustrates the construction of a labeling system corresponding to a training system in an embodiment of the present disclosure.

[0020] Figure 7 The overall process flow diagram of constructing a label in an embodiment of the present disclosure is schematically shown.

[0021] Figure 8 The following is a schematic diagram of the process of performing object detection in an embodiment of the present disclosure.

[0022] Figure 9 The following schematically shows a block diagram of an image recognition device in an embodiment of the present disclosure.

[0023] Figure 10 A block diagram schematically illustrates an electronic device in an embodiment of the present disclosure. DETAILED DESCRIPTION

[0024] Example embodiments will now be described more fully with reference to the accompanying drawings. However, example embodiments can be implemented in a variety of forms and should not be construed as limited to the examples set forth herein; rather, these embodiments are provided so that the present disclosure will be more comprehensive and complete and will fully convey the concepts of the example embodiments to those skilled in the art. The described features, structures, or characteristics may be combined in any suitable manner in one or more embodiments. In the following description, many specific details are provided to provide a full understanding of the embodiments of the present disclosure. However, those skilled in the art will appreciate that the technical solutions of the present disclosure may be practiced while omitting one or more of the specific details, or that other methods, components, devices, steps, etc. may be employed. In other cases, well-known technical solutions are not shown or described in detail to avoid obscuring various aspects of the present disclosure.

[0025] In addition, the accompanying drawings are merely schematic illustrations of the present disclosure and are not necessarily drawn to scale. Identical reference numerals in the figures denote identical or similar parts, and thus repetitive descriptions thereof will be omitted. Some of the block diagrams shown in the accompanying drawings are functional entities that do not necessarily correspond to physically or logically separate entities. These functional entities may be implemented in software, in one or more hardware modules or integrated circuits, or in different networks and / or processor devices and / or microcontroller devices.

[0026] In order to solve the technical problems in the related art, an image recognition method is provided in an embodiment of the present disclosure, which can be applied to various types of application scenarios such as object detection.

[0027] Figure 1A schematic diagram showing a system architecture to which the image recognition method and apparatus according to the embodiments of the present disclosure can be applied.

[0028] like Figure 1 As shown, the system architecture 100 may include a client 101, a network 102, and a server 103. The client may be a terminal such as a smart phone, a computer, a tablet computer, a smart speaker, etc. The network 102 is used to provide a medium for a communication link between the client 101 and the server 103. The network 102 may include various connection types, such as a wired communication link, a wireless communication link, etc. In the embodiment of the present disclosure, the network 102 between the client 101 and the server 103 may be a wired communication link, such as a communication link provided by a serial port connection line, or a wireless communication link, such as a communication link provided by a wireless network. The server 103 may be a server or a client with computing functions, such as a portable computer, a desktop computer, a smart phone, and other terminal devices with computing functions, which are used to process images or models sent by the client.

[0029] This image recognition method can be applied to the application scenario of detecting objects in the image to be processed obtained from the client, and identifying the location and category of objects contained therein. Figure 1 As shown in , the client 101 can send the image to be processed to the server 103 through the network 102. The server 103 calls the trained target detection model to perform object annotation and recognition on the image to be processed to obtain the recognition result corresponding to the image to be processed, which may specifically include the object position and object category contained in the image to be processed. The recognition result can also be sent to the client 101 for display and other processing operations. The server can be the same as the client, that is, the client and the server are both smart devices, such as smart phones. Based on this, it is possible to use the client to obtain the image to be processed and perform object detection on the image to be processed. In addition, the server can also be different from the client, which is not particularly limited here.

[0030] It should be noted that the image recognition method provided in the embodiments of the present disclosure can be completely executed by a server or a client. Accordingly, the image recognition device can be set in the server or in the client.

[0031] Next, refer to Figure 2 The image recognition method in the embodiment of the present disclosure is described in detail.

[0032] In step S210 , a plurality of semantic tags are obtained by referring to the object, and a visual representation corresponding to each of the semantic tags is obtained.

[0033] In the embodiment of the present disclosure, the reference object can be determined based on the classification list and ordinary objects (objects in life that are of the same category as the classification list), and the reference object is determined based on the usage scenario. The usage scenario can be object recognition, document recognition, or code recognition, etc. When the usage scenario is object recognition, the classification list can be a classification list for a shopping scenario, and the reference object can be a commodity in the classification list and an ordinary commodity. When the usage scenario is document recognition, the reference object can be an ordinary document. When the usage scenario is code recognition, the reference object can be an ordinary code (such as a QR code, barcode, etc.). The usage scenario of object recognition is used as an example for explanation. The type of an object can be determined based on the type of scene it is in. In the embodiment of the present disclosure, object recognition in a shopping scenario is used as an example for explanation. In a shopping scenario, the object can be a commodity. Based on this, multiple semantic tags can be determined based on the objects in the classification list of the shopping scenario and ordinary objects. The semantic tag can be a tag used to represent the type, function, purpose, and other attributes of each object. For example, the semantic tag can be the type of object, such as beverage in a shopping scenario. By including reference objects in the classification list and common objects, the amount of data can be increased, avoiding the limitations caused by a small amount of data, thereby improving the accuracy and comprehensiveness of semantic labels.

[0034] After obtaining multiple semantic tags, the visual representation corresponding to each semantic tag can be obtained. The visual representation is used to describe the form, shape, size and other external features of an object, such as bottled, bagged, canned, etc. When obtaining the visual representation, the image associated with each semantic tag can be obtained according to the semantic tag, and the image associated with each semantic tag can be analyzed to obtain the corresponding visual representation. Here, the image associated with each semantic tag refers to all images that match each semantic tag. For example, if the semantic tag is beverage, the images that match the semantic tag are images of all types of beverages, for example, image A, image B, image C, etc., and the type and content presented by each image can be the same or different.

[0035] Furthermore, the images matching the semantic labels can be analyzed to obtain the visual representations corresponding to the semantic labels. Specifically, the features of each image can be extracted through a neural network model to obtain the visual representation of each image, and the visual representations of all images belonging to the same semantic label can be combined to obtain the visual representation corresponding to each semantic label. That is, the visual representations of the images are clustered according to the semantic labels to obtain the visual representations of each semantic label. For example, refer to Figure 3As shown in , for semantic label 1, the images associated with it include image A, image B, and image C. The visual representation of image A includes representation a and representation b; the visual representation of image B includes representation c; and the visual representation of image C includes representation d, representation e, and representation f. Since there are multiple images associated with semantic label 1, the visual representation of all images can be combined to obtain the visual representation of the semantic label. That is, the visual representation of semantic label 1 includes representation a, representation b, representation c, representation d, representation e, and representation f. For example, when the semantic label is beverage, its visual representations include bottled, bagged, canned, boxed, and so on.

[0036] In step S220, a label system corresponding to the application system is established based on the semantic labels and the visual representation.

[0037] In the disclosed embodiments, the labeling system includes multiple levels of labels, and the number of levels can be determined based on actual conditions. Multiple levels of labels have a hierarchical relationship, so the labeling system of the application system can be determined based on the hierarchical relationship between the multiple levels of labels. Different application systems have different ways of determining the labeling system, but the most detailed classification labels of different application systems remain consistent.

[0038] In the disclosed embodiment, the application system may include a labeling system and a training system. The labeling system is used to implement data collection, and the training system is used to implement model training. For the labeling system and the training system, their first-level labels are determined according to the usage scenario and the reference object itself. For example, when the usage scenario is object recognition and the reference object is an object, its first-level label is object. When the reference object is a document, its first-level label is document. When the reference object is a code class, its first-level label is a code class. For example, in a shopping scenario, the usage scenario may be product recognition, and the first-level label is product.

[0039] For both the annotation system and the training system, secondary labels and subordinate labels can be jointly determined based on semantic labels and visual representations. Secondary labels can be the next level down from primary labels, and subordinate labels can be the next level down from secondary labels.

[0040] Figure 4 A flow chart for determining a labeling system is schematically shown in FIG. Figure 4 As shown in , it mainly includes the following steps:

[0041] In step S410, if the application system is a tagging system, the semantic tag is used as a secondary tag, the joint tag consisting of the semantic tag and the visual representation is used as a subordinate tag of the secondary tag, and the tag system of the tagging system is determined based on the primary tag, the secondary tag and the subordinate tag.

[0042] In an embodiment of the present disclosure, for the labeling system, the reference object can be used as a first-level label, the semantic label can be used as a second-level label, and the joint label composed of the semantic label and the visual representation can be used as the next-level label of the second-level label, that is, the subordinate label. Among them, each semantic label can be split according to the visual representation to obtain a plurality of joint labels composed of the visual representation and the semantic label. Splitting the semantic label according to the visual representation can be understood as combining the semantic label and the visual representation to obtain a joint label with a range smaller than the semantic label. The joint label can be obtained by combining the visual representation and the semantic label, or by fusing the visual representation and the semantic label and then optimizing them (the optimization can be simplified processing, etc.). Here, the combination is used as an example for explanation. Furthermore, the label system of the labeling system is determined based on the first-level label, the second-level label and the subordinate label.

[0043] For example, refer to Figure 5 As shown in , the reference object (commodity) is used as the primary label, and the semantic label (beverage, milk) is used as the secondary label. For the semantic label "beverage", its visual representation includes bottled, bagged, canned, boxed, etc. For the semantic label "milk", its visual representation also includes bottled, bagged, canned, boxed, etc. Furthermore, the semantic labels can be classified according to the visual representation, and a joint label is obtained as a subordinate label based on the combination of the semantic label and the visual representation. The subordinate labels may include but are not limited to: bottled beverages, bagged beverages, canned beverages, boxed beverages, bottled milk, bagged milk, canned milk, boxed milk. Based on this, a label system can be formed according to the commodity, beverage, and milk, bottled beverages, bagged beverages, canned beverages, boxed beverages, bottled milk, bagged milk, canned milk, boxed milk. By repeatedly splitting the semantic labels according to the visual representation, the subordinate labels corresponding to each semantic label (secondary label) can be obtained, and then the label system of the annotation system can be determined based on the primary label, multiple secondary labels, and the subordinate labels corresponding to each secondary label.

[0044] In step S420, if the application system is a training system, the visual representation is used as a secondary label, the joint label composed of the semantic label and the visual representation is used as a subordinate label of the secondary label, and the label system of the training system is determined based on the primary label, the secondary label and the subordinate label.

[0045] In an embodiment of the present disclosure, for a training system, a reference object can be used as a primary label, a visual representation can be used as a secondary label, and a joint label composed of a semantic label and the visual representation can be used as the next-level label of the secondary label, i.e., a subordinate label. Among them, multiple joint labels composed of visual representation and semantic labels are clustered according to the visual representation to form a visual representation label as the secondary label. Clustering joint labels according to visual representation can be understood as clustering all joint labels with the same visual representation in the joint labels that combine semantic labels and visual representations to obtain multiple secondary labels for representing different visual representations. Based on this, each secondary label can include different types of objects, and the visual representations of different types of objects are the same. For example, the secondary label can be bottled, canned, and so on. When the secondary label is bottled, it can include all bottled objects, that is, it can include bottled drinks and bottled milk. Further, the label system of the training system is determined based on the primary label, the secondary label, and the subordinate label.

[0046] For example, refer to Figure 6 As shown in , in the shopping scenario, the reference object (commodity) is used as the first-level label. For the semantic labels of beverages and milk, their visual representations include bottled, bagged, canned, boxed, etc. Furthermore, a joint label can be obtained as a subordinate label based on the combination of semantic labels and visual representations. Subordinate labels may include but are not limited to: bottled beverages, bagged beverages, canned beverages, boxed beverages, bottled milk, bagged milk, canned milk, boxed milk. According to the visual representation, the joint labels are clustered to obtain the visual representation as the second-level label, and the visual representation is used as the second-level label, and the second-level label includes bottled, canned, bagged, and boxed. Based on this, a label system can be formed according to the second-level labels represented by commodities, bottles, cans, bags, and boxes, and the subordinate labels represented by bottled beverages, bagged beverages, canned beverages, boxed beverages, bottled milk, bagged milk, canned milk, and boxed milk. This process is repeated to obtain the corresponding sub-labels for each visual representation label (secondary label). The label system of the training system is then determined based on the primary label, all secondary labels, and the corresponding sub-labels. It should be noted that the label system of the annotation system can be included in the label system of the training system, but the scope of the label system of the training system is larger than that of the annotation system.

[0047] Figure 7 The flowchart of building a label system is shown schematically in Figure 7 As shown in , it mainly includes the following steps:

[0048] In step S701, semantic tags are determined. For example, multiple semantic tags may be determined based on a category list of a shopping platform and common objects.

[0049] In step S702 , images corresponding to semantic tags are collected and possible visual representations of the semantic tags are analyzed.

[0050] In step S703, each semantic tag is split according to the visual representation to obtain a joint tag consisting of multiple visual representations and semantic tags.

[0051] In step S704 , during labeling, the joint label consisting of the visual representation and the semantic label is used as a subordinate label of the semantic label.

[0052] In step S705 , during training, a joint label composed of the visual representation and the semantic label is used to generate a secondary label of the previous level according to the visual representation.

[0053] In step S706 , the annotation and the finest training label are kept consistent to facilitate mapping.

[0054] In the disclosed embodiment, by combining semantic labels and visual representations to construct a labeling system for the annotation system and the training system, the limitation of establishing a labeling system based only on semantics is avoided, the application scope and completeness of the constructed labeling system are increased, and the accuracy is also improved.

[0055] It should be noted that in the embodiment of the present disclosure, the entire labeling system is not limited to three levels of labels, and can be a labeling system with four or more levels. On this basis, the annotation system can continue to split the subordinate labels according to other features to update the subordinate labels, and repeat the above steps until there are no other features that can be split. The training system can continue to aggregate the secondary labels according to other features to update the secondary labels, and repeat the above steps until there are no other features that can be aggregated. Other features are any type of information associated with the item, such as evaluation information or recommendation information, etc.

[0056] Next, continue to refer to Figure 2 As shown in , in step S230 , the image to be processed is annotated and identified based on the label system to determine the object position and object category in the image to be processed.

[0057] In the disclosed embodiments, the image to be processed refers to an image that needs to be recognized. The image to be processed can be captured in real time by a camera called in an application, or it can be a pre-stored image. Object detection can be performed on the image to be processed using a target detection model. The model can be trained based on the label system of the training system to obtain the target detection model.

[0058] It should be noted that before model training, manual collection can be performed to obtain sub-labels. Since the sub-labels of the annotation system are the same as those of the training system, the sub-labels of the images obtained by the annotation system can be directly mapped to the categories to be trained for model training. During manual collection, multiple images can be captured based on the semantic labels. When annotating the images, a semantic label corresponding to the captured image can be selected from the semantic labels provided by the system. After determining the semantic label, the sub-labels can be displayed based on the label system, and the sub-label of the image can be determined based on the selection operation. Since the sub-labels are composed of visual representations and semantic labels, the most detailed category label for each image can be determined based on the sub-labels. This avoids the limitations of classification based solely on visual representations and improves versatility and label accuracy. Furthermore, since the most detailed category labels of the data acquisition and annotation system are consistent with the training system, the annotated sub-labels can be directly mapped into the training process, allowing model training to be performed based on the sub-labels for the categories to be trained (i.e., the categories corresponding to the sub-labels). For example, multiple images can be captured for each semantic label, such as beverage, milk, and television. Image 1 is annotated using the semantic labels represented by the secondary labels of the annotation system to obtain the semantic labels corresponding to image 1. Furthermore, the sub-labels of image 1 are determined based on the labeling system of the annotation system to obtain the finest classification labels for image 1. Based on this, the sub-labels of image 1 are mapped to the categories to be trained (the categories corresponding to the sub-labels), facilitating model training using the secondary labels of the training system.

[0059] When training the model, specifically, the neural network model can be merged and trained based on the secondary labels (visual representation labels) in the label system corresponding to the training system, and the model parameters of the neural network model can be adjusted until the model converges to obtain a target detection model. By merging the model with the secondary labels of the training system, when the granularity required for the classification result is large (for example, when the classification result is a label or a secondary label), the problem of low classification scores and classification errors caused by the inability to distinguish details when training the model based on the finest classification label (lower-level label) is avoided, and the accuracy and versatility of the classification are improved, avoiding the situation where large differences within the class and small differences between classes are caused, and the function of small differences within the class and large differences between classes can be achieved, thereby improving the accuracy of the model and improving the accuracy of object detection.

[0060] The target detection model is used to identify which objects are present in the image and determine the location of the objects. The location of the objects can be marked according to the identification box. In the embodiment of the present disclosure, the output result of the target detection model can also be the location of the object and the object category. Among them, the object category can be determined according to the primary label or secondary label in the label system. For example Figure 8As shown in FIG, an image to be processed 801 is input, and after passing through the target detection model 802, the output result is the object position 803 and the object category 804. The object category can be a commodity, or a bottle or a can.

[0061] The technical solution in the embodiment of the present disclosure constructs a labeling system through semantic labels and visual representations, which can obtain a more accurate labeling system. Furthermore, the model is merged and trained based on the secondary labels in the labeling system. Since the most detailed labels of different application systems remain consistent, the annotation system can be directly mapped to the training system, and the classification branch of the training test can be directly classified on the visual representation label, avoiding the situation where the difference within the class is large and the difference between the classes is small. The accuracy of the model is improved, the limitations of classification based only on visual representations or semantic labels are avoided, and the accuracy and comprehensiveness are improved.

[0062] The present disclosure provides an image recognition device, referring to Figure 9 As shown in , the image recognition device 900 may include:

[0063] A visual representation acquisition module 901 is configured to acquire multiple semantic tags by referring to an object, and to acquire a visual representation corresponding to each of the semantic tags;

[0064] A label system building module 902 is used to build a label system corresponding to the application system based on the semantic labels and the visual representations;

[0065] The labeling and recognition module 903 is configured to perform labeling and recognition on the image to be processed based on the labeling system, and determine the position and category of objects in the image to be processed.

[0066] In an exemplary embodiment of the present disclosure, the label system construction module includes: a construction control module for determining multiple level labels according to the semantic labels and the visual representation, and establishing a label system corresponding to the application system according to the multiple level labels.

[0067] In an exemplary embodiment of the present disclosure, the construction control module includes: a first construction module, which is used to, if the application system is a labeling system, use the semantic label as a secondary label, use the joint label composed of the semantic label and the visual representation as a subordinate label of the secondary label, and determine the label system of the labeling system based on the primary label, the secondary label and the subordinate label; a second construction module, which is used to, if the application system is a training system, use the visual representation label as a secondary label, use the joint label composed of the semantic label and the visual representation as a subordinate label, and determine the label system of the training system based on the primary label, the secondary label and the subordinate label.

[0068] In an exemplary embodiment of the present disclosure, the apparatus further includes: a label splitting module, configured to split each of the semantic labels according to the visual representation to obtain a plurality of joint labels consisting of the visual representation and the semantic label.

[0069] In an exemplary embodiment of the present disclosure, the second building module includes: a label clustering module for clustering multiple joint labels consisting of visual representations and semantic labels according to visual representations to form visual representation labels and use the visual representation labels as the secondary labels.

[0070] In an exemplary embodiment of the present disclosure, the annotation recognition module includes: a model training module, which is used to perform model training based on the label system to obtain a target detection model; an object detection module, which is used to input the image to be processed into the target detection model for detection processing, obtain the object position in the image to be processed, and obtain the object category; the object category is a primary label or a secondary label.

[0071] In an exemplary embodiment of the present disclosure, the model training module includes: a training control module, which is used to perform model training based on visual representation labels in a label system corresponding to the training system to obtain the target detection model.

[0072] It should be noted that the specific details of each module in the above-mentioned image recognition device have been described in detail in the corresponding image recognition method, and therefore will not be repeated here.

[0073] Figure 10 Schematic diagram of an electronic device suitable for implementing an exemplary embodiment of the present disclosure is shown. The terminal of the present disclosure can be configured as follows Figure 10 The form of the electronic device shown, however, needs to be explained. Figure 10 The electronic device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present disclosure.

[0074] The electronic device of the present disclosure includes at least a processor and a memory, where the memory is used to store one or more programs. When the one or more programs are executed by the processor, the processor can implement the method of the exemplary embodiment of the present disclosure.

[0075] Specifically, such as Figure 10As shown, the electronic device 1000 may include: a processor 1010, an internal memory 1021, an external memory interface 1022, a Universal Serial Bus (USB) interface 1030, a charging management module 1040, a power management module 1041, a battery 1042, an antenna 1, an antenna 2, a mobile communication module 1050, a wireless communication module 1060, an audio module 1070, a speaker 1071, a receiver 1072, a microphone 1073, an earphone interface 1074, a sensor module 1080, a display screen 1090, a camera module 1091, an indicator 1092, a motor 1093, a button 1094 and a subscriber identification module (Subscriber Identification Module, SIM) card interface 1095, etc. The sensor module 1080 may include a depth sensor, a pressure sensor, a gyroscope sensor, an air pressure sensor, a magnetic sensor, an acceleration sensor, a distance sensor, a proximity light sensor, a fingerprint sensor, a temperature sensor, a touch sensor, an ambient light sensor, and a bone conduction sensor, etc.

[0076] It is understood that the structures illustrated in the embodiments of the present application do not constitute a specific limitation on the electronic device 1000. In other embodiments of the present application, the electronic device 1000 may include more or fewer components than shown, or may combine or separate certain components, or arrange the components differently. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

[0077] The processor 1010 may include one or more processing units, for example: the processor 1010 may include an application processor, a modem processor, a graphics processor, an image signal processor, a controller, a video codec, a digital signal processor, a baseband processor and / or a neural network processor (Neural-etwork Processing Unit, NPU), etc. Among them, different processing units can be independent devices or integrated into one or more processors. In addition, a memory can be provided in the processor 1010 for storing instructions and data. The image processing method in this exemplary embodiment can be executed by an application processor, a graphics processor or an image signal processor. When the method involves processing related to a neural network, it can be executed by an NPU.

[0078] The internal memory 1021 can be used to store computer executable program code, which includes instructions. The internal memory 1021 can include a program storage area and a data storage area. The external memory interface 1022 can be used to connect an external memory card, such as a Micro SD card, to expand the storage capacity of the electronic device 1000.

[0079] The communication functions of mobile terminal 1000 are implemented through a mobile communication module, antenna 1, a wireless communication module, antenna 2, a modem processor, and a baseband processor. Antenna 1 and antenna 2 are used to transmit and receive electromagnetic wave signals. The mobile communication module can provide 2G, 3G, 4G, and 5G mobile communication solutions for mobile terminal 1000. The wireless communication module can provide wireless communication solutions such as wireless LAN, Bluetooth, and near-field communication for mobile terminal 200.

[0080] The display module is used to implement display functions, such as displaying user interfaces, images, and videos. The camera module is used to implement shooting functions, such as capturing images and videos. The audio module is used to implement audio functions, such as playing audio and capturing voice. The power module is used to implement power management functions, such as charging the battery, powering the device, and monitoring the battery status.

[0081] The present application also provides a computer-readable storage medium, which may be included in the electronic device described in the above embodiment; or may exist independently without being assembled into the electronic device.

[0082] Computer-readable storage media can be, for example, but not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices, or any combination thereof. More specific examples of computer-readable storage media can include, but are not limited to, an electrical connection having one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In the present disclosure, a computer-readable storage medium can be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, device, or device.

[0083] Computer-readable storage media can transmit, propagate, or transfer programs for use by or in conjunction with an instruction execution system, apparatus, or device. Program code contained on a computer-readable storage medium can be transmitted using any suitable medium, including but not limited to wireless, wireline, optical cable, RF, or any suitable combination thereof.

[0084] The computer-readable storage medium carries one or more programs. When the one or more programs are executed by an electronic device, the electronic device implements the method described in the following embodiments.

[0085] Through the description of the above embodiments, it is easy for those skilled in the art to understand that the example embodiments described herein can be implemented by software or by combining software with necessary hardware. Therefore, the technical solution according to the embodiments of the present disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash drive, a mobile hard disk, etc.) or on a network, and includes several instructions to enable a computing device (which can be a personal computer, a server, a terminal device, or a network device, etc.) to execute the method according to the embodiments of the present disclosure.

[0086] Furthermore, the figures above are merely illustrative of the processes included in the methods according to exemplary embodiments of the present disclosure and are not intended to be limiting. It is readily understood that the processes illustrated in the figures above do not indicate or limit the temporal order of these processes. Furthermore, it is readily understood that these processes may be executed synchronously or asynchronously, for example, in multiple modules.

[0087] It should be noted that although several modules or units of the device for action execution are mentioned in the detailed description above, this division is not mandatory. In fact, according to the embodiments of the present disclosure, the features and functions of two or more modules or units described above can be concretized in one module or unit. Conversely, the features and functions of one module or unit described above can be further divided into multiple modules or units to be concretized.

[0088] Those skilled in the art will readily conceive of other embodiments of the present disclosure after considering the specification and practicing what is disclosed herein. This application is intended to cover any variations, uses, or adaptations of the present disclosure that follow the general principles of the present disclosure and include common knowledge or customary technical means in the art that are not disclosed in the present disclosure. The description and examples are to be regarded as exemplary only, and the true scope and spirit of the present disclosure are indicated by the claims. It should be understood that the present disclosure is not limited to the precise structures described above and shown in the accompanying drawings, and that various modifications and changes can be made without departing from its scope. The scope of the present disclosure is limited only by the appended claims.

Claims

1. An image recognition method, characterized in that: include: Acquire multiple semantic labels by referring to the object, and acquire visual representations corresponding to the respective semantic labels; Determining a plurality of level labels according to the semantic label and the visual representation, and establishing a label system corresponding to the application system according to the plurality of level labels; Annotate and identify the image to be processed based on the label system to determine the location and category of objects in the image to be processed; The step of determining multiple level labels based on the semantic labels and the visual representation, and establishing a label system corresponding to the application system based on the multiple level labels, includes: If the application system is a tagging system, the semantic tag is used as a secondary tag, a joint tag consisting of the semantic tag and the visual representation is used as a subordinate tag of the secondary tag, and a tag system of the tagging system is determined based on the primary tag, the secondary tag, and the subordinate tag; If the application system is a training system, the visual representation label is used as the secondary label, the joint label composed of the semantic label and the visual representation is used as the subordinate label, and the label system of the training system is determined based on the primary label, the secondary label and the subordinate label.

2. The image recognition method according to claim 1, wherein: The method further comprises: Each of the semantic labels is split according to the visual representation to obtain a plurality of joint labels consisting of the visual representation and the semantic label.

3. The image recognition method according to claim 1, wherein: The step of using the visual representation label as a secondary label includes: A plurality of joint labels consisting of visual representations and semantic labels are clustered according to the visual representations to form visual representation labels, and the visual representation labels are used as the secondary labels.

4. The image recognition method according to claim 1, wherein: The tagging and identifying the image to be processed based on the tag system to determine the position and category of the object in the image to be processed includes: Perform model training based on the label system to obtain a target detection model; The image to be processed is input into the target detection model for detection processing, the object position in the image to be processed is obtained, and the object category is obtained; the object category is a primary label or a secondary label.

5. The image recognition method according to claim 4, characterized in that The model training based on the label system to obtain the target detection model includes: Model training is performed based on the visual representation labels in the label system corresponding to the training system to obtain the target detection model.

6. An image recognition device, characterized in that: include: A visual representation acquisition module, configured to acquire a plurality of semantic tags by referring to an object, and to acquire a visual representation corresponding to each of the semantic tags; a label system building module, configured to determine a plurality of level labels based on the semantic labels and the visual representation, and to establish a label system corresponding to the application system based on the plurality of level labels; A labeling and recognition module, configured to perform labeling and recognition on the image to be processed based on the labeling system, and determine the position and category of objects in the image to be processed; The step of determining multiple level labels based on the semantic labels and the visual representation, and establishing a label system corresponding to the application system based on the multiple level labels, includes: If the application system is a tagging system, the semantic tag is used as a secondary tag, a joint tag consisting of the semantic tag and the visual representation is used as a subordinate tag of the secondary tag, and a tag system of the tagging system is determined based on the primary tag, the secondary tag, and the subordinate tag; If the application system is a training system, the visual representation label is used as the secondary label, the joint label consisting of the semantic label and the visual representation is used as the subordinate label, and the label system of the training system is determined based on the primary label, the secondary label and the subordinate label.

7. An electronic device, characterized in that: include: processor; as well as a memory for storing executable instructions of the processor; The processor is configured to execute the image recognition method according to any one of claims 1 to 5 by executing the executable instructions.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the image recognition method according to any one of claims 1 to 5 is implemented.

Images