Image recognition method, device, storage medium and electronic device

Abstract

The present application provides an image recognition method, device, storage medium and electronic device, which relate to the field of image recognition. Among them, the electronic device receives an image to be recognized and a recognition instruction of the image to be recognized; determines the auxiliary operation instruction required to complete the recognition instruction for the current image to be recognized; performs an editing operation corresponding to the auxiliary operation instruction on the current image to be recognized to obtain an updated image to be recognized; determines whether the recognition result of the current image to be recognized meets the recognition accuracy; if not, returns to the step of determining the auxiliary operation instruction required to complete the recognition instruction for the current image to be recognized, until a recognition result that meets the recognition accuracy is obtained. In this way, by adding recognition auxiliary information through a series of multimodal data fusion and editing operations, the defect of one-time recognition of an image by a single modality is overcome, and the accuracy and robustness of image recognition are significantly improved.

Description

Technical Field

[0001] The present application relates to the field of image recognition, and more specifically, to an image recognition method, device, storage medium and electronic device. Background Art

[0002] Most existing image recognition technologies use a single modality to perform one-time end-to-end recognition of images. For example, they directly output recognition results based on visual information in the input image. However, in actual applications, it is found that this method of using a single model to perform one-time recognition of images is difficult to fully and accurately judge the image content when faced with complex and changing scenes. For example, in environments such as construction, factors such as lighting conditions, occlusion, and changes in viewing angles may lead to inaccurate recognition results. Summary of the invention

[0003] In order to overcome at least one of the deficiencies in the prior art, the present application provides an image recognition method, device, storage medium and electronic device, specifically including:

[0004] In a first aspect, the present application provides an image recognition method, the method comprising:

[0005] Receiving an image to be identified and an identification instruction of the image to be identified;

[0006] Determine the auxiliary operation instructions required to complete the recognition instruction for the current image to be recognized;

[0007] Performing an editing operation corresponding to the auxiliary operation instruction on the current image to be identified to obtain an updated image to be identified;

[0008] Determine whether the recognition result of the current image to be recognized meets the recognition accuracy;

[0009] If not, the process returns to the step of determining the auxiliary operation instructions required to complete the recognition instruction for the current image to be recognized, until a recognition result that satisfies the recognition accuracy is obtained.

[0010] In conjunction with the optional implementation manner of the first aspect, the determining of the auxiliary operation instruction required to complete the recognition instruction for the current image to be recognized includes:

[0011] Obtain context information generated by calling the image recognition strategy model;

[0012] The image recognition strategy model is called to process the context information and the current image to be recognized, and the auxiliary operation instruction generated by the image recognition strategy model is obtained.

[0013] In combination with the optional implementation manner of the first aspect, performing an editing operation corresponding to the auxiliary operation instruction on the current image to be recognized to obtain an updated image to be recognized includes:

[0014] Get description information of multiple image recognition tools;

[0015] Calling an image editing model to process the description information and the auxiliary operation instruction to generate an image editing code, wherein the image editing code is used to provide an editing operation corresponding to the auxiliary operation instruction;

[0016] The image editing code is called to process the current image to be identified to obtain an updated image to be identified.

[0017] In conjunction with the optional implementation manner of the first aspect, the context information includes:

[0018] At least one historical auxiliary operation instruction generated by the image recognition strategy model, a historical editing operation corresponding to each of the historical auxiliary operation instructions, and description information of the historical image to be recognized after each of the historical editing operations is executed.

[0019] In conjunction with the optional implementation manner of the first aspect, the determining whether the recognition result of the current image to be recognized meets the recognition accuracy includes:

[0020] Calling multiple image recognition models to process the recognition instruction and the current image to be recognized to obtain multiple recognition results;

[0021] Determining whether the recognition results of the multiple image recognition models are consistent according to the multiple recognition results;

[0022] If so, a recognition result satisfying the recognition accuracy is obtained.

[0023] In conjunction with the optional implementation manner of the first aspect, the determining whether the recognition results of the multiple image recognition models are consistent based on the multiple recognition results includes:

[0024] Obtaining divergence indicators of the multiple image recognition models according to the multiple recognition results;

[0025] If the divergence index is less than the divergence threshold, then the recognition results of the multiple image recognition models are consistent.

[0026] In conjunction with an optional implementation manner of the first aspect, the multiple recognition results include a first recognition result and / or a second recognition result, the first recognition result and the second recognition result are mutually exclusive, and the relationship between the multiple recognition results and the divergence indicator is:

[0027]

[0028] In the formula, represents the divergence index, represents the number of the plurality of image recognition models, Represents the current image to be recognized. represents the identification instruction, Indicates The recognition result of the image recognition model for the current image to be recognized is the first recognition result. Indicates The recognition result of the image recognition model for the current image to be recognized is the second recognition result.

[0029] In a second aspect, the present application provides an image recognition device, the device comprising:

[0030] An image acquisition module, used for receiving an image to be identified and an identification instruction of the image to be identified;

[0031] An auxiliary editing module, used for determining the auxiliary operation instructions required to complete the recognition instruction for the current image to be recognized;

[0032] The auxiliary editing module is further used to perform an editing operation corresponding to the auxiliary operation instruction on the current image to be identified, so as to obtain an updated image to be identified;

[0033] An image recognition module is used to determine whether the recognition result of the current image to be recognized meets the recognition accuracy;

[0034] If not, the process returns to the step of determining the auxiliary operation instructions required to complete the recognition instruction for the current image to be recognized, until a recognition result that satisfies the recognition accuracy is obtained.

[0035] In conjunction with the optional implementation manner of the second aspect, the auxiliary editing module is further specifically used for:

[0036] Obtain context information generated by calling the image recognition strategy model;

[0037] The image recognition strategy model is called to process the context information and the current image to be recognized, and the auxiliary operation instruction generated by the image recognition strategy model is obtained.

[0038] In conjunction with the optional implementation manner of the second aspect, the auxiliary editing module is further specifically used for:

[0039] Get description information of multiple image recognition tools;

[0040] Calling an image editing model to process the description information and the auxiliary operation instruction to generate an image editing code, wherein the image editing code is used to provide an editing operation corresponding to the auxiliary operation instruction;

[0041] The image editing code is called to process the current image to be identified to obtain an updated image to be identified.

[0042] In conjunction with the optional implementation manner of the second aspect, the context information includes:

[0043] At least one historical auxiliary operation instruction generated by the image recognition strategy model, a historical editing operation corresponding to each of the historical auxiliary operation instructions, and description information of the historical image to be recognized after each of the historical editing operations is executed.

[0044] In conjunction with the optional implementation manner of the second aspect, the image recognition module is further specifically used for:

[0045] Calling multiple image recognition models to process the recognition instruction and the current image to be recognized to obtain multiple recognition results;

[0046] Determining whether the recognition results of the multiple image recognition models are consistent according to the multiple recognition results;

[0047] If so, a recognition result satisfying the recognition accuracy is obtained.

[0048] In conjunction with the optional implementation manner of the second aspect, the image recognition module is further specifically used for:

[0049] Obtaining divergence indicators of the multiple image recognition models according to the multiple recognition results;

[0050] If the divergence index is less than the divergence threshold, then the recognition results of the multiple image recognition models are consistent.

[0051] In conjunction with an optional implementation manner of the second aspect, the multiple recognition results include a first recognition result and / or a second recognition result, the first recognition result and the second recognition result are mutually exclusive, and the relationship between the multiple recognition results and the divergence indicator is:

[0052]

[0053] In the formula, represents the divergence index, represents the number of the plurality of image recognition models, Represents the current image to be recognized. represents the identification instruction, Indicates The recognition result of the image recognition model for the current image to be recognized is the first recognition result. Indicates The recognition result of the image recognition model for the current image to be recognized is the second recognition result.

[0054] In a third aspect, the present application further provides a storage medium, wherein the storage medium stores a computer program, and when the computer program is executed by a processor, the image recognition method is implemented.

[0055] In a fourth aspect, the present application further provides an electronic device, comprising a processor and a memory, wherein the memory stores a computer program, and when the computer program is executed by the processor, the image recognition method is implemented.

[0056] Compared with the prior art, this application has the following beneficial effects:

[0057] The present application provides an image recognition method, device, storage medium and electronic device. Among them, the electronic device receives an image to be recognized and a recognition instruction of the image to be recognized; determines the auxiliary operation instruction required to complete the recognition instruction for the current image to be recognized; performs an editing operation corresponding to the auxiliary operation instruction on the current image to be recognized to obtain an updated image to be recognized; determines whether the recognition result of the current image to be recognized meets the recognition accuracy; if not, returns to the step of determining the auxiliary operation instruction required to complete the recognition instruction for the current image to be recognized, until a recognition result that meets the recognition accuracy is obtained. In this way, by adding recognition auxiliary information through a series of multimodal data fusion and editing operations, the defect of one-time recognition of an image by a single modality is overcome, and the accuracy and robustness of image recognition are significantly improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0058] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the embodiments will be briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present application and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0059] Figure 1 A schematic diagram of the process of an image recognition method provided in an embodiment of the present application;

[0060] Figure 2 A schematic diagram of a thought chain provided for an embodiment of the present application;

[0061] Figure 3 A schematic diagram of the structure of an image recognition device provided in an embodiment of the present application;

[0062] Figure 4 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0063] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, rather than all the embodiments. The components of the embodiments of the present application described and shown in the drawings here can be arranged and designed in various different configurations.

[0064] Therefore, the following detailed description of the embodiments of the present application provided in the accompanying drawings is not intended to limit the scope of the present application for which protection is sought, but merely represents selected embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in the field without creative work are within the scope of protection of the present application.

[0065] It should be noted that similar reference numerals and letters denote similar items in the following drawings, and therefore, once an item is defined in one drawing, further definition and explanation thereof is not required in subsequent drawings.

[0066] In the description of the present application, it should be noted that the terms "center", "upper", "lower", "left", "right", "vertical", "horizontal", "inside", "outside", etc. indicate positions or positional relationships based on the positions or positional relationships shown in the accompanying drawings, or the positions or positional relationships in which the invented product is usually placed when in use. They are only for the convenience of describing the present application and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific position, be constructed and operated in a specific position, and therefore cannot be understood as a limitation on the present application. In addition, the terms "first", "second", "third", etc. are only used to distinguish the description, and cannot be understood as indicating or implying relative importance.

[0067] In addition, the terms "comprises," "comprising," or any other variation thereof are intended to encompass non-exclusive inclusion, such that a process, method, article, or apparatus that includes a list of elements includes not only those elements, but also other elements not explicitly listed, or elements inherent to such process, method, article, or apparatus. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of additional identical elements in the process, method, article, or apparatus that includes the element.

[0068] In addition, the terms "horizontal", "vertical", "overhanging" and the like do not mean that the components are required to be absolutely horizontal or overhanging, but can be slightly tilted. For example, "horizontal" only means that its direction is more horizontal than "vertical", and does not mean that the structure must be completely horizontal, but can be slightly tilted.

[0069] In the description of this application, it should also be noted that, unless otherwise clearly specified and limited, the terms "set", "install", "connect", and "connect" should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection, or it can be indirectly connected through an intermediate medium, or it can be the internal communication of two elements. For ordinary technicians in this field, the specific meanings of the above terms in this application can be understood according to specific circumstances.

[0070] Based on the above statement, as introduced in the background technology, most of the existing image recognition technologies use a single modality to perform one-time end-to-end recognition on images, for example, directly outputting recognition results only by relying on the visual information in the input image. However, in actual applications, it is found that this method of using a single model to perform one-time recognition on images is difficult to comprehensively and accurately judge the image content when facing complex and changing scenes.

[0071] For this, let's take image recognition in a construction scene at a construction site as an example. In a construction scene, safety is always a crucial consideration. Construction scenes are usually complex and changeable, and there are many potential safety risks. Among them, the correct wearing of seat belts plays a key role in protecting the lives of construction workers. With the continuous development of engineering construction, the scale of construction is expanding, and the safety management requirements for construction workers are also getting higher and higher. Traditional seat belt wearing inspections mainly rely on manual inspections, which is not only inefficient, but also prone to omissions and misjudgments.

[0072] Therefore, in the related art, a seat belt wearing recognition method based on computer vision is proposed. These methods usually collect images or video data of construction workers through cameras, and then use image recognition algorithms to determine whether the seat belts are worn correctly. However, simple visual detection methods are often greatly affected by environmental factors, such as lighting conditions, occlusion, and viewing angle. In a complex construction environment, the light may be insufficient or uneven, resulting in a decrease in image quality; construction equipment and structures may block construction workers, making the seat belts partially or completely invisible; different shooting angles may also affect the recognition accuracy of the seat belts. Therefore, most of the existing seat belt wearing recognition technologies use a single modality to perform a one-time recognition of the image of the construction scene, which is actually difficult to comprehensively and accurately judge the wearing status of the seat belt. For example, it may not be possible to determine whether the seat belt is correctly fastened or whether there is damage based on visual images alone.

[0073] Based on the discovery of the above technical problems, the inventors have proposed the following technical solutions to solve or improve the above problems through creative work. It should be noted that the defects in the solutions in the above prior art are the results obtained by the inventors after practice and careful research. Therefore, the discovery process of the above problems and the solutions proposed in the embodiments of the present application for the above problems below should all be the contributions made by the inventors to the present application in the process of invention and creation, and should not be understood as technical contents known to those skilled in the art.

[0074] In view of this, the present embodiment provides an image recognition method. In the method, an electronic device receives an image to be recognized and a recognition instruction of the image to be recognized; determines the auxiliary operation instruction required to complete the recognition instruction for the current image to be recognized; performs an editing operation corresponding to the auxiliary operation instruction on the current image to be recognized to obtain an updated image to be recognized; determines whether the recognition result of the current image to be recognized meets the recognition accuracy; if not, returns to the step of determining the auxiliary operation instruction required to complete the recognition instruction for the current image to be recognized, until a recognition result that meets the recognition accuracy is obtained. In this way, by adding recognition auxiliary information through a series of multimodal data fusion and editing operations, the defect of one-time recognition of an image by a single modality is overcome, and the accuracy and robustness of image recognition are significantly improved.

[0075] It is worth noting that the electronic device implementing the method may be, but is not limited to, a mobile terminal, a tablet computer, a laptop computer, a desktop computer, and a server, etc. Among them, the server may be a single server or a server group. The server group may be centralized or distributed (for example, the server may be a distributed system). In some embodiments, the server may be local or remote relative to the user terminal. In some embodiments, the server may be implemented on a cloud platform; as an example only, the cloud platform may include a private cloud, a public cloud, a hybrid cloud, a community cloud (Community Cloud), a distributed cloud, an inter-cloud (Inter-Cloud), a multi-cloud (Multi-Cloud), etc., or any combination thereof. In some embodiments, the server may be implemented on an electronic device having one or more components.

[0076] To make the solution provided by this embodiment clearer, the following uses a server as an electronic device to implement the method, and combines Figure 1Each step of the method is described in detail. However, it should be understood that the operations of the flowchart may not be implemented in order, and steps without logical contextual relationships may be reversed in order or implemented simultaneously. In addition, those skilled in the art may add one or more other operations to the flowchart, or remove one or more operations from the flowchart under the guidance of the content of this application. Figure 1 As shown, the method includes:

[0077] S1, receiving an image to be recognized and a recognition instruction of the image to be recognized.

[0078] In this embodiment, the image to be identified can come from scenes such as construction, security monitoring, medical diagnosis, and automatic driving. The recognition instruction can be in text form or voice form. Taking this form as an example, when the image to be identified is an image taken in a construction scene, and the image needs to be used to detect the wearing of the construction workers' seat belts. At this time, the recognition instruction can be selected from the three semantically mutually exclusive texts of "correctly wearing seat belts", "wrongly wearing seat belts", and "no seat belts", which are used to instruct the multimodal image recognition model to determine whether the content in the image is consistent with the semantics in the recognition instruction. Of course, the expression method of the voice recognition instruction is not limited to the above method, and those skilled in the art can make adaptive adjustments. For example, the above three recognition instructions can also be expressed as "determine whether to wear seat belts correctly", "determine whether to wear seat belts incorrectly", and "determine whether there are no seat belts".

[0079] Based on the introduction of the above embodiment to the image to be recognized and the recognition instruction, continue to refer to Figure 1 , the image recognition method provided in this embodiment also includes:

[0080] S2, determining the auxiliary operation instructions required to complete the recognition instructions for the current image to be recognized.

[0081] It should be noted that in the current image recognition method, the server receives the image to be recognized, inputs the image to be recognized into a pre-trained image recognition model, and directly obtains the recognition result. In other words, the current image recognition method only recognizes the image once. In contrast, in this embodiment, in order to improve the accuracy and reliability of recognition, it is necessary to perform at least one reasoning and judgment on the image to be recognized according to the thinking chain of "mining observation-execution plan-inference and judgment" until the result corresponding to the recognition instruction is obtained. Each time a reasoning and judgment is performed, auxiliary information that facilitates recognition and judgment will be added to the received image to be recognized. Therefore, for the received image to be recognized, after each round of reasoning and judgment, the image content changes due to the editing operation. In this embodiment, the current image to be recognized refers to the image after the most recent editing operation. In order to enable a clearer understanding of the above-mentioned thinking chain, each link in the above-mentioned thinking chain is described in detail below:

[0082] Mining observation refers to the process by which the server obtains key information through a comprehensive analysis of the current context. This includes understanding the query target, reviewing all previous thinking results, and evaluating the actions taken and the observations obtained. By combining this information, the system can identify key areas or details that require further attention, thereby generating the auxiliary operation instructions required for the next step to complete the recognition instructions.

[0083] The execution plan refers to the process by which the server formulates and implements specific actions based on the auxiliary operation instructions obtained during the mining and observation phase. The core task of this phase is to generate specific codes or instructions to implement the auxiliary operation instructions. Through these operations, the system can obtain more accurate and detailed data, providing necessary identification auxiliary information for further reasoning and judgment. For example, when identifying whether construction workers are wearing safety belts correctly, auxiliary information such as bounding box annotations, segmentation masks, and human skeletons can be added to the image to be identified.

[0084] Reasoning and judgment refers to the process in which the server conducts in-depth analysis and judgment on the image to be recognized after processing in the above execution planning stage to reach a final conclusion. In this stage, the server will determine whether it has collected enough information to answer the recognition instruction through calculation and logical analysis based on the collected detailed information. If the information is insufficient, the system will continue to iterate; if the information is sufficient, the final judgment result will be generated and a clear recognition result will be provided.

[0085] For the above mining and observation link, this embodiment provides an image recognition strategy model, which can use a pre-trained multimodal language model and can simultaneously process the input recognition instructions and the image to be recognized. Based on the image recognition strategy model, this embodiment provides the following optional implementation methods of step S2:

[0086] S2-1, obtaining context information generated by calling the image recognition strategy model.

[0087] The context information includes at least one historical auxiliary operation instruction generated by the image recognition strategy model, a historical editing operation corresponding to each historical auxiliary operation instruction, and description information of the historical image to be recognized after each historical editing operation is executed.

[0088] S2-2, calling the image recognition strategy model to process the context information and the current image to be recognized, and obtaining the auxiliary operation instructions generated by the image recognition strategy model.

[0089] Exemplarily, for the received image to be identified, it is continued to be assumed that it comes from a construction scene. The server first identifies the objects in the image to be identified through the image recognition strategy model, and represents these objects with category names, for example, identifying the names of categories such as "construction workers", "safety belts", and "safety helmets"; and generates global description information for the entire image to be identified based on these identified objects, for example, "This image shows a scene where two construction workers are working, among which construction workers A and B are both wearing safety helmets, and construction worker A is also wearing a safety belt. The construction workers are surrounded by tool boxes, ladders, and warning signs on the ground. Overall, the construction workers have taken necessary safety measures." The server further uses the image recognition strategy model to process these identified objects and description information, and generates auxiliary operation instructions required to complete the query instruction.

[0090] Assuming that the recognition instruction is "determine whether the seat belt is worn correctly", the auxiliary operation instruction that the image recognition strategy model may generate is "further check the position and status of the seat belt". In order to complete this auxiliary operation instruction, the editing operation required is "drawing the seat belt boundary box". Since the image recognition strategy model may be called multiple times to complete the recognition instruction, a certain number of historical auxiliary operation instructions, historical editing operations corresponding to the historical auxiliary operation instructions, and description information of the historical images to be recognized after the execution of the historical editing operations will be generated during the calling process.

[0091] Based on the introduction of the auxiliary operation instructions in the above embodiment, continue to refer to Figure 1 , the image recognition method provided in this embodiment also includes:

[0092] S3, performing an editing operation corresponding to the auxiliary operation instruction on the current image to be recognized to obtain an updated image to be recognized.

[0093] In this embodiment, in order to adapt to flexible and varied auxiliary operation instructions, multiple image recognition tools are provided, and in order to facilitate the image editing model to understand how to use these image recognition tools, description information of these image recognition tools is also provided. Based on these image recognition tools, this embodiment provides the following optional implementations of step S3:

[0094] S3-1, obtaining description information of multiple image recognition tools.

[0095] Exemplarily, multiple image recognition tools include detection tools, segmentation and labeling tools, depth estimation tools, visual search modules, and other image operation tools (e.g., zooming in and cropping, image overlay, etc.). The description information of each tool describes the function and calling method of the tool, so that the image editing model can generate the code to call these tools for edge editing operations on the current image to be recognized. For the above multiple images being tools, some of the tools are described in detail below:

[0096] The detection tool is mainly used to draw bounding boxes in an image to clearly identify the location of a specific object. For example, in a construction scene, the detection tool can mark the location of key objects such as construction workers, hard hats, and safety belts. The drawing of bounding boxes is not limited to a single object, but can also be applied to multiple objects in an image, helping to quickly locate and identify all important elements in the image. Therefore, the application scenarios of the detection tool are very wide, especially in situations where it is necessary to quickly identify and locate targets, which can significantly improve recognition efficiency and accuracy.

[0097] For example, the detection tool can be a Faster R-CNN model that can predict the location of objects in an image and the corresponding category labels. Its description information can include the Python function signature and code block corresponding to the tool, as well as the function signature and corresponding code examples for drawing bounding boxes. Through these description information, the image editing model can generate code that calls the Faster R-CNN model to process images containing construction personnel and construction facilities, and detect category labels such as "construction workers", "safety helmets", and "safety belts". The location information of each object can be represented by the coordinates of the bounding box so that the position of the object in the image can be accurately located in subsequent processing.

[0098] The segmentation and labeling tool is mainly used to mark and segment different parts of an object in an image, and distinguish different parts by drawing color masks and using digital marker segments. This tool is particularly suitable for situations where a detailed understanding of the internal structure of an object is required. For example, when judging the degree of wear of a seat belt, the segmentation and labeling tool can mark the different parts of the seat belt in detail. In this way, the system can analyze the status of each part more accurately, thereby providing more detailed evaluation results. In addition, the segmentation and labeling tool can also mark the relationship between different objects in the image, such as the relative position between a construction worker and a toolbox, further enhancing the system's understanding of the image content.

[0099] The depth estimation tool is mainly used to estimate the distance relationship between different objects and the background in the image. This tool is particularly important in determining the spatial position and safety status of objects. For example, in a construction scene, the depth estimation tool can estimate the distance between construction workers and the ground, ladders, or other equipment to ensure that construction workers are in a safe position. Through depth estimation, the system can not only identify the location of objects, but also further analyze the relative depth and spatial distribution of objects, thereby providing more comprehensive spatial information.

[0100] In summary, these image tools can be used to collaboratively process images to improve the recognition accuracy of the system and enhance the deep understanding of image content, making it more efficient and reliable in practical applications.

[0101] S3-2, calling the image editing model to process the description information and auxiliary operation instructions to generate image editing code.

[0102] The image editing code is used to provide editing operations corresponding to the auxiliary operation instructions. It should be noted that the image editing model is also a pre-trained multimodal language model, which can be the same model as the above-mentioned image recognition strategy model, or a different dedicated model. This embodiment does not specifically limit this.

[0103] In this regard, assuming that the auxiliary operation instruction is "further check the position and status of the seat belt", in order to complete the auxiliary operation instruction, the editing operation to be taken is "drawing the seat belt bounding box", then it is necessary to generate an image editing code that can "draw the seat belt bounding box" through the image editing model. The image editing code needs to import the necessary libraries and modules, initialize the image detection tool, and load the pre-trained model. Then, read the input image, call the detection tool method to detect the position of the seat belt, and finally draw the bounding box on the image. Therefore, step S3 also includes:

[0104] S3-3, calling the image editing code to process the current image to be recognized, and obtaining an updated image to be recognized.

[0105] As described in the above embodiment, this embodiment performs at least one reasoning and judgment on the image to be identified according to the thinking chain of "mining observation-execution plan-inference and judgment". Therefore, after completing the "execution plan" through the above image editing code, it is necessary to further perform "inference and judgment". If it is impossible to obtain a recognition result that meets the recognition accuracy from the image to be identified after being edited by the image editing code, it means that the recognition auxiliary information added to the current image to be identified is not sufficient to complete the recognition instruction, and then it is necessary to add more recognition auxiliary information according to the thinking chain of "mining observation-execution plan-inference and judgment" again to continue the reasoning and judgment.

[0106] Therefore, compared with the traditional single-modal recognition method, the recognition process is often a black box operation, and it is difficult to explain why a certain result occurs. In the recognition process of the image to be recognized, this embodiment gradually adds recognition auxiliary information as an intermediate sketch to perform explicit reasoning, making the recognition process more transparent and explainable.

[0107] Based on the above introduction to the editing operation of the image to be recognized, continue to refer to Figure 1 , the image recognition method provided in this embodiment also includes:

[0108] S4, judging whether the recognition result of the current image to be recognized satisfies the recognition accuracy.

[0109] If not, return to step S2, if yes, go to step S5. In this regard, in the related art, whether the probability of the recognition result is greater than the set probability threshold can be used as a criterion for judging whether the recognition result of the current image to be recognized can meet the recognition accuracy. For example, continue to assume that the recognition instruction is "determine whether the seat belt is worn correctly", and its probability threshold is 0.7. If the probability of wearing the seat belt correctly obtained by the image recognition model is 0.6, it is considered that the recognition accuracy is not met, otherwise it is considered that the recognition accuracy is met.

[0110] In other related technologies provided in this embodiment, multiple image recognition models are provided, and whether the recognition results of multiple images are consistent is used as a criterion for judging whether the recognition accuracy is met. In this regard, the optional implementation of step S4 includes:

[0111] S4-1, calling multiple image recognition models to process the recognition instruction and the current image to be recognized, and obtaining multiple recognition results;

[0112] S4-2, judging whether the recognition results of multiple image recognition models are consistent based on multiple recognition results.

[0113] In this embodiment, the server can obtain the divergence index of multiple image recognition models based on multiple recognition results; if the divergence index is less than the divergence threshold, the recognition results of the multiple image recognition models are consistent. The multiple recognition results include the first recognition result and / or the second recognition result, the first recognition result and the second recognition result are mutually exclusive, and the relationship between the multiple recognition results and the divergence index is:

[0114]

[0115] In the formula, represents the divergence index, Represents the number of multiple image recognition models, Represents the current image to be recognized. Indicates recognition instruction. Indicates The recognition result of the image recognition model for the current image to be recognized is the first recognition result. Indicates The recognition result of the image recognition model for the current image to be recognized is the second recognition result.

[0116] For example, assume that there are 10 image recognition models, the divergence threshold can be set to 0.5, and the recognition instruction is "determine whether the seat belt is worn correctly". If the recognition results of 5 of the 10 image recognition models are "wearing the seat belt correctly", then , the remaining 5 recognition results are "not wearing seat belts correctly" , the divergence rate is 1, which is much greater than 0.5, indicating that there are great differences among the 10 image recognition models. On the contrary, if 8 of the recognition results are "correctly wearing seat belts" and the remaining 2 are "not wearing seat belts correctly", the divergence rate is 0.4, which is less than 0.5, indicating that the recognition results among the 10 image recognition models are basically consistent.

[0117] S5, obtaining a recognition result that satisfies the recognition accuracy.

[0118] In this way, this embodiment uses a multimodal thinking chain to perform explicit reasoning during the recognition process of the image to be recognized, and gradually adds recognition auxiliary information to the image to be recognized. This process imitates the auxiliary line drawing and marking methods used by humans when solving problems, and significantly enhances the visual perception ability of the model. Therefore, when faced with complex and changing scenes, the system can still comprehensively and accurately judge the image content, thereby improving the accuracy and robustness of recognition.

[0119] In order to have a clearer understanding of the above-mentioned image recognition method provided by this embodiment, the above-mentioned scheme is fully described below with reference to examples. Figure 2 The image to be identified shown has an identification instruction of “seat belt wearing status”.

[0120] (1) The server calls the image recognition strategy model Figure 2 The image to be identified in the image is processed with the identification instruction, and the auxiliary operation instruction "detect construction personnel" is output. That is, "observation and thinking 1".

[0121] (2) The server further calls the image editing model to generate code for object detection operations and executes it, marking the bounding box of the construction worker in the image to be identified. This is “Execution Plan 1”.

[0122] (3) If it is not possible to infer a recognition result that satisfies the recognition accuracy from the image to be recognized with the bounding box added, the server continues to call the image recognition strategy model to process the image to be recognized with the bounding box added and the context information generated during the call, and outputs the auxiliary operation instruction "behavior skeleton recognition". That is, "observation and thinking 2".

[0123] (4) The server further calls the image editing model to generate code for skeleton recognition operations and executes it, marking the human skeleton of the construction worker in the image to be recognized. This is “Execution Plan 2”.

[0124] (5) If it is not possible to infer a recognition result that satisfies the recognition accuracy from the image to be recognized with the human skeleton added, the server continues to call the image recognition strategy model to process the image to be recognized with the human skeleton added and the context information generated during the call process, and outputs the auxiliary operation instruction "Detect seat belt". That is, "Observe and think 3".

[0125] (6) The server further calls the image editing model to generate code for seat belt recognition operations and executes it to segment the seat belt from the image to be recognized. This is “Execution Plan 3”.

[0126] The same process is repeated until a recognition result that satisfies the recognition accuracy can be inferred, and the recognition result is "not worn".

[0127] Based on the same inventive concept as the image recognition method provided in this embodiment, this embodiment also provides an image recognition device, which includes at least one software function module that can be stored in a memory or fixed in an electronic device in the form of software. The processor in the electronic device is used to execute the executable module stored in the memory. For example, the software function module and computer program included in the image recognition device. Please refer to Figure 3 , from the functional point of view, the image recognition device may include:

[0128] The image acquisition module 11 is used to receive the image to be identified and the identification instruction of the image to be identified;

[0129] The auxiliary editing module 12 is used to determine the auxiliary operation instructions required to complete the recognition instructions for the current image to be recognized;

[0130] The auxiliary editing module 12 is further used to perform an editing operation corresponding to the auxiliary operation instruction on the current image to be identified, so as to obtain an updated image to be identified;

[0131] The image recognition module 13 is used to determine whether the recognition result of the current image to be recognized meets the recognition accuracy;

[0132] If not, the process returns to the step of determining the auxiliary operation instructions required to complete the recognition instruction for the current image to be recognized, until a recognition result satisfying the recognition accuracy is obtained.

[0133] In this embodiment, the image acquisition module 11 is used to implement Figure 1 In step S1, the auxiliary editing module 12 is used to implement Figure 1 In steps S2 and S3, the image recognition module 13 is used to implement Figure 1 Therefore, for the detailed description of the above modules, please refer to the specific implementation of the corresponding steps. In addition, it should be noted that, in view of the same inventive concept as the image recognition method provided in the above embodiment, the image recognition device can also implement other steps or sub-steps of the method through the above modules or other possible modules, which will not be described in detail in this embodiment.

[0134] In addition, the functional modules in the various embodiments of the present application may be integrated together to form an independent part, or each module may exist separately, or two or more modules may be integrated to form an independent part.

[0135] It should also be understood that if the above implementation is implemented in the form of a software function module and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, which is stored in a storage medium and includes a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present application.

[0136] Therefore, this embodiment further provides a storage medium, which is a computer-readable storage medium. The storage medium stores a computer program, and when the computer program is executed by the processor, the image recognition method provided in this embodiment is implemented. Among them, the storage medium can be a U disk, a mobile hard disk, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, Random Access Memory), a disk or an optical disk, etc., which can store program codes.

[0137] This embodiment provides an electronic device for implementing the image recognition method. Figure 4 As shown, the electronic device may include a processor 22 and a memory 21. In addition, the memory 21 stores a computer program, and the processor implements the image recognition method provided in this embodiment by reading and executing the computer program corresponding to the above implementation in the memory 21.

[0138] Continue to see Figure 4 The electronic device further includes a communication unit 23. The memory 21, the processor 22 and the communication unit 23 are electrically connected to each other directly or indirectly through a system bus 24 to achieve data transmission or interaction.

[0139] The memory 21 may be an information recording device based on any electronic, magnetic, optical or other physical principle, used to record execution instructions, data, etc. In some embodiments, the memory 21 may be, but is not limited to, a volatile memory, a non-volatile memory, a storage drive, etc.

[0140] In some embodiments, the volatile memory may be a random access memory (RAM); in some embodiments, the non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable read-only memory (EEPROM), a flash memory, etc.; in some embodiments, the storage drive may be a disk drive, a solid-state drive, any type of storage disk (such as a CD, a DVD, etc.), or a similar storage medium, or a combination thereof, etc.

[0141] The communication unit 23 is used to send and receive data through a network. In some embodiments, the network may include a wired network, a wireless network, a fiber optic network, a telecommunication network, an intranet, the Internet, a local area network (LAN), a wide area network (WAN), a wireless local area network (WLAN), a metropolitan area network (MAN), a wide area network (WAN), a public switched telephone network (PSTN), a Bluetooth network, a ZigBee network, or a near field communication (NFC) network, or any combination thereof. In some embodiments, the network may include one or more network access points. For example, the network may include a wired or wireless network access point, such as a base station and / or a network switching node, and one or more components of the service request processing system may be connected to the network through the access point to exchange data and / or information.

[0142] The processor 22 may be an integrated circuit chip with signal processing capability, and the processor may include one or more processing cores (e.g., a single-core processor or a multi-core processor). By way of example only, the processor may include a central processing unit (CPU), an application specific integrated circuit (ASIC), an application specific instruction set processor (ASIP), a graphics processing unit (GPU), a physical processing unit (PPU), a digital signal processor (DSP), a field programmable gate array (FPGA), a programmable logic device (PLD), a controller, a microcontroller unit, a reduced instruction set computer (RISC), or a microprocessor, or any combination thereof.

[0143] Understandably, Figure 4The structure shown is for illustration only. The electronic device may also have Figure 4 More or fewer components than shown, or with Figure 4 Different configurations shown. Figure 4 The components shown may be implemented in hardware, software or a combination thereof.

[0144] It should be understood that the apparatus and method disclosed in the above-mentioned embodiments can also be implemented in other ways. The apparatus embodiments described above are merely schematic. For example, the flowcharts and block diagrams in the accompanying drawings show the possible architecture, functions and operations of the apparatus, methods and computer program products according to the multiple embodiments of the present application. In this regard, each box in the flowchart or block diagram can represent a module, a program segment or a part of a code, and the module, a program segment or a part of a code contains one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two consecutive boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or the flowchart, and the combination of boxes in the block diagram and / or the flowchart can be implemented with a dedicated hardware-based system that performs a specified function or action, or can be implemented with a combination of dedicated hardware and computer instructions.

[0145] The above are only various implementations of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. An image recognition method, characterized in that: The method comprises: Receiving an image to be identified and an identification instruction of the image to be identified; Determining the auxiliary operation instructions required to complete the recognition instruction for the current image to be recognized, comprising: obtaining context information generated by calling the image recognition strategy model, wherein the image recognition strategy model is a pre-trained multimodal language model, and the context information comprises: at least one historical auxiliary operation instruction generated by the image recognition strategy model, a historical editing operation corresponding to each of the historical auxiliary operation instructions, and description information of the historical image to be recognized after each of the historical editing operations is executed; Calling the image recognition strategy model to process the context information and the current image to be recognized, and obtaining the auxiliary operation instruction generated by the image recognition strategy model; Performing an editing operation corresponding to the auxiliary operation instruction on the current image to be identified, to obtain an updated image to be identified, wherein the editing operation adds identification auxiliary information to the current image to be identified; Determine whether the recognition result of the current image to be recognized meets the recognition accuracy; If not, the process returns to the step of determining the auxiliary operation instructions required to complete the recognition instruction for the current image to be recognized, until a recognition result that satisfies the recognition accuracy is obtained.

2. The image recognition method according to claim 1, characterized in that: The step of performing an editing operation corresponding to the auxiliary operation instruction on the current image to be identified to obtain an updated image to be identified includes: Get description information of multiple image recognition tools; Calling an image editing model to process the description information and the auxiliary operation instruction to generate an image editing code, wherein the image editing code is used to provide an editing operation corresponding to the auxiliary operation instruction; The image editing code is called to process the current image to be identified to obtain an updated image to be identified.

3. The image recognition method according to claim 1, characterized in that: The step of judging whether the recognition result of the current image to be recognized satisfies the recognition accuracy includes: Calling multiple image recognition models to process the recognition instruction and the current image to be recognized to obtain multiple recognition results; Determining whether the recognition results of the multiple image recognition models are consistent according to the multiple recognition results; If so, a recognition result satisfying the recognition accuracy is obtained.

4. The image recognition method according to claim 3, characterized in that: The determining, based on the multiple recognition results, whether the recognition results of the multiple image recognition models are consistent includes: Obtaining divergence indicators of the multiple image recognition models according to the multiple recognition results; If the divergence index is less than the divergence threshold, the recognition results of the multiple image recognition models are consistent.

5. The image recognition method according to claim 4, characterized in that: The multiple recognition results include a first recognition result and / or a second recognition result, the first recognition result and the second recognition result are mutually exclusive, and the relationship between the multiple recognition results and the divergence index is: In the formula, represents the divergence index, represents the number of the plurality of image recognition models, Represents the current image to be recognized. represents the identification instruction, Indicates The recognition result of the image recognition model for the current image to be recognized is the first recognition result. Indicates The recognition result of the image recognition model for the current image to be recognized is the second recognition result.

6. An image recognition device, characterized in that: The device comprises: An image acquisition module, used for receiving an image to be identified and an identification instruction of the image to be identified; The auxiliary editing module is used to determine the auxiliary operation instructions required to complete the recognition instruction for the current image to be recognized, including: Acquire context information generated by calling an image recognition strategy model, wherein the image recognition strategy model is a pre-trained multimodal language model, and the context information includes: at least one historical auxiliary operation instruction generated by the image recognition strategy model, a historical editing operation corresponding to each of the historical auxiliary operation instructions, and description information of a historical image to be recognized after each of the historical editing operations is executed; Calling the image recognition strategy model to process the context information and the current image to be recognized, and obtaining the auxiliary operation instruction generated by the image recognition strategy model; The auxiliary editing module is further used to perform an editing operation corresponding to the auxiliary operation instruction on the current image to be identified, so as to obtain an updated image to be identified, wherein the editing operation adds identification auxiliary information to the current image to be identified; An image recognition module is used to determine whether the recognition result of the current image to be recognized meets the recognition accuracy; If not, the process returns to the step of determining the auxiliary operation instructions required to complete the recognition instruction for the current image to be recognized, until a recognition result that satisfies the recognition accuracy is obtained.

7. A storage medium, characterized in that: The storage medium stores a computer program, and when the computer program is executed by a processor, the image recognition method according to any one of claims 1 to 5 is implemented.

8. An electronic device, characterized in that: The electronic device includes a processor and a memory, wherein the memory stores a computer program, and when the computer program is executed by the processor, the image recognition method according to any one of claims 1 to 5 is implemented.

Images