High-resolution image recognition method and device, equipment and storage medium

Abstract

The invention belongs to the technical field of artificial intelligence, and relates to a high-resolution image recognition method and device, equipment and a storage medium. Performing co-occurrence prediction and strong detection on the image through reasoning; screening out a definite object, an illusion object and a new detection object; and obtaining a final image description text according to the initial image description text, the image local area description text of the definite object, the illusion object list and the image local area description text of the new detection object. According to the method, interference of an illusion object is firstly checked on generation of a final image description text, then image local area description texts of a definite object and a new detection object are supplemented into corresponding image initial description texts, and it is guaranteed that richer and more accurate image description texts are generated; when the method is applied to traffic accident image recognition description or medical image recognition description scenes, business personnel can be assisted in obtaining more detailed and accurate image recognition description texts.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology and is applied to the scenario of recognizing high-resolution images to generate more accurate descriptive text. It relates to a high-resolution image recognition method, apparatus, device, and storage medium. Background Technology

[0002] Currently, Visual Language Models (VLMs) have made significant progress in image description generation tasks. These models are typically pre-trained on a large number of image-text pairs to learn the fine semantic relationships between visual content and natural language descriptions. In particular, VLMs are increasingly being applied to financial or healthcare image description scenarios, such as image recognition and description for vehicle accidents in financial claims or medical images.

[0003] Existing visual language models still face significant bottlenecks when processing high-resolution images. These bottlenecks are primarily manifested in the following ways: These models are pre-trained on relatively low-resolution image inputs. When a high-resolution image is input, the model must downsample to its supported low-resolution size, resulting in severe loss of visual detail and an inability to accurately identify visual features in the image. Consequently, important objects in the image may be missed, especially small or distant objects that become blurry after downsampling. Furthermore, when visual features are unclear, the model may overly rely on prior language knowledge learned from training data, identifying non-existent or "illusory" objects. Ultimately, it cannot generate detailed and accurate descriptive text for high-resolution images. Summary of the Invention

[0004] The purpose of this application is to provide a high-resolution image recognition method, apparatus, device, and storage medium to solve the technical problem that current visual language models cannot generate detailed and accurate descriptive text for high-resolution images.

[0005] In a first aspect, embodiments of this application provide a high-resolution image recognition method, which employs the following technical solution: A high-resolution image recognition method includes the following steps: Obtain the initial image description text output by the visual language model for the target high-resolution image; The initial description text of the image is input into the text language model to obtain the list of objects to be detected output by the text language model; The target high-resolution image and the list of objects to be detected are input into the target object detector to obtain the list of high-confidence detected objects and the bounding box coordinate information of the high-confidence detected objects output by the target object detector. The list of high-confidence detection objects and the list of objects to be detected are input into the differentiation labeling component. All objects are differentiated and labeled according to the preset differentiation labeling strategy to obtain the differentiation labeling result. Based on the distinguishing marker results, confirmed objects, hallucinatory objects, and newly detected objects are filtered out to construct a list of hallucinatory objects; The bounding box coordinates of the target high-resolution image and the high-confidence detection object are input into the visual language model to generate image local region description text for the confirmed object and image local region description text for the newly detected object; The initial image description text, the image local region description text of the confirmed object, the list of hallucination objects, and the image local region description text of the newly detected object are input into the text language model to obtain the final image description text output by the text language model.

[0006] Secondly, embodiments of this application also provide a high-resolution image recognition device, which adopts the technical solution described below: A high-resolution image recognition device, comprising: The image initial description text acquisition module is used to acquire the image initial description text output by the visual language model for the target high-resolution image; The module for obtaining the list of objects to be detected is used to input the initial description text of the image into the text language model and obtain the list of objects to be detected output by the text language model. The high-confidence detection information acquisition module is used to input the target high-resolution image and the list of objects to be detected into the target object detector, and to acquire the list of high-confidence detection objects and the bounding box coordinate information of the high-confidence detection objects output by the target object detector. The object differentiation and marking processing module is used to input the list of high-confidence detection objects and the list of objects to be detected into the differentiation and marking component, and perform differentiation and marking processing on all objects according to the preset differentiation and marking strategy to obtain the differentiation and marking results; The object filtering module is used to filter out confirmed objects, hallucinatory objects and newly detected objects based on the difference marking results, and construct a list of hallucinatory objects; The image local region description text generation module is used to input the bounding box coordinate information of the target high-resolution image and the high-confidence detection object into the visual language model to generate image local region description text for the confirmed object and image local region description text for the newly detected object; The final image description text acquisition module is used to input the initial image description text, the image local region description text of the confirmed object, the list of hallucination objects, and the image local region description text of the newly detected object into the text language model, and obtain the final image description text output by the text language model.

[0007] Thirdly, embodiments of this application also provide a computer device that adopts the technical solution described below: A computer device includes a memory and a processor, wherein the memory stores computer-readable instructions, and the processor executes the computer-readable instructions to implement the steps of the high-resolution image recognition method described above.

[0008] Fourthly, embodiments of this application also provide a computer-readable storage medium, which adopts the technical solutions described below: A computer-readable storage medium storing computer-readable instructions that, when executed by a processor, implement the steps of the high-resolution image recognition method described above.

[0009] Compared with the prior art, the embodiments of this application have the following main advantages: The high-resolution image recognition method described in this application obtains the initial image description text output by a visual language model for a target high-resolution image; it then uses inference co-occurrence prediction by a text language model and strong detection of the image by a target object detector; it filters out confirmed objects, hallucinatory objects, and newly detected objects; it generates image local region description texts for confirmed objects and newly detected objects; and it inputs the initial image description text, the image local region description texts for confirmed objects, the list of hallucinatory objects, and the image local region description texts for newly detected objects into a text language model to obtain the final image description text output by the text language model. This method achieves the following: in the generation of the final image description text, it first eliminates interference from hallucinatory objects, and then supplements the corresponding initial image description text with the image local region description texts of confirmed objects and newly detected objects, ensuring the generation of richer and more accurate image description text. Applying this method to scenarios such as traffic accident image recognition and description or medical image recognition and description can assist business personnel in obtaining more detailed and accurate image recognition description texts. Attached Figure Description

[0010] To more clearly illustrate the solutions in this application, the accompanying drawings used in the description of the embodiments of this application will be briefly introduced below. Obviously, the accompanying drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0011] Figure 1 This is an exemplary system architecture diagram to which this application can be applied; Figure 2 This is a flowchart of an embodiment of a high-resolution image recognition method according to this application; Figure 3 yes Figure 2 A flowchart of a specific embodiment of step 202 shown; Figure 4 yes Figure 2 A flowchart of a specific embodiment of step 203 shown; Figure 5 yes Figure 2 A flowchart of a specific embodiment of step 204 shown; Figure 6 yes Figure 2 A flowchart of a specific embodiment of step 205 shown; Figure 7 yes Figure 2 A flowchart of a specific embodiment of step 206 shown; Figure 8 yes Figure 2 A flowchart of a specific embodiment of step 207 shown; Figure 9 This is a schematic diagram of one embodiment of a high-resolution image recognition device according to this application; Figure 10 This is a schematic diagram of the structure of one embodiment of the computer device according to this application. Detailed Implementation

[0012] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application pertains; the terminology used herein in the specification of the application is for the purpose of describing particular embodiments only and is not intended to be limiting of the application; the terms "comprising" and "having," and any variations thereof, in the specification, claims, and foregoing drawings of this application, are intended to cover non-exclusive inclusion. The terms "first," "second," etc., in the specification, claims, or foregoing drawings of this application are used to distinguish different objects, not to describe a particular order.

[0013] In this document, the term "embodiment" means that a particular feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of this application. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a separate or alternative embodiment mutually exclusive with other embodiments. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described herein can be combined with other embodiments.

[0014] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings.

[0015] like Figure 1 As shown, system architecture 100 may include terminal device 101, network 102, and server 103. Terminal device 101 may be a laptop 1011, tablet 1012, or mobile phone 1013. Network 102 is used as a medium to provide a communication link between terminal device 101 and server 103. Network 102 may include various connection types, such as wired, wireless communication links, or fiber optic cables.

[0016] Users can use terminal device 101 to interact with server 103 via network 102 to receive or send messages, etc. Various communication client applications can be installed on terminal device 101, such as web browser applications, shopping applications, search applications, instant messaging tools, email clients, social media platform software, etc.

[0017] Terminal device 101 can be various electronic devices with a display screen and support web browsing. In addition to laptops 1011, tablets 1012, or mobile phones 1013, terminal device 101 can also be an e-book reader, an MP3 player (Moving Picture Experts Group Audio Layer III), an MP4 player (Moving Picture Experts Group Audio Layer IV), a laptop computer, and a desktop computer, etc.

[0018] Server 103 can be a server that provides various services, such as a backend server that provides support for the pages displayed on terminal device 101.

[0019] It should be noted that the high-resolution image recognition method provided in this application embodiment is generally executed by a server, and correspondingly, a high-resolution image recognition device is generally set in the server.

[0020] It should be understood that Figure 1 The number of terminal devices, networks, and servers shown is merely illustrative. Depending on implementation needs, any number of terminal devices, networks, and servers can be included.

[0021] Continue to refer to Figure 2A flowchart of an embodiment of a high-resolution image recognition method according to this application is shown. The high-resolution image recognition method includes the following steps: Step 201: Obtain the initial image description text output by the visual language model for the target high-resolution image.

[0022] In this embodiment, the visual language model refers to Vision-Language Models, or VLMs for short. They are typically pre-trained on a large number of image-text pairs to learn the fine semantic relationship between visual content and natural language description. This visual language model can be used to identify image content and generate descriptive text.

[0023] In this embodiment, the target high-resolution image includes high-resolution images in financial business scenarios, such as high-resolution traffic accident images provided during car insurance claims; it also includes high-resolution medical images in health and medical business scenarios, such as two-dimensional images of the human body generated by using X-ray radiation and medical detectors, such as X-ray medical images; three-dimensional images with higher resolution generated by CT scanners, such as CT medical scan images; three-dimensional images with higher contrast created by magnetic resonance imaging technology; images of the human body generated by ultrasound imaging technology; and medical images of human organs generated by nuclear medicine imaging technology. The high-resolution image refers to an image that exceeds a certain image resolution value. Here, "high" is a relative value. For example, if the preset image resolution value is 336x336 pixels, then anything exceeding this pixel value can be understood as a corresponding target high-resolution image.

[0024] In this embodiment, the initial image description text output by the visual language model for the target high-resolution image is obtained. This initial image description text for the target high-resolution image is generated using the visual language model to assist business personnel in recognizing and analyzing the target high-resolution image. Specifically, the target high-resolution image can be obtained first, then input into the visual language model. Finally, the initial image description text for the target high-resolution image output by the visual language model based on the description prompt text is obtained. The description prompt text, for example, is: "Describe in detail the recognition objects contained in this image."

[0025] Step 202: Input the initial image description text into the text language model to obtain the list of objects to be detected output by the text language model.

[0026] In this embodiment, the initial image description text is already in plain text descriptive form. The text language model includes a large language model (LLM). By inputting the initial image description text into the text language model, the text language model analyzes the plain text description text of the initial image description text, thereby obtaining the objects to be detected contained in the target high-resolution image.

[0027] Step 203: Input the target high-resolution image and the list of objects to be detected into the target object detector to obtain the list of high-confidence detected objects and the bounding box coordinate information of the high-confidence detected objects output by the target object detector.

[0028] In this embodiment, in order to improve the detection accuracy of high-resolution target images, a target object detector is introduced on the basis of the joint processing of VLM and LLM. The target object detector can be an integrated detector with high detection performance, which is generally integrated by multiple basic detectors. It generally has high detection accuracy and is stronger than the visual language model in image object recognition.

[0029] By introducing a target object detector, a high-resolution image recognition model is constructed that combines a visual language model, a text language model, and a target object detector. This significantly improves the recognition accuracy of high-resolution images and generates more detailed and accurate image description text.

[0030] Step 204: Input the list of high-confidence detection objects and the list of objects to be detected into the differentiation labeling component, and perform differentiation labeling processing on all objects according to the preset differentiation labeling strategy to obtain the differentiation labeling result.

[0031] Specifically, since the objects in the high-confidence detection object list are detected and output by the target object detector, and the objects in the object to be detected list are identified and predicted by the visual language model and the text language model, in step 204, the objects in the two lists are distinguished and marked so that the object recognition can be performed by combining the distinguishing and marking results.

[0032] Step 205: Based on the differentiation marker results, filter out the confirmed objects, hallucination objects, and newly detected objects to construct a list of hallucination objects.

[0033] In this embodiment, the confirmed object refers to an object detected by both the visual language model and the target object detector; the hallucinatory object refers to an object detected by the visual language model but not by the target object detector; and the newly detected object refers to an object not detected by the visual language model but detected by the target object detector.

[0034] Step 206: Input the bounding box coordinates of the target high-resolution image and the high-confidence detection object into the visual language model to generate image local region description text for the confirmed object and image local region description text for the newly detected object.

[0035] Specifically, for the confirmed object, since both the visual language model and the target object detector can detect it in the image, it indicates that the corresponding object should be in the high-resolution image. Therefore, in order to generate more detailed image description text, specific regions of the confirmed object are identified to generate image local region description text for the confirmed object. Similarly, for the newly detected object, it may be an object that was ignored by the visual language model during step 201. Therefore, in step 206, a corresponding remedial detection is performed to generate image local region description text for the newly detected object, so that the final image description text can be generated using the image local region description text, thereby improving the accuracy and precision of the image text description.

[0036] Step 207: Input the initial image description text, the image local region description text of the confirmed object, the list of hallucination objects, and the image local region description text of the newly detected object into the text language model to obtain the final image description text output by the text language model.

[0037] Specifically, by inputting the list of hallucination objects, along with the initial image description text, the image local region description text of the confirmed objects, and the image local region description text of the newly detected objects, into the text language model, the misidentification of the visual language model can be corrected by combining the list of hallucination objects before the final image description text is generated, thus ensuring the accuracy of the final generated image description text.

[0038] In this embodiment, the initial image description text output by the visual language model for the target high-resolution image is obtained; the image is strongly detected by the text language model through inference co-occurrence prediction and the target object detector; confirmed objects, hallucinatory objects, and newly detected objects are filtered out; image local region description texts for confirmed objects and newly detected objects are generated; the initial image description text, the image local region description texts for confirmed objects, the list of hallucinatory objects, and the image local region description texts for newly detected objects are input into the text language model to obtain the final image description text output by the text language model. This achieves the goal of first eliminating the interference of hallucinatory objects in the generation of the final image description text, and then supplementing the corresponding initial image description text with the image local region description texts of confirmed objects and newly detected objects, ensuring the generation of richer and more accurate image description text. Applying this method to scenarios such as traffic accident image recognition description or medical image recognition description can assist business personnel in obtaining more detailed and accurate image recognition description text.

[0039] Continue to refer to Figure 3 , Figure 3 yes Figure 2 A flowchart of a specific embodiment of step 202 shown includes: Step 301: Use the text language model to perform text parsing on the initial description text of the image, and extract the objects explicitly contained in the initial description text of the image; Specifically, the step of using the text language model to parse the initial image description text and extract the explicitly contained objects in the initial image description text includes: performing word segmentation on the initial image description text using the text language model; performing part-of-speech analysis on the word segmentation results to identify the explicitly contained noun fields in the initial image description text; and extracting the explicitly contained objects in the initial image description text based on the explicitly contained noun fields.

[0040] Step 302: Generate a list of key objects according to the order in which the objects explicitly contained in the initial image description text are presented in the initial image description text. Step 303: Use the text language model to perform semantic parsing on the initial description text of the image to obtain the semantic parsing result; Step 304: Use the objects explicitly included in the key object list as context information, and combine the semantic parsing results to predict the implicit description objects and / or undescribed objects in the target high-resolution image that have a possibility of co-occurring with the objects explicitly included in the key object list; Specifically, for example, when using a visual language model to generate image and text descriptions, assuming that the list of key objects explicitly includes pedestrians and cars, then in step 304, the implicit description objects and / or undescribed objects that have a possibility of co-occurring with pedestrians and cars are predicted using the prior knowledge in the text language model, such as traffic signs and zebra crossings.

[0041] In this embodiment, an inference co-occurrence mechanism is introduced into the text language model, which enables the preliminary enrichment of the objects that may appear in the target high-resolution image by utilizing the inference co-occurrence mechanism.

[0042] Step 305: Organize the implicitly described objects and / or the undescribed objects into a list to obtain an extended object list; Step 306: Merge the key object list and the extended object list to generate the object list to be detected.

[0043] Specifically, by merging the key object list and the extended object list, the list of objects to be detected is generated, so that when the target object detector performs object detection on the target high-resolution image, it not only detects the extended objects predicted by the inference co-occurrence mechanism, but also performs a re-examination and verification of the objects initially identified by the visual language model.

[0044] Continue to refer to Figure 4 , Figure 4 yes Figure 2 A flowchart of a specific embodiment of step 203 shown includes: Step 401: Parse the list of objects to be detected to obtain all objects to be detected; In this embodiment, the objects to be detected include both objects explicitly contained in the initial image description text and implicitly described objects or / and undescribed objects that have a possibility of co-occurring with objects explicitly contained in the key object list.

[0045] Step 402: Sequentially select different objects to be detected as the current detection objects, and use the target object detector to detect whether the current detection object exists in the target high-resolution image; Step 403: If the current detected object is located and identified in the target high-resolution image by the target object detector with high confidence, then extract the current detected object and the bounding box coordinate information of the current detected object in the target high-resolution image, wherein the high confidence refers to exceeding the preset confidence threshold in the target object detector; Specifically, if the current detected object is located and identified in the target high-resolution image with high confidence by the target object detector, it means that the current detected object is highly likely to be in the target high-resolution image. In order to enrich the text description of the local area of ​​the image, the target object detector is used to extract the current detected object and the bounding box coordinate information of the current detected object in the target high-resolution image.

[0046] Step 404: Count all the objects located and identified by the target object detector in the target high-resolution image with high confidence, and construct a high-confidence detection object list.

[0047] Specifically, using statistical methods, all detected objects located and identified with high confidence by the target object detector in the target high-resolution image are integrated together to construct a high-confidence detection object list, so as to facilitate subsequent supplementation and generation of image text descriptions using objects in the high-confidence detection object list.

[0048] Continue to refer to Figure 5 , Figure 5 yes Figure 2 A flowchart of a specific embodiment of step 204 shown includes: Step 501: Parse the list of high-confidence detection objects and identify all high-confidence detection objects; Step 502: parse the list of objects to be detected, and identify objects explicitly contained in the initial description text of the image, implicit description objects that have a possibility of co-occurring with objects explicitly contained in the list of key objects, and / or undescribed objects; Step 503: Compare the objects in the high-confidence detection object list with the objects in the object to be detected list; Step 504: Identify objects that exist only in the high-confidence detection object list and not in the object to be detected list, and mark such objects with a first distinguishing identifier; In this embodiment, the objects that exist only in the high-confidence detection object list and not in the object to be detected list are objects that the target object detector can detect but are not given under both visual language model recognition and text language model inference, i.e., new detection objects of the first category.

[0049] Step 505: Identify objects that exist only in the list of objects to be detected but not in the list of high-confidence detection objects, and mark such objects with a second distinguishing identifier; In this embodiment, the objects that exist only in the list of objects to be detected but not in the list of high-confidence detection objects are those that the target object detector cannot detect. These objects, which exist under visual language model recognition and text language model inference, are hallucinatory objects.

[0050] Step 506: Identify objects that exist in both the list of objects to be detected and the list of high-confidence detection objects, and use them as common objects. Continue to identify each common object. Specifically, the objects that exist in both the list of objects to be detected and the list of high-confidence objects to be detected are those that the target object detector can detect. These objects exist under the visual language model recognition and text language model inference. Specifically, the objects recognized by the visual language model or the objects inferred by the text language model need to be further identified.

[0051] Step 507: If the current common object is identified as an object explicitly contained in the initial description text of the image, the current common object is marked using a third distinguishing identifier; In this embodiment, if the current common object belongs to the object explicitly contained in the initial description text of the image, it means that the object can be detected by both the target object detector and the visual language model, and its existence can be confirmed, i.e., the object is confirmed.

[0052] Step 508: If it is identified that the current common object belongs to an implicitly described object or / and an undescribed object that has a common occurrence possibility with the objects explicitly included in the list of key objects, the current common object is marked with a fourth distinguishing identifier.

[0053] In this embodiment, if the current common object belongs to an implicitly described object or / and an undescribed object that has a common occurrence possibility with the objects explicitly included in the list of key objects, it means that the object has been detected by the target object detector, not recognized by the visual language model, but inferred by the text language model, and belongs to the second type of new detection object.

[0054] In this embodiment, the visual language model can be understood as a relatively vague object detection component, which is easily affected by interference and has low detection accuracy. The text language model can be understood as the object obtained by co-occurrence inference prediction based on the visual language model recognition. The target object detector can be understood as a pre-trained strong object detection component, which assumes the object exists in the image when its detection result is high confidence. Through the initial recognition of the visual language model, the co-occurrence prediction of the text language model, and the strong detection of the target object detector, the joint processing mode ensures that subsequent targeted supplementary descriptions of the initial description text of the target high-resolution image are generated, resulting in richer and more accurate image description text.

[0055] Continue to refer to Figure 6 , Figure 6 yes Figure 2 A flowchart of a specific embodiment of step 205 shown includes: Step 601: By identifying the differentiation marker results, select the objects marked by the first differentiation identifier and the fourth differentiation identifier as new detection objects; Specifically, in steps 504 and 508, the new detection objects are marked by the first distinguishing identifier and the fourth distinguishing identifier, respectively. Therefore, when identifying the distinguishing identifier results, it is necessary to filter out the objects marked by the first distinguishing identifier and the fourth distinguishing identifier as the new detection objects.

[0056] Step 602: By identifying the differentiation mark results, filter out the objects marked by the second differentiation mark as hallucination objects, and construct a list of hallucination objects; Specifically, since the second distinguishing identifier marks objects that the target object detector cannot detect but exist under visual language model recognition and text language model inference, such objects are more likely to be hallucinatory objects.

[0057] Step 603: By identifying the differentiation mark results, the objects marked by the third differentiation mark are selected as confirmed objects.

[0058] Specifically, the object marked by the third distinguishing identifier can be detected by both the target object detector and the visual language model, and its existence can be confirmed. Therefore, the object marked by the third distinguishing identifier is regarded as the confirmed object.

[0059] Continue to refer to Figure 7 , Figure 7 yes Figure 2 A flowchart of a specific embodiment of step 206 shown includes: Step 701: Using the bounding box coordinates of the high-confidence detection object as a cropping parameter, the cropping component in the visual language model is used to crop the target high-resolution image to obtain the close-up image region corresponding to the high-confidence detection object. Specifically, since the bounding box coordinates of all high-confidence detected objects have been identified in step 203 using the target object detector, in step 701, the bounding box coordinates of the high-confidence detected objects can be used as cropping parameters, and the cropping component in the visual language model can be used to crop the target high-resolution image to obtain the close-up image region corresponding to the high-confidence detected object.

[0060] Step 702: Input the close-up image region into the visual language model to generate image local region description text for the high-confidence detection object; Specifically, by inputting the close-up image region into the visual language model, image local region description text for the high-confidence detection object is generated, thereby enabling the extraction of more detailed and richer local image information using the visual language model.

[0061] Step 703: Identify the distinguishing identifiers of the high-confidence detection objects; In this embodiment, since the high-confidence detection objects are further classified into confirmed objects and newly detected objects, a distinguishing identification method is used here to further identify the image local area description text of confirmed objects and the image local area description text of newly detected objects. This facilitates the subsequent fusion of all image local area description text classifications into the initial image description text.

[0062] Step 704: If the distinguishing identifier of the high-confidence detection object is a third distinguishing identifier, then the image local region description text of the high-confidence detection object is the image local region description text of the confirmed object; Step 705: If the distinguishing identifier of the high-confidence detection object is a first distinguishing identifier or a fourth distinguishing identifier, then the image local region description text of the high-confidence detection object is the image local region description text of the new detection object.

[0063] Continue to refer to Figure 8 , Figure 8 yes Figure 2 A flowchart of a specific embodiment of step 207 shown includes: Step 801: Parse the list of hallucination objects and identify all hallucination objects; Step 802: Delete the description text corresponding to all hallucination objects from the initial image description text, and obtain the latest initial image description text; Specifically, all hallucination objects are first identified, and the descriptive text of the hallucination objects is deleted from the initial image description text to avoid interference when describing local areas of the image later.

[0064] Step 803: Parse the image local region description text of the confirmed object to obtain the bounding box coordinate information of the confirmed object in the target high-resolution image; Step 804: Parse the image local region description text of the newly detected object to obtain the bounding box coordinate information of the newly detected object in the target high-resolution image; Specifically, since the generation of the image local region description text of the confirmed object and the image local region description text of the newly detected object are combined with the bounding box coordinate information detected by the target object detector in step 204, steps 803 and 804 can obtain the bounding box coordinate information of the confirmed object in the target high-resolution image and the bounding box coordinate information of the confirmed object in the target high-resolution image through reverse parsing.

[0065] Step 805: Based on the bounding box coordinates of all confirmed objects in the target high-resolution image and the bounding box coordinates of all newly detected objects in the target high-resolution image, construct the spatial relative relationships between all objects; Specifically, when obtaining the image local region description text of the confirmed object and the image local region description text of the newly detected object, not only is the image local region description text information added to the latest initial image description text, but also the bounding box coordinate information of the confirmed object in the target high-resolution image and the bounding box coordinate information of the newly detected object in the target high-resolution image are used to fully integrate the spatial relative relationship into the latest initial image description text, ensuring the accuracy and high quality of the final image description text.

[0066] Step 806: The image local region description text of the confirmed object and the image local region description text of the newly detected object are fused into the latest image initial description text according to their spatial relative relationship to generate the image final description text.

[0067] In this embodiment, through the preliminary recognition of the visual language model, the inference co-occurrence prediction of the text language model, and the strong detection of the target object detector, the confirmed object, the hallucinatory object, and the newly detected object are screened out in the joint processing mode. This achieves the following in the generation of the final image description text: first, the interference of the hallucinatory object is eliminated, and then the image local region description text of the confirmed object and the newly detected object is added to the corresponding initial image description text, ensuring that a richer and more accurate image description text is generated.

[0068] In this embodiment, the initial image description text output by the visual language model for the target high-resolution image is obtained; the image is strongly detected by the text language model through inference co-occurrence prediction and the target object detector; confirmed objects, hallucinatory objects, and newly detected objects are filtered out; image local region description texts for confirmed objects and newly detected objects are generated; the initial image description text, the image local region description texts for confirmed objects, the list of hallucinatory objects, and the image local region description texts for newly detected objects are input into the text language model to obtain the final image description text output by the text language model. This achieves the goal of first eliminating the interference of hallucinatory objects in the generation of the final image description text, and then supplementing the corresponding initial image description text with the image local region description texts of confirmed objects and newly detected objects, ensuring the generation of richer and more accurate image description text. Applying this method to scenarios such as traffic accident image recognition description or medical image recognition description can assist business personnel in obtaining more detailed and accurate image recognition description text.

[0069] The embodiments of this application can acquire and process relevant data based on artificial intelligence technology. Artificial intelligence (AI) refers to the theories, methods, technologies, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to obtain optimal results.

[0070] Foundational technologies for artificial intelligence generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing, operating / interactive systems, and mechatronics. AI software technologies mainly encompass computer vision, robotics, biometrics, speech processing, natural language processing, and machine learning / deep learning.

[0071] In this embodiment, the initial image description text output by the visual language model for the target high-resolution image is obtained; the image is strongly detected by the text language model through inference co-occurrence prediction and the target object detector; confirmed objects, hallucinatory objects, and newly detected objects are filtered out; image local region description texts for confirmed objects and newly detected objects are generated; the initial image description text, the image local region description texts for confirmed objects, the list of hallucinatory objects, and the image local region description texts for newly detected objects are input into the text language model to obtain the final image description text output by the text language model. This achieves the goal of first eliminating the interference of hallucinatory objects in the generation of the final image description text, and then supplementing the corresponding initial image description text with the image local region description texts of confirmed objects and newly detected objects, ensuring the generation of richer and more accurate image description text. Applying this method to scenarios such as traffic accident image recognition description or medical image recognition description can assist business personnel in obtaining more detailed and accurate image recognition description text.

[0072] Further reference Figure 9 As a response to the above Figure 2 The implementation of the method shown in this application provides an embodiment of a high-resolution image recognition device, which is similar to... Figure 2 Corresponding to the method embodiments shown, this device can be specifically applied to various electronic devices.

[0073] like Figure 9 As shown, the high-resolution image recognition device 900 described in this embodiment includes: an initial image description text acquisition module 901, a list of objects to be detected acquisition module 902, a high-confidence detection information acquisition module 903, an object differentiation marker processing module 904, a distinguishable object filtering module 905, an image local region description text generation module 906, and an image final description text acquisition module 907. Wherein: The image initial description text acquisition module 901 is used to acquire the image initial description text output by the visual language model for the target high-resolution image; The object to be detected list acquisition module 902 is used to input the initial description text of the image into the text language model and obtain the object to be detected list output by the text language model; The high-confidence detection information acquisition module 903 is used to input the target high-resolution image and the list of objects to be detected into the target object detector, and to acquire the list of high-confidence detection objects and the bounding box coordinate information of the high-confidence detection objects output by the target object detector. The object differentiation and marking processing module 904 is used to input the list of high-confidence detection objects and the list of objects to be detected into the differentiation and marking component, and to perform differentiation and marking processing on all objects according to a preset differentiation and marking strategy to obtain differentiation and marking results; The object filtering module 905 is used to filter out confirmed objects, hallucinatory objects and newly detected objects based on the difference marking results, and construct a list of hallucinatory objects; The image local region description text generation module 906 is used to input the bounding box coordinate information of the target high-resolution image and the high-confidence detection object into the visual language model to generate image local region description text for the confirmed object and image local region description text for the newly detected object; The image final description text acquisition module 907 is used to input the image initial description text, the image local region description text of the confirmed object, the list of hallucination objects, and the image local region description text of the newly detected object into the text language model, and obtain the image final description text output by the text language model.

[0074] This application obtains the initial image description text output by a visual language model for a target high-resolution image; it then uses a text language model for inference co-occurrence prediction and a target object detector for strong image detection; it filters out confirmed objects, hallucinatory objects, and newly detected objects; it generates image local region description texts for confirmed objects and newly detected objects; and it inputs the initial image description text, the image local region description texts for confirmed objects, the list of hallucinatory objects, and the image local region description texts for newly detected objects into the text language model to obtain the final image description text output by the text language model. This achieves the goal of first eliminating the interference of hallucinatory objects in the generation of the final image description text, and then supplementing the corresponding initial image description text with the image local region description texts of confirmed objects and newly detected objects, ensuring the generation of richer and more accurate image description text. Applying this method to scenarios such as traffic accident image recognition and description or medical image recognition and description can assist business personnel in obtaining more detailed and accurate image recognition description texts.

[0075] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by instructing related hardware through computer-readable instructions. These computer-readable instructions can be stored in a computer-readable storage medium. When the program is executed, it can include the processes of the embodiments of the methods described above. The aforementioned storage medium can be a non-volatile storage medium such as a magnetic disk, optical disk, or read-only memory (ROM), or random access memory (RAM).

[0076] It should be understood that although the steps in the flowcharts of the accompanying figures are shown sequentially as indicated by the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the accompanying figures may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but can be executed at different times, and their execution order is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the sub-steps or stages of other steps.

[0077] To address the aforementioned technical problems, embodiments of this application also provide a computer device. Please refer to [link / reference needed]. Figure 10 , Figure 10 This is a basic structural block diagram of the computer device in this embodiment.

[0078] The computer device 10 includes a memory 10a, a processor 10b, and a network interface 10c, which are interconnected via a system bus. It should be noted that... Figure 10 Only a computer device 10 with component memory 10a, processor 10b, and network interface 10c is shown. However, it should be understood that it is not required to implement all the components shown, and more or fewer components may be implemented instead. Those skilled in the art will understand that the computer device described herein is a device capable of automatically performing numerical calculations and / or information processing according to pre-set or stored instructions. Its hardware includes, but is not limited to, microprocessors, application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), digital signal processors (DSPs), embedded devices, etc.

[0079] The computer device can be a desktop computer, laptop, handheld computer, or cloud server, etc. The computer device can interact with the user via a keyboard, mouse, remote control, touchpad, or voice control.

[0080] The memory 10a includes at least one type of readable storage medium, including flash memory, hard disk, multimedia card, card-type memory (e.g., SD or DX memory), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, magnetic disk, optical disk, etc. In some embodiments, the memory 10a may be an internal storage unit of the computer device 10, such as the hard disk or memory of the computer device 10. In other embodiments, the memory 10a may also be an external storage device of the computer device 10, such as a plug-in hard disk, smart media card (SMC), secure digital (SD) card, flash card, etc. Of course, the memory 10a may include both internal storage units and external storage devices of the computer device 10. In this embodiment, the memory 10a is typically used to store the operating system and various application software installed on the computer device 10, such as computer-readable instructions for a high-resolution image recognition method. In addition, the memory 10a can also be used to temporarily store various types of data that have been output or will be output.

[0081] In some embodiments, the processor 10b may be a central processing unit (CPU), controller, microcontroller, microprocessor, or other data processing chip. The processor 10b is typically used to control the overall operation of the computer device 10. In this embodiment, the processor 10b is used to execute computer-readable instructions stored in the memory 10a or to process data, for example, to execute computer-readable instructions for the high-resolution image recognition method described above.

[0082] The network interface 10c may include a wireless network interface or a wired network interface, which is typically used to establish communication connections between the computer device 10 and other electronic devices.

[0083] The computer device proposed in this embodiment belongs to the field of artificial intelligence technology and is applied to scenarios involving the recognition and generation of more accurate descriptive text for high-resolution images. This application obtains the initial image description text output by a visual language model for a target high-resolution image; it then uses inference co-occurrence prediction by a text language model and strong detection of the image by a target object detector; it filters out confirmed objects, hallucinatory objects, and newly detected objects; it generates image local region description texts for confirmed objects and newly detected objects; and it inputs the initial image description text, the image local region description texts for confirmed objects, the list of hallucinatory objects, and the image local region description texts for newly detected objects into a text language model to obtain the final image description text output by the text language model. This achieves the goal of first eliminating the interference of hallucinatory objects in the generation of the final image description text, and then supplementing the corresponding initial image description text with the image local region description texts of confirmed objects and newly detected objects, ensuring the generation of richer and more accurate image description text. Applying this method to scenarios involving traffic accident image recognition and description or medical image recognition and description can assist professionals in obtaining more detailed and accurate image recognition description text.

[0084] This application also provides another embodiment, namely, a computer-readable storage medium storing computer-readable instructions that can be executed by a processor to cause the processor to perform the steps of the high-resolution image recognition method described above.

[0085] The computer-readable storage medium proposed in this embodiment belongs to the field of artificial intelligence technology and is applied to scenarios involving the recognition and generation of more accurate descriptive text for high-resolution images. This application obtains the initial image description text output by a visual language model for a target high-resolution image; it uses inference co-occurrence prediction by a text language model and strong detection of the image by a target object detector; it filters out confirmed objects, hallucinatory objects, and newly detected objects; it generates image local region description texts for confirmed objects and newly detected objects; and it inputs the initial image description text, the image local region description texts for confirmed objects, the list of hallucinatory objects, and the image local region description texts for newly detected objects into a text language model to obtain the final image description text output by the text language model. This achieves the goal of first eliminating the interference of hallucinatory objects in the generation of the final image description text, and then supplementing the corresponding initial image description text with the image local region description texts of confirmed objects and newly detected objects, ensuring the generation of richer and more accurate image description text. Applying this method to scenarios involving traffic accident image recognition and description or medical image recognition and description can assist business personnel in obtaining more detailed and accurate image recognition description text.

[0086] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal device (which may be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0087] Obviously, the embodiments described above are only some embodiments of this application, not all embodiments. The accompanying drawings show preferred embodiments of this application, but do not limit the patent scope of this application. This application can be implemented in many different forms; rather, the purpose of providing these embodiments is to make the disclosure of this application more thorough and comprehensive. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions described in the foregoing specific embodiments, or make equivalent substitutions for some of the technical features. Any equivalent structures made using the content of this application's specification and drawings, directly or indirectly applied to other related technical fields, are similarly within the scope of patent protection of this application. Software tools or components not belonging to this company that appear in the embodiments of this application are merely illustrative examples and do not represent actual use.

Claims

1. A high-resolution image recognition method, characterized in that, Includes the following steps: Obtain the initial image description text output by the visual language model for the target high-resolution image; The initial description text of the image is input into the text language model to obtain the list of objects to be detected output by the text language model; The target high-resolution image and the list of objects to be detected are input into the target object detector to obtain the list of high-confidence detected objects and the bounding box coordinate information of the high-confidence detected objects output by the target object detector. The list of high-confidence detection objects and the list of objects to be detected are input into the differentiation labeling component. All objects are differentiated and labeled according to the preset differentiation labeling strategy to obtain the differentiation labeling result. Based on the distinguishing marker results, confirmed objects, hallucinatory objects, and newly detected objects are filtered out to construct a list of hallucinatory objects; The bounding box coordinates of the target high-resolution image and the high-confidence detection object are input into the visual language model to generate image local region description text for the confirmed object and image local region description text for the newly detected object; The initial image description text, the image local region description text of the confirmed object, the list of hallucination objects, and the image local region description text of the newly detected object are input into the text language model to obtain the final image description text output by the text language model.

2. The high-resolution image recognition method according to claim 1, characterized in that, The step of obtaining the list of objects to be detected output by the text language model specifically includes: The text language model is used to parse the initial description text of the image and extract the objects explicitly contained in the initial description text of the image. A list of key objects is generated according to the order in which the objects explicitly contained in the initial image description text are presented in the initial image description text. The text language model is used to perform semantic parsing on the initial description text of the image to obtain the semantic parsing result; Using the objects explicitly included in the key object list as context information, and combining the semantic parsing results, predict the implicit description objects and / or undescribed objects in the target high-resolution image that have a possibility of co-occurring with the objects explicitly included in the key object list; The implicitly described objects and / or the undescribed objects are organized into a list to obtain an extended object list; The key object list and the extended object list are merged to generate the object list to be detected.

3. The high-resolution image recognition method according to claim 2, characterized in that, The step of using the text language model to parse the initial image description text and extracting the objects explicitly contained in the initial image description text specifically includes: The initial description text of the image is segmented using the text language model described above; Part-of-speech analysis is performed on the word segmentation results to identify noun fields explicitly contained in the initial description text of the image; Based on the noun fields explicitly contained in the initial image description text, extract the objects explicitly contained in the initial image description text.

4. The high-resolution image recognition method according to claim 2, characterized in that, The step of obtaining the list of high-confidence detected objects and the bounding box coordinate information of the high-confidence detected objects output by the target object detector specifically includes: Parse the list of objects to be detected to obtain all objects to be detected; Different objects to be detected are sequentially taken as the current detection objects, and the target object detector is used to detect whether the current detection objects exist in the target high-resolution image; If the current detected object is located and identified in the target high-resolution image by the target object detector with high confidence, then the current detected object and the bounding box coordinate information of the current detected object in the target high-resolution image are extracted, wherein the high confidence refers to exceeding the preset confidence threshold in the target object detector; All objects located and identified with high confidence by the target object detector in the target high-resolution image are counted, and a high-confidence detection object list is constructed.

5. The high-resolution image recognition method according to claim 4, characterized in that, The step of inputting the high-confidence detection object list and the object to be detected list into the differentiation labeling component, and performing differentiation labeling processing on all objects according to a preset differentiation labeling strategy to obtain the differentiation labeling result specifically includes: The list of high-confidence detection objects is parsed to identify all high-confidence detection objects; Parse the list of objects to be detected and identify objects explicitly contained in the initial description text of the image, implicit description objects that have a possibility of co-occurring with objects explicitly contained in the list of key objects, and / or undescribed objects. Compare the objects in the high-confidence detection object list with the objects in the object to be detected list; Identify objects that exist only in the high-confidence detection object list and not in the object to be detected list, and mark such objects with a first distinguishing identifier; Identify objects that exist only in the list of objects to be detected but not in the list of high-confidence objects to be detected, and mark such objects with a second distinguishing identifier. Objects that exist in both the list of objects to be detected and the list of high-confidence objects to be detected are identified as common objects, and each common object is further identified. If the current common object is identified as an object explicitly contained in the initial description text of the image, the current common object is marked with a third distinguishing identifier; If the current common object is identified as an implicitly described object or / and an undescribed object that has a possibility of co-occurring with objects explicitly included in the list of key objects, the current common object is marked using a fourth distinguishing identifier.

6. The high-resolution image recognition method according to claim 5, characterized in that, The step of inputting the bounding box coordinates of the target high-resolution image and the high-confidence detection object into the visual language model to generate image local region description text for the confident object and image local region description text for the newly detected object specifically includes: Using the bounding box coordinates of the high-confidence detection object as a cropping parameter, the cropping component in the visual language model is used to crop the target high-resolution image to obtain the close-up image region corresponding to the high-confidence detection object. The close-up image region is input into the visual language model to generate image local region description text for the high-confidence detection object; Identify the distinguishing identifiers of the high-confidence detection objects; If the distinguishing identifier of the high-confidence detection object is a third distinguishing identifier, then the image local region description text of the high-confidence detection object is the image local region description text of the confirmed object; If the distinguishing identifier of the high-confidence detection object is a first distinguishing identifier or a fourth distinguishing identifier, then the image local region description text of the high-confidence detection object is the image local region description text of the new detection object.

7. The high-resolution image recognition method according to claim 1, characterized in that, The step of inputting the initial image description text, the image local region description text of the confirmed object, the list of hallucination objects, and the image local region description text of the newly detected object into the text language model, and obtaining the final image description text output by the text language model, specifically includes: The list of hallucination objects is parsed to identify all hallucination objects; Delete the description text corresponding to all hallucination objects from the initial image description text, and obtain the latest initial image description text; Parse the image local region description text of the confirmed object to obtain the bounding box coordinate information of the confirmed object in the target high-resolution image; The image local region description text of the newly detected object is parsed to obtain the bounding box coordinate information of the newly detected object in the target high-resolution image; Based on the bounding box coordinates of all confirmed objects in the target high-resolution image and the bounding box coordinates of all newly detected objects in the target high-resolution image, the spatial relative relationships between all objects are constructed. The image local region description text of the confirmed object and the image local region description text of the newly detected object are fused into the latest image initial description text according to the spatial relative relationship to generate the image final description text.

8. A high-resolution image recognition device, characterized in that, include: The image initial description text acquisition module is used to acquire the image initial description text output by the visual language model for the target high-resolution image; The module for obtaining the list of objects to be detected is used to input the initial description text of the image into the text language model and obtain the list of objects to be detected output by the text language model. The high-confidence detection information acquisition module is used to input the target high-resolution image and the list of objects to be detected into the target object detector, and to acquire the list of high-confidence detection objects and the bounding box coordinate information of the high-confidence detection objects output by the target object detector. The object differentiation and marking processing module is used to input the list of high-confidence detection objects and the list of objects to be detected into the differentiation and marking component, and perform differentiation and marking processing on all objects according to the preset differentiation and marking strategy to obtain the differentiation and marking results; The object filtering module is used to filter out confirmed objects, hallucinatory objects and newly detected objects based on the difference marking results, and construct a list of hallucinatory objects; The image local region description text generation module is used to input the bounding box coordinate information of the target high-resolution image and the high-confidence detection object into the visual language model to generate image local region description text for the confirmed object and image local region description text for the newly detected object; The final image description text acquisition module is used to input the initial image description text, the image local region description text of the confirmed object, the list of hallucination objects, and the image local region description text of the newly detected object into the text language model, and obtain the final image description text output by the text language model.

9. A computer device, characterized in that, The method includes a memory and a processor, wherein the memory stores computer-readable instructions, and the processor executes the computer-readable instructions to implement the steps of the high-resolution image recognition method as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-readable instructions that, when executed by a processor, implement the steps of the high-resolution image recognition method as described in any one of claims 1 to 7.

Images