General instrument image processing method and device and electronic equipment

Abstract

The invention provides a general instrument image processing method, a general instrument image processing device and electronic equipment, which are suitable for identification and reading tasks of various types of instruments. The method comprises the following steps: acquiring a target instrument image and a target problem text; inputting the target instrument image and the target question text into the trained visual language model to match a plurality of candidate skill functions in a predefined skill function set through the visual language model; constructing a skill function operation chain corresponding to the plurality of candidate skill functions through a visual language model based on the input and output dependency relationship of each candidate skill function; executing the skill functions in the skill function operation chain according to the sequence of the skill function operation chain to obtain a final answer corresponding to the target question text.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer vision and multi-modal artificial intelligence, in particular to a general instrument image processing method and device and electronic equipment. BACKGROUND

[0002] The existing instrument recognition system usually adopts a fixed algorithm process to complete the recognition and reading tasks, mainly relying on preset image processing steps to complete the tasks, such as sequentially extracting the instrument region through a target detection algorithm, then performing key point detection, optical character recognition (OCR) and other sub-tasks, to complete the analysis of instrument reading. Such processing flow is often fixed in the system through hard coding, which lacks flexibility and is difficult to dynamically adjust the processing logic according to the task content or user questions. SUMMARY

[0003] The purpose of the embodiments of the present application is to provide a general instrument image processing method, device and electronic equipment, to improve the problem of low flexibility of the instrument recognition system in processing dynamic tasks.

[0004] The embodiments of the present application provide a general instrument image processing method, comprising: obtaining a target instrument image and a target question text; inputting the target instrument image and the target question text into a trained visual language model, to match a plurality of candidate skill function sets from the visual language model in a pre-defined skill function set; constructing a skill function operation chain corresponding to each candidate skill function set based on the input-output dependency relationship of each candidate skill function set through the visual language model; executing the skill functions in the skill function operation chain in the order of the skill function operation chain, to obtain the final answer corresponding to the target question text. In the implementation process of the above scheme, by inputting the target instrument image and the target question text into the trained visual language model, the visual language model jointly understands the cross-modal context of the instrument image and the natural language task, matches the candidate skill function set from the pre-defined skill function set, and generates a multi-step skill function operation chain according to the automatic derivation of the input-output dependency relationship of the skill function. This combination of the context cross-modal understanding ability of the visual language model and the dynamic construction ability of the skill function operation chain enables the system to dynamically combine and execute the multi-step skill function operation chain according to the core demand of the target question text task, so that the system has strong versatility and scalability, can adapt to variable task demands, effectively overcomes the adaptability limitation of traditional fixed process in complex scenarios, and improves the flexibility and versatility in processing dynamic tasks.

[0005] Optionally, in this embodiment, executing the skill functions in the skill function operation chain according to the order of the skill function operation chain includes: obtaining the currently executing skill function in the skill function operation chain according to the order of the skill function operation chain, and determining whether the current skill function has a corresponding utility function. If the current skill function has a corresponding utility function, the final execution result is determined based on the execution result of the current skill function. In the implementation of the above scheme, by strictly following the order of the skill function operation chain, it can be ensured that the execution conditions and dependencies of each step are clear. This linear execution mode avoids the uncontrollable state that may be caused by concurrent or asynchronous execution, making the execution path and result of the entire process highly predictable, thereby allowing for precise prediction of the output of each step.

[0006] Optionally, in this embodiment, determining the final execution result based on the execution result of the current skill function includes: determining whether the execution result of the current skill function meets the preset calling conditions; if the execution result of the current skill function meets the preset calling conditions, then calling the utility function corresponding to the current skill function, so that the utility function corresponding to the current skill function and the skill function operation chain can work together to obtain the final execution result. In the implementation of the above scheme, by using the preset calling conditions as the interface rules between the skill function and the utility function, the utility function is allowed to intervene in the processing flow under specific conditions. This design makes the utility function an optional "enhancement module" rather than a mandatory dependency, so that developers can flexibly control the scope of participation of the utility function by adjusting the calling conditions, thereby supporting the functional adaptation needs in multiple scenarios; furthermore, since the timing of the intervention of the utility function is dynamically determined by the execution result, rather than a fixed choice in a fixed process, this collaborative processing mode allows the skill function and the utility function to adjust the collaboration logic according to the real-time data status, thereby further improving the functional adaptability in multiple scenarios.

[0007] Optionally, in this embodiment, calling the tool function corresponding to the current skill function includes: parsing the function operation name and operation parameters of the current skill function from the skill function operation chain generated by the visual language model; and transmitting the function operation name and operation parameters of the current skill function to the interface of the tool function corresponding to the current skill function to call and execute the tool function corresponding to the current skill function. In the implementation of the above scheme, by parsing the function operation name and parameters in the skill function operation chain generated by the visual language model (VLM), natural language instructions are directly converted into executable tool function calls. This mechanism overcomes the limitation of the traditional system where the "skill function operation chain" relies on predefined static processes. This approach, by transforming abstract semantics into operable parameters, achieves a closed loop from cross-modal reasoning to precise execution, significantly improving the flexibility and accuracy of task processing.

[0008] Optionally, in this embodiment, after determining whether the execution result of the current skill function meets the preset calling conditions, the method further includes: if the execution result of the current skill function does not meet the preset calling conditions, then the execution result of the current skill function is determined as the final execution result. In the implementation of the above scheme, by directly using the result as the final output when the execution result of the current skill function does not meet the calling conditions, without calling utility functions or performing additional processing, this mechanism avoids resource waste (such as memory allocation, computational overhead, etc.) caused by redundant operations triggered due to unmet conditions, thereby reducing system load and improving overall execution efficiency.

[0009] Optionally, in this embodiment, the predefined skill function set includes: semantic localization skill function, key point localization skill function, optical character recognition skill function, and reading recognition skill function; executing the skill functions in the skill function operation chain in the order of the skill function operation chain includes: when executing the semantic localization skill function in the order of the skill function operation chain, locating the instrument position area from the target instrument image and returning the bounding box information; when executing the key point localization skill function in the order of the skill function operation chain, determining the pointer center point position and pointer endpoint position from the image area of ​​the instrument bounding box; when executing the optical character recognition skill function in the order of the skill function operation chain, recognizing the scale value text and the corresponding scale position from the image area of ​​the instrument bounding box; when executing the reading recognition skill function in the order of the skill function operation chain, determining the current reading value of the target instrument image based on the pointer center point position, pointer endpoint position, scale value text, and corresponding scale position. In the implementation of the above scheme, by executing various skill functions such as semantic localization skill function, key point localization skill function, optical character recognition skill function and reading recognition skill function in the skill function operation chain, the system can dynamically identify the core requirements of the task. Thus, based on the joint understanding of cross-modal context, it can execute multi-step skill function operation chains, which can effectively overcome the defects of fixed algorithm flow and improve the flexibility of handling dynamic tasks.

[0010] Optionally, in this embodiment, after identifying the scale value text and corresponding scale position from the image area of ​​the instrument boundary box, the method further includes: determining whether the scale value text meets preset verification conditions. The preset verification conditions include: the scale value text contains preset characters; or, the ratio between the number of preset characters in the scale value text and the total number of characters in the scale value text is greater than a preset threshold; or, the scale value trend in the scale value text does not show a monotonically increasing trend or a monotonically decreasing trend. If so, the PaddleOCR tool function corresponding to the optical character recognition skill function is called. The PaddleOCR tool function is used to identify the scale value text and corresponding scale position from the image area of ​​the instrument boundary box. In the implementation of the above scheme, by introducing preset verification conditions, the initially identified scale value text is logically verified before calling the PaddleOCR tool to identify non-compliant abnormal text. When non-compliant abnormal text exists, the PaddleOCR tool is called to re-identify the scale value text and corresponding scale position from the image area of ​​the instrument boundary box. The above scheme can effectively filter out non-compliant abnormal text (such as data with incorrect formatting, fragmented content, or contradictory trends), improving the overall recognition accuracy. The monotonic trend judgment of the scale value trend in the above preset verification conditions can ensure the rationality of the scale value, avoid numerical jumps caused by image noise or misrecognition, and enhance the robustness of the system to instrument readings in complex scenarios.

[0011] Optionally, in this embodiment, before inputting the target instrument image and target question text into the trained visual language model, the method further includes: obtaining a multimodal training dataset for a general instrument, the multimodal training dataset including: sample instrument images, sample question text, sample answer text corresponding to the sample question text, and sample skill function operation chains; using the sample instrument images and sample question text as training data, and the sample answer text and sample skill function operation chains as training labels, fine-tuning the visual language model; wherein, the sample skill function operation chains are used for semantic consistency verification based on the visual language model, and for correcting the sample skill functions using a general language model. After the operation chain is added to the multimodal training dataset, semantic consistency verification is used to ensure that the description after each skill function operation in the sample skill function operation chain matches the actual collected visual information. The sample skill function operation chain is corrected using a general language model, including: if a missing key step or an abnormal step order is identified in the sample skill function operation chain, the general language model is used to correct the sample skill function operation chain, obtaining a corrected skill function operation chain, which is then added to the multimodal training dataset. The corrected skill function operation chain is a skill function operation chain that conforms to the semantics of the sample question text. In the implementation of the above scheme, fine-tuning is performed using a general instrument multimodal training dataset containing sample instrument images, sample question text, sample answer text, and sample skill function operation chains. This allows the Visual Language Model (VLM) to learn the deep semantic association between instrument images and natural language tasks. During the inference phase, the model can more accurately parse user intent and generate skill function operation chains that highly match the instrument images, thereby significantly improving the visual language model's cross-modal alignment ability between instrument images and task instructions.

[0012] This application also provides a general instrument image processing device, including: an image question text acquisition module for acquiring a target instrument image and a target question text; an image question text input module for inputting the target instrument image and the target question text into a trained visual language model to match multiple candidate skill functions in a predefined skill function set through the visual language model; a skill function operation chain generation module for constructing a skill function operation chain corresponding to multiple candidate skill functions based on the input-output dependency relationship of each candidate skill function through the visual language model; and a skill function operation chain execution module for executing the skill functions in the skill function operation chain in the order of the skill function operation chain to obtain the final answer corresponding to the target question text.

[0013] Optionally, in this embodiment, the skill function operation chain execution module includes: a skill function judgment submodule, used to obtain the currently executing skill function in the skill function operation chain according to the order of the skill function operation chain, and to determine whether the current skill function has a corresponding utility function; and a final result determination submodule, used to determine the final execution result based on the execution result of the current skill function if the current skill function has a corresponding utility function.

[0014] Optionally, in this embodiment, the final result determination submodule includes: a call condition judgment unit, used to determine whether the execution result of the current skill function meets the preset call conditions; and a tool function call unit, used to call the tool function corresponding to the current skill function so that the tool function corresponding to the current skill function and the skill function operation chain work together to obtain the final execution result.

[0015] Optionally, in this embodiment, the tool function execution submodule includes: a skill function parsing unit, used to parse the function operation name and operation parameters of the current skill function from the skill function operation chain generated by the visual language model; and a tool function calling unit, used to transmit the function operation name and operation parameters of the current skill function to the interface of the tool function corresponding to the current skill function, so as to call and execute the tool function corresponding to the current skill function.

[0016] Optionally, in this embodiment of the application, the final result determination submodule further includes: an execution result determination unit, used to determine the execution result of the current skill function as the final execution result if the execution result of the current skill function does not meet the preset calling conditions.

[0017] Optionally, in this embodiment, the predefined skill function set includes: semantic localization skill function, key point localization skill function, optical character recognition skill function, and reading recognition skill function; the skill function operation chain execution module includes: an instrument boundary localization submodule, used to locate the instrument position area from the target instrument image and return the instrument boundary box information when the semantic localization skill function is executed in the order of the skill function operation chain; a pointer position determination submodule, used to determine the pointer center point position and pointer endpoint position from the image area of ​​the instrument boundary box when the key point localization skill function is executed in the order of the skill function operation chain; an instrument scale recognition submodule, used to recognize the scale value text and scale corresponding position from the image area of ​​the instrument boundary box when the optical character recognition skill function is executed in the order of the skill function operation chain; and a current reading determination submodule, used to determine the current reading value of the target instrument image based on the pointer center point position, pointer endpoint position, scale value text, and scale corresponding position when the reading recognition skill function is executed in the order of the skill function operation chain.

[0018] Optionally, in this embodiment, the skill function operation chain execution module further includes: a scale text judgment submodule, used to determine whether the scale value text meets preset verification conditions, the preset verification conditions including: the scale value text includes preset characters, or the ratio between the number of preset characters in the scale value text and the total number of characters in the scale value text is greater than a preset threshold, or the scale value trend in the scale value text does not show a monotonically increasing trend or does not show a monotonically decreasing trend; and a scale value recognition submodule, used to call the PaddleOCR tool function corresponding to the optical character recognition skill function if the scale value text meets the preset verification conditions, the PaddleOCR tool function being used to identify the scale value text and the corresponding scale position from the image area of ​​the instrument boundary box.

[0019] Optionally, in this embodiment of the application, the general instrument image processing device further includes: a training data acquisition module, used to acquire a multimodal training dataset, the multimodal training dataset including: sample instrument images, sample question text, sample answer text corresponding to the sample question text, and sample skill function operation chain; and a model fine-tuning training module, used to fine-tune the visual language model using the sample instrument images and sample question text as training data, and the sample answer text and sample skill function operation chain as training labels. The sample skill function operation chain is defined as follows: semantic consistency verification is performed based on a visual language model, and the sample skill function operation chain is then corrected using a general language model before being added to the multimodal training dataset. Semantic consistency verification ensures that the description of each skill function operation in the sample skill function operation chain matches the actual collected visual information. Correcting the sample skill function operation chain using a general language model includes: if a missing key step or an abnormal step order is identified in the sample skill function operation chain, the sample skill function operation chain is corrected using a general language model to obtain a corrected skill function operation chain, which is then added to the multimodal training dataset. The corrected skill function operation chain is a skill function operation chain that conforms to the semantics of the sample question text.

[0020] This application also provides an electronic device, including a processor and a memory, wherein the memory stores machine-readable instructions executable by the processor, and the machine-readable instructions are executed by the processor to perform the methods described above.

[0021] This application also provides a computer-readable storage medium storing a computer program, which is executed by a processor to perform the methods described above.

[0022] This application also provides a computer program product, including: a computer program or computer instructions, which are executed by a processor to perform the method described above. Attached Figure Description

[0023] To more clearly illustrate the technical solutions of the embodiments of this application, the accompanying drawings used in the embodiments of this application will be briefly introduced below. It should be understood that the following drawings only show some embodiments of this application and should not be regarded as a limitation of the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0024] Figure 1 The illustration shows a flowchart of a general instrument image processing method provided in an embodiment of this application; Figure 2 The diagram shown illustrates the fine-tuning training of the visual language model provided in this embodiment of the application. Figure 3 The diagram shown is a structural schematic of the general-purpose instrument image processing device provided in an embodiment of this application; Figure 4 The diagram shows a structural schematic of an electronic device provided in an embodiment of this application. Detailed Implementation

[0025] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. It should be understood that the accompanying drawings in the embodiments of this application are for illustrative and descriptive purposes only and are not intended to limit the protection scope of the embodiments of this application. Furthermore, it should be understood that the schematic drawings are not drawn to scale. The flowcharts used in the embodiments of this application illustrate operations implemented according to some embodiments of this application. It should be understood that the operations in the flowcharts may not be implemented in sequence, and steps without logical contextual relationships may be reversed or implemented simultaneously. In addition, those skilled in the art, guided by the content of the embodiments of this application, may add one or more other operations to the flowcharts, or remove one or more operations from the flowcharts.

[0026] Furthermore, the described embodiments are merely a part of the embodiments of this application, and not all of them. The components of the embodiments of this application described and illustrated herein can generally be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of this application provided in the accompanying drawings is not intended to limit the scope of the claimed embodiments of this application, but merely to illustrate selected embodiments of this application.

[0027] It is understood that the terms "first" and "second" in the embodiments of this application are used to distinguish similar objects. Those skilled in the art will understand that the terms "first" and "second" do not limit the quantity or execution order, and that "first" and "second" do not necessarily imply that they are different. In the description of the embodiments of this application, the term "and / or" is merely a description of the association relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, and B existing alone. Additionally, the character " / " in this document generally indicates that the preceding and following related objects have an "or" relationship. The term "multiple" refers to two or more (including two), and similarly, "multiple groups" refers to two or more groups (including two groups).

[0028] It should be noted that the general instrument image processing method provided in this application embodiment can be executed by an electronic device. Here, electronic device refers to a device terminal with the function of executing computer programs or the aforementioned server. Device terminals include, for example, robots, smartphones, personal computers, tablet computers, personal digital assistants, or mobile internet devices. Servers refer to devices that provide computing services through a network, such as x86 servers and non-x86 servers.

[0029] The following are application scenarios for this general instrument image processing method: industrial production and energy management, where various instruments (such as pressure gauges, electricity meters, and water meters) play a crucial role in status monitoring and process control. Instrument recognition systems in these scenarios typically employ fixed algorithmic flows to complete the recognition and reading tasks. This involves pre-defined image processing steps, such as extracting the instrument area using target detection algorithms, performing key point detection, and then optical character recognition (OCR) to parse the instrument readings. These processing flows are often hard-coded into the system, lacking flexibility and making it difficult to dynamically adjust the processing logic based on task content or user queries. In practical applications, traditional methods are often only applicable to specific types of instruments and predefined problems. For example, when a user's task is "Which scale does the instrument pointer point to?", the system needs to be pre-designed with a fixed process: "Detecting the instrument's location area, detecting the pointer position at key points, extracting the scale, OCR recognition, and analyzing the scale value." However, if the user's task becomes "What is the instrument's range?", and the system lacks pre-defined processing logic (such as "OCR recognition, returning the instrument's range"), it cannot respond to the user's task request. Therefore, current instrument recognition systems lack the ability to understand task semantics, resulting in low flexibility and poor versatility in handling dynamic tasks, and the processing flow is rigid.

[0030] For the above issues, please refer to Figure 1 The illustrated flowchart shows a general instrument image processing method provided in an embodiment of this application; the implementation of the above-described general instrument image processing method may include: Step S110: Obtain the target instrument image and the target problem text.

[0031] The target instrument image is an image of a specific instrument that needs to be processed, such as an image of a pressure gauge, thermometer, or meter. For example, a user may take a picture of an analog pressure gauge to query its current pressure value.

[0032] The target question text is a specific question posed in response to the target instrument image. This target question text falls into many categories, including: instrument image description questions, instrument identification questions, instrument counting questions, instrument location questions, instrument key point detection questions, instrument OCR capability questions, and instrument reading capability questions. Understandably, the aforementioned target question text is usually expressed in natural language, such as "Describe the instrument," "Identify the type of instrument in the image?", "How many instruments are there?", "Where is the instrument?", "What is the current reading of the instrument?", or "What is the range of the instrument?".

[0033] Step S120: Input the target instrument image and the target question text into the trained visual language model to match multiple candidate skill functions in a predefined skill function set through the visual language model.

[0034] A Vision-Language Model (VLM) is a multimodal artificial intelligence model that can simultaneously understand and process visual information (such as images or videos) and natural language information (such as text or speech-to-text), and establish semantic connections and interactive reasoning capabilities between the two. This VLM can adopt the CogCoM model; where CoM stands for Chain of Manipulations, which is an existing reasoning mechanism for Vision-Language Models (VLM).

[0035] A predefined set of skill functions, also known as the semantic space of skill functions, is a set of predefined, reusable image processing or logical reasoning functions. This semantic space is the set of linguistic representations of all predefined skill functions. In visual tasks such as general instrument recognition and reading, to achieve flexible invocation and combinatorial reasoning for general tasks, it is usually necessary to predefine the semantic space of skill functions, i.e., the predefined set of skill functions.

[0036] Understandably, the skill functions in the aforementioned predefined skill function set are semantic skill functions, specifically functions described using natural language text. These are essentially "pseudo" functions. For example, a predefined Grounding function might be described as: "Using the Grounding skill function to define the gauges in the graph, we obtain the gauge coordinates as [x1, y1, x2, y2]." This is achieved by fine-tuning the large visual language model (specifically CogCoM) using a constructed gauge manipulation chain dataset, enabling it to perform visual reasoning through these "pseudo" skill functions, such as identifying gauge positions and reading gauge readings.

[0037] Step S130: Construct skill function operation chains corresponding to multiple candidate skill functions based on the input-output dependencies of each candidate skill function using a visual language model.

[0038] Candidate skill functions are functions that make up the skill function operation chain. For example, based on the user question text and the content of the meter image, VLM filters out the skill functions that may need to be called from the predefined skill function set.

[0039] Input-output dependency describes the logical relationship of data transfer between skill functions. Since candidate skill functions have definite input-output relationships, the visual language model can determine the logical relationship between multiple candidate skill functions based on their input-output relationships. For example, the output of skill A is the input of skill B. Skill A can be "detecting pointer position" and outputting pointer coordinates, while skill B can be "matching scale values ​​according to pointer position" and corresponding to the input pointer coordinates, while outputting scale values.

[0040] A skill function operation chain is a processing flow text composed of multiple skill functions arranged in a logical order. Specifically, it is a skill function operation chain text generated by a visual language model, and its form is a piece of natural language text. For example, the skill function operation chain for extracting the range could be "Using GROUNDING(meter) to locate the meter in the image and return the bounding box as [[334,175,589,620]]), get otag2Based on OCR(identify all values ​​of the meter) to identify all scale values ​​of the meter, which is 0,2,4,6,8,10. So ultimately, the conclusive answer to the question in discussion is [['0', '2', '4', '6', '8', '10']]".

[0041] Understandably, since the semantic definition of each skill function remains general and does not depend on a specific task context, visual language models can autonomously combine these skill functions into skill function operation chains in any task environment, thereby enabling visual language models to use these skill function operation chains to accomplish very complex tasks.

[0042] Step S140: Execute the skill functions in the skill function operation chain in the order of the skill function operation chain to obtain the final answer corresponding to the target question text.

[0043] The above solution uses a visual language model to understand the user's question text, automatically matches and calls the appropriate image processing skill function, and realizes an end-to-end closed loop from "asking a question in one sentence" to "automatically performing image analysis". This ability to directly map natural language into image processing logic breaks through the previous limitation that the algorithm process had to be designed manually.

[0044] It is understandable that steps S110 to S140 above can be expressed by the following formula:

[0045]

[0046] ; in, Representing a visual language model, Represents the skill function operation chain, Represents the target instrument image. Represents the target question text. This represents the final answer to the target question text. This indicates the first operation in the skill function operation chain. Each skill function Indicates the first The name of the operation function of each skill function. This represents a set consisting of multiple candidate skill functions. Indicates the first Each skill function is described in its descriptive text, such as its purpose, dependent input parameters, and expected output, in natural language. Since the skill function is semantically expressed in natural language, it can be invoked as a skill function of the visual language model during model inference. Multiple skill functions are organized into an executable, ordered inference chain according to inference logic, thus obtaining the skill function operation chain. Leveraging the powerful cross-modal semantic understanding capabilities of the visual language model, the system can generate skill function operation chains by combining question semantics and image content, even without training data for a specific meter type. This allows for accurate invocation and execution of the skill function operation chain to complete the task assigned by the existing skill function. For example, even when faced with a new type of digital meter that has never been trained on, the VLM can still understand "what is the reading" and correctly identify the reading.

[0047] In the implementation of the above scheme, by inputting the target instrument image and the target question text into the trained visual language model, the visual language model can jointly understand the cross-modal context of the instrument image and the natural language task. It matches candidate skill functions from a predefined skill function set and generates a multi-step skill function operation chain based on the input-output dependency relationship of the automatically derived skill functions. This combination of the visual language model's contextual cross-modal understanding ability and the dynamic construction ability of the skill function operation chain enables the system to dynamically combine and execute multi-step skill function operation chains based on the task intent of the target question text. This gives the system strong versatility and scalability, and it can adapt to the changing task requirements. It effectively overcomes the adaptability limitations of traditional fixed processes in complex scenarios. It can automatically identify different models, ranges and types of digital and pointer instruments in various scenarios. It not only improves the general recognition and reading ability of instruments for various tasks (such as information extraction and reading inference tasks), but also improves the flexibility and versatility in handling dynamic tasks.

[0048] As an optional implementation of step S140 above, the implementation of executing skill functions in the skill function operation chain in the order of the skill function operation chain may include: Step S141: Obtain the currently executing skill function in the skill function operation chain according to the order of the skill function operation chain, and determine whether the current skill function has a corresponding utility function.

[0049] Utility functions refer to real functions that are not internal to the visual language model but are implemented by calling external tools, such as PaddleOCR. These utility functions can be executed by calling external interfaces.

[0050] For example, step S141 can be implemented as follows: The currently executing skill function in the skill function operation chain is obtained according to the order of the skill function operation chain. This can be achieved by parsing the current skill function from the text corresponding to the skill function operation chain generated by the visual language model using regular expression matching, for example: (func_name(input)->output,func_desc). Here, func_name is the language call identifier of the skill function (e.g., the unique identifier for a semantic positioning skill function is grounding, while the unique identifier for a statistical meter border quantity skill function is counting), input is the operation input value of the skill function, which can be an image region, text, and / or a set of location points, etc., output is the operation output value of the skill function, and func_desc is the natural language interpretation of the skill function.

[0051] Step S142: If the current skill function has a corresponding utility function, then determine the final execution result based on the execution result of the current skill function.

[0052] It is understandable that the aforementioned utility functions can be integrated through a unified registration format, enabling dynamic invocation by the visual language model at runtime. This unified registration format can include: function name, functional description, input parameters, output parameters, and invocation type. The function name is a unique identifier used to identify the specific function being invoked during dynamic calls. The functional description uses natural language to describe the function's purpose. Input parameters are the specific data required by the function. Output parameters are the data output after execution. The invocation type can be a local function call (i.e., a function implemented using native code). Furthermore, utility functions can be designed for specific tasks with lightweight features (e.g., accelerated using TensorRT), significantly improving execution efficiency compared to the general computation of general skill functions, thereby effectively reducing computational resource consumption and execution time.

[0053] As an optional implementation of step S142 above, the implementation of calling the utility function above may include: Step S142a: Parse the function operation name and operation parameters of the current skill function from the skill function operation chain generated by the visual language model.

[0054] For example, the implementation of step S142a above is as follows: if the currently executing skill function is parsed from the skill function operation chain generated by the visual language model and is in the form of keypoint(bbx_1)->pts_1, where the function operation name of the current skill function is keypoint and the operation parameter of the current skill function is bbx_1.

[0055] Step S142b: Pass the function operation name and operation parameters of the current skill function to the interface of the utility function corresponding to the current skill function, so as to call and execute the utility function corresponding to the current skill function.

[0056] An example implementation of step S142b above is as follows: After parsing the function operation name and operation parameters of the current skill function, the function operation name keypoint and operation parameter bbx_1 of the current skill function can be transmitted to the interface of the tool function corresponding to the current skill function (i.e., the tool function corresponding to keypoint), so that the tool function corresponding to the current skill function can be called and executed through this interface. After calling and executing the interface of the tool function corresponding to the current skill function, the return result of the tool function corresponding to the current skill function through the interface is in a structured format (such as keypoint coordinates, numerical readings, recognized text, etc.). This structured return result can then be used as the input of the next skill function or the tool function in the skill function operation chain. Therefore, by calling and executing the interface of the tool function corresponding to the current skill function, the robustness of the entire skill function operation chain can be improved.

[0057] In implementing the above scheme, by parsing the function operation names and parameters in the skill function operation chain generated by the Visual Language Model (VLM), natural language instructions are directly converted into executable utility function calls. This mechanism overcomes the limitation of traditional systems where the "skill function operation chain" relies on predefined static processes. By transforming abstract semantics into operable parameters, this approach achieves a closed loop from cross-modal reasoning to precise execution, significantly improving the flexibility and accuracy of task processing.

[0058] Optionally, as an alternative implementation of step S142 above, in the step S142 where the final execution result is determined based on the execution result of the current skill function if a corresponding utility function exists, the implementation of determining the final execution result may include: Step S142c: Determine whether the execution result of the current skill function meets the preset calling conditions.

[0059] The current skill function refers to the specific function that is currently being processed or executed in the skill function operation chain. Its execution result refers to the data, status, or return value output by the current skill function after completing its task, which is used to determine whether the conditions for further calling of utility functions are met.

[0060] Predefined invocation conditions are predefined rules or standards used to determine whether the execution result of the current skill function meets the requirements for invoking the utility function.

[0061] Step S142d: If the execution result of the current skill function meets the preset calling conditions, then the utility function corresponding to the current skill function is called so that the utility function corresponding to the current skill function and the skill function operation chain work together to obtain the final execution result.

[0062] A utility function is an auxiliary function or module tool associated with the current skill function. It is used to further process or supplement the current skill function when certain conditions are met. This utility function can be an independently written functional module function.

[0063] Optionally, after determining whether the execution result of the current skill function meets the preset calling conditions in the above steps, the method may further include: Step S142e: If the execution result of the current skill function does not meet the preset calling conditions, then the execution result of the current skill function is determined as the final execution result.

[0064] For example, in implementing step S142e above: if the execution result of the current skill function does not meet the preset calling conditions, then after still executing the currently predefined semantic skill function, the execution result corresponding to the current skill function is obtained and output, and the execution result of the current skill function is determined as the final execution result. This final execution result is the final data or state output after the skill function operation chain and utility function have been collaboratively processed. In the implementation of the above scheme, by directly using the result as the final output when the execution result of the current skill function does not meet the calling conditions, without calling the utility function or performing additional processing, this mechanism avoids the waste of resources (such as memory allocation, computational overhead, etc.) caused by redundant operations triggered due to unmet conditions, thereby reducing system load and improving overall execution efficiency.

[0065] As an optional implementation of step S140 above, the predefined skill function set may include: a semantic localization skill function (grounding), a keypoint localization skill function (keypoint), an optical character recognition (OCR) skill function, and a reading recognition skill function (reading). It is understood that, theoretically, each skill function in the predefined skill function set can selectively call its corresponding utility function. In some implementations, some skill functions may not require corresponding utility functions; for example, the semantic localization skill function (grounding) and the keypoint localization skill function (keypoint) can be directly called without setting utility functions for them. In other implementations, some skill functions need to determine whether to call their corresponding utility function based on the execution result of the skill function. For example, it may be based on whether the execution result of the optical character recognition (OCR) skill function meets a preset calling condition. If the execution result of the optical character recognition (OCR) skill function meets the preset calling condition, then the utility function corresponding to the optical character recognition (OCR) skill function is called. In other implementations, some skill functions are directly replaced with the corresponding utility functions during execution. For example, the reading recognition skill function (reading) is directly replaced with the utility function corresponding to the reading function during execution.

[0066] The above-described implementation of executing skill functions in the skill function operation chain in the order of the skill function operation chain may include steps S143-S146: Step S143: When the semantic localization skill function is executed in the order of the skill function operation chain, the instrument location area is located from the target instrument image and the instrument bounding box information is returned.

[0067] The implementation of step S143 above is as follows: It is understood that each skill function has a unique identifier in its corresponding semantic space. For example, the unique identifier for the semantic localization skill function is "grounding," while the unique identifier for the skill function that counts the number of meter borders is "counting." When the visual language model executes the semantic localization skill function (grounding), it can locate the meter region from the target meter image, thereby obtaining the located meter position region and returning the meter bounding box information. Specifically, if the skill function operation chain generated by the visual language model includes grounding(meter) → bbx_1, then the semantic localization skill function (grounding) can be called to locate the meter in the current meter image and return the meter bounding box bbx_1.

[0068] Step S144: When executing the key point positioning skill function in the order of the skill function operation chain, determine the position of the pointer center point and the position of the pointer endpoint from the image area of ​​the instrument bounding box.

[0069] The implementation of step S144 above is as follows: It is understood that each skill function has a unique identifier in its corresponding semantic space; for example, the unique identifier of the keypoint localization skill function is `keypoint`. It is understood that since the keypoint localization skill function (`keypoint`) is directly called and no tool function needs to be set for it, when the visual language model executes the keypoint localization skill function (`keypoint`) in the order of the skill function operation chain, it can directly call the keypoint localization skill function (`keypoint`) to determine the pointer center point position and pointer endpoint position from the image region of the instrument bounding box, and then obtain the specific coordinate values ​​of the pointer center point position and the pointer endpoint position. Specifically, if the skill function operation chain generated by the visual language model includes the form `keypoint(bbx_1)→pts_1`, then the keypoint localization skill function (`keypoint`) can be directly called to locate the pointer center and endpoint positions based on the image region corresponding to the instrument bounding box `bbx_1`, and return the pointer keypoint position `pts_1`.

[0070] Step S145: When the optical character recognition skill function is executed in the order of the skill function operation chain, the scale value text and the corresponding scale position are identified from the image area of ​​the instrument boundary box.

[0071] For example, in implementing step S145 above, when the visual language model executes the optical character recognition (OCR) skill function according to the skill function operation chain, it identifies the scale value text and the specific coordinates of the corresponding scale position from the image area of ​​the instrument bounding box. Specifically, if the skill function operation chain generated by the visual language model includes ocr(bbx_1)→txt_1, that is, the execution result of the optical character recognition skill function ocr(bbx_1) is in the form of txt_1, then if the execution result of the optical character recognition skill function (ocr) (i.e., txt_1) meets the preset calling conditions (such as recognition accuracy being lower than the preset accuracy), the tool function corresponding to the optical character recognition skill function (ocr) can be used to identify the scale value text and its position in the image area corresponding to the instrument bounding box bbx_1, and return the scale value text and the corresponding position [txt_i,pts_i]. Wherein, if the execution result of the optical character recognition skill function (ocr) (i.e., txt_1) does not meet the preset calling conditions, the execution result of the optical character recognition skill function (ocr) is determined as the final execution result of the current step. The process for determining whether the execution result (i.e., txt_1) of the Optical Character Recognition (OCR) skill function meets the preset calling conditions is as follows: The process checks whether the scale value text meets the preset verification conditions. These conditions include: the scale value text contains preset characters; or, the ratio between the number of preset characters in the scale value text and the total number of characters in the scale value text is greater than a preset threshold; or, the scale value trend in the scale value text does not show a monotonically increasing or decreasing trend. Preset characters can be abnormal characters (e.g., non-numeric characters). Specifically, for the ratio between the number of preset characters in the scale value text and the total number of characters in the scale value text to be greater than the preset threshold, the preset threshold can be set to 0.2. The process then compares the number of preset characters (i.e., the number of abnormal characters) obtained after executing the OCR skill function with the total number of characters obtained to determine if the ratio exceeds the preset threshold.

[0072] Therefore, as an optional implementation of the above-mentioned general instrument image processing method, after executing the OCR operation in the order of the skill function operation chain and identifying the scale value text and the corresponding scale position from the image region of the instrument bounding box, it may further include: Step S145a: Determine whether the scale value text meets the preset verification conditions. The preset verification conditions include: the scale value text contains preset characters, or the ratio between the number of preset characters in the scale value text and the total number of characters in the scale value text is greater than a preset threshold, or the scale value trend in the scale value text does not show a monotonically increasing trend or does not show a monotonically decreasing trend.

[0073] Scale text refers to the numerical or character information marked on the instrument scale, used to indicate the instrument's measured value or status. This text is usually located next to the scale lines of the instrument and is the target content for optical character recognition (OCR).

[0074] Preset validation conditions are a set of predefined rules or standards used to determine whether the scale value text needs further OCR processing. These preset validation conditions include: the scale value text contains preset characters; or, the ratio between the number of preset characters in the scale value text and the total number of characters in the scale value text is greater than a preset threshold; or, the scale value trend in the scale value text does not show a monotonically increasing trend or a monotonically decreasing trend (i.e., the scale value trend has fluctuations or non-linear changes). Preset characters refer to characters or symbols in the scale value text that require special attention, which may be unit symbols, special marks, or other key information. The preset threshold is a pre-set value used to compare the ratio of the number of preset characters in the scale value text to the total number of characters. The scale value trend refers to whether the arrangement order of the values ​​in the scale value text conforms to a monotonically increasing or monotonically decreasing pattern.

[0075] Step S145b: If the scale value text meets the preset verification conditions, the PaddleOCR tool function corresponding to the optical character recognition skill function is called. The PaddleOCR tool function is used to identify the scale value text and the corresponding scale position from the image area of ​​the instrument bounding box.

[0076] The Optical Character Recognition (OCR) skill function is a software module that encapsulates OCR technology to recognize text content from images. This function may call PaddleOCR utility functions or execute the OCR skill function directly. The PaddleOCR utility functions are the specific functional interfaces provided by the PaddleOCR library, used to recognize text and location information from images.

[0077] Step S145c: If the scale value text does not meet the preset verification conditions, the optical character recognition skill function is executed directly, and the execution result of the optical character recognition skill function is determined as the final execution result.

[0078] Optionally, the above-described general instrument image processing method can also handle the task of identifying instrument types. For example, if the target question text is "Identify the type of instrument in the image?", the instrument region can first be located from the target instrument image using a semantic localization skill function (grounding) to obtain the bounding box of the located instrument. Let's assume the bounding box here is [[334,175,589,620]]. Then, an optical character recognition (OCR) skill function is used to recognize this bounding box [[334,175,589,620]] to identify the specific type of the target instrument. If the specific type of the target instrument is a pressure gauge, then the returned answer text could be "pressure gauge (MPa)".

[0079] Optionally, the above-described general instrument image processing method can also handle tasks involving the extraction of measurement range types. For example, if the target question text is "Identify the measurement range of the instrument in the image?", a semantic localization skill function (grounding) can be used to locate the instrument region in the target instrument image, thereby obtaining the bounding box where the instrument is located. Assuming the bounding box here is [[334,175,589,620]]. Then, an optical character recognition skill function (OCR) is used to recognize the scale within the bounding box [[334,175,589,620]], thereby obtaining all scale values ​​as 0, 2, 4, 6, 8, 10. The returned answer text could then be [['0', '2', '4', '6', '8', '10']].

[0080] Step S146: When the reading recognition skill function is executed in the order of the skill function operation chain, the current reading value of the target instrument image is determined based on the pointer center point position, pointer endpoint position, scale value text and scale corresponding position.

[0081] In the process of determining the final execution result based on the execution result of the reading recognition skill function, the implementation method of step S146 above is as follows: When the visual language model executes the reading recognition skill function (reading) in the order of the skill function operation chain, it first executes the reading recognition skill function (reading) to obtain the current reading value of the target instrument image; then, after determining that the tool function corresponding to the recognition skill function (reading) meets its calling conditions, it determines whether the current reading value is within the range covered by the two scale values ​​closest to the pointer position. If so, the current reading value is output; if not, the tool function corresponding to the reading recognition skill function (reading) is executed, and the reading result of the corresponding tool function is output. The method for determining whether the current reading value is within the range covered by the two scale values ​​closest to the pointer position is as follows: obtain the two scale values ​​closest to the pointer position; obtain the range covered by the two scale values, and then determine whether the current reading value is within that range.

[0082] Specifically, the execution steps of the tool function corresponding to the reading skill function are as follows: Based on the pointer key point position pts_1 and the scale value information [txt_i, pts_i], determine the current reading value value_1 of the instrument. The current reading value of the instrument is represented below as... r .

[0083] Assume the pointer's center point is represented as (px1, py1), the pointer's endpoints as (px2, py2), and the instrument scale text and its corresponding scale coordinates are represented as [r i , (x i ,y i Specifically, the scale value on the left side of the pointer and its coordinates are [ , (px3, py3)], to obtain the tick value to the right of the pointer and its coordinates are [ [px4, py4], the angle between the pointer and the line connecting the pointer's center point (px1, py1) to the pointer's endpoint (px2, py2) and the pointer's geometric angle can be calculated. The angle between the instrument pointer and the line connecting the pointer's right-hand scale value. and using the formula By performing calculations, the current reading of the instrument can be obtained. This indicates the current reading of the instrument. Indicates the tick value to the left of the pointer. This indicates the angle between the instrument pointer and the line connecting the left-hand scale value. This indicates the angle between the instrument pointer and the line connecting the pointer's right-hand scale value. This indicates the tick mark value to the left of the pointer.

[0084] Optionally, after constructing the skill function operation chain corresponding to multiple candidate skill functions in step S130 above, it is also possible to check whether the skill function operation chain has missing key steps or abnormal step operation order. This implementation may include: Step S150: Use a general language model to check the skill function operation chain and obtain the check result of the skill function operation chain.

[0085] An example implementation of step S150 above is as follows: A pre-constructed structured prompt template is obtained, and the skill function operation chain and target question text are filled into the structured prompt template to obtain the prompt engineering text. Then, the prompt engineering text is input into a general language model, which checks the skill function operation chain to obtain the check result of the skill function operation chain. The constructed structured prompts include, but are not limited to: "For the task description '[target question text]', the current skill function chain is '[preliminarily generated skill function chain]'. Please check whether the skill function operation chain has missing key steps or an abnormal step order. Confirm whether the semantic consistency verification result of the skill function operation chain indicates missing key steps or an abnormal step order, and output the check result of the skill function operation chain."

[0086] Step S160: If the inspection result of the skill function operation chain is that there are missing key steps or abnormal step order, the sample skill function operation chain can be corrected using a general language model to obtain a corrected skill function operation chain.

[0087] For example, in implementing step S160 above: if an executable program written in a programming language detects that the check result of the skill function operation chain is that there are missing key steps or abnormal step order, then the sample skill function operation chain can be corrected again using the operation chain correction prompt template. For example, the operation chain correction prompt template can be input into a general language model so that the general language model can correct the skill function operation chain and output the corrected skill function operation chain. The operation chain correction prompt template above could be "For the task description '[target problem text]', the current skill function chain is '[preliminary generated skill function chain]', please adjust or complete the necessary skill functions in a reasonable order to generate a complete skill function chain that meets the task requirements."

[0088] Step S170: Execute the skill functions in the skill function operation chain in the order of the modified skill function operation chain above to obtain the final answer corresponding to the target question text.

[0089] The implementation of step S170 is similar to that of step S140. If there are any unclear points, please refer to the implementation of step S140 above. It will not be repeated here.

[0090] Optionally, in practice, the skill function operation chain can be directly modified using a general language model to obtain a modified skill function operation chain. For example, an operation chain modification prompt template can be input into the general language model, allowing the model to directly modify the sample skill function operation chain and output the modified one. The aforementioned operation chain modification prompt template could be something like, "For the task description '[target problem text]', the current skill function chain is '[preliminary generated skill function chain]'. If the above skill function operation chain has missing key steps or an abnormal step order, please adjust or complete the necessary skill functions in a reasonable order to generate a complete skill function chain that meets the task requirements; if the above skill function operation chain does not have missing key steps or an abnormal step order, then directly output the current skill function chain."

[0091] Optionally, after constructing the skill function operation chain corresponding to multiple candidate skill functions in step S130 above, semantic consistency verification can also be performed on the skill function operation chain. This semantic consistency verification is used to ensure that the description after each skill function operation in the sample skill function operation chain matches the actually collected visual information. The above-described implementation of semantic consistency verification for the skill function operation chain may include: Step S180: Perform semantic consistency verification on the skill function operation chain based on multiple visual language models, and obtain the semantic consistency verification result of the skill function operation chain.

[0092] One embodiment of step S180 above is as follows: The first visual language model among multiple visual language models is used to perform semantic consistency verification on the skill function operation chain, obtaining a first verification result for the skill function operation chain; the second visual language model among multiple visual language models is used to perform semantic consistency verification on the skill function operation chain, obtaining a second verification result for the skill function operation chain; the third visual language model among multiple visual language models is used to perform semantic consistency verification on the skill function operation chain, obtaining a third verification result for the skill function operation chain; if at least two of the first verification result, the second verification result, and the third verification result are consistent with the description of each step of the skill function operation chain in the sample skill function operation chain and the actually collected visual information, then the semantic consistency verification result of the skill function operation chain can be set as verified as passed.

[0093] One embodiment of step S180 above is as follows: The skill function operation chain is semantically validated using a first visual language model from among multiple visual language models to obtain a first validation result; the skill function operation chain is semantically validated using a second visual language model from among multiple visual language models to obtain a second validation result; the skill function operation chain is semantically validated using a third visual language model from among multiple visual language models to obtain a third validation result; the skill function operation chain is semantically validated using a fourth visual language model from among multiple visual language models to obtain a fourth validation result; the skill function operation chain is semantically validated using a fifth visual language model from among multiple visual language models to obtain a fifth validation result; if at least three of the first, second, third, fourth, and fifth validation results are consistent with the description of each skill function operation step in the sample skill function operation chain and the actually collected visual information, then the semantic consistency validation result of the skill function operation chain can be set as validated successfully.

[0094] Please see Figure 2 The diagram shown is a fine-tuning training illustration of the visual language model provided in this application embodiment; as an optional implementation of the above-mentioned general instrument image processing method, before inputting the target instrument image and the target question text into the trained visual language model, it further includes: Step S210: Obtain the multimodal training dataset of the general instrument. The multimodal training dataset includes: sample instrument images, sample question texts, sample answer texts corresponding to the sample question texts, and sample skill function operation chains.

[0095] The aforementioned sample instrument images can be pre-captured images of instruments of different models, from multiple angles, and in complex environments. Instrument types can include pointer pressure gauges, thermometers, voltmeters, digital instruments, etc. The sample instrument images can encompass various types of instruments with different ranges, units, and shapes. "Multi-angle" refers to capturing images of each instrument from different perspectives, including front, side, and top views, simulating real-world usage scenarios. The diversity of environments refers to capturing instrument images under different lighting conditions (strong light, weak light, shadow) and background interference (reflection, obstruction, complex background). Optionally, after capturing the sample instrument images, a unified resolution can be applied, i.e., processing the instrument images to a uniform size and resolution to ensure consistency of input data.

[0096] The sample question text and corresponding sample answer text mentioned above can be obtained through pre-annotated data. Specifically, to collect more comprehensive instrument image data, this data provides structured supervision information for training the visual language model. Not only can the instrument type (i.e., the category of each instrument), instrument panel location (regional rectangle), and pointer location (coordinates of key points between the pointer's center and end points) be annotated in the sample question text and corresponding sample answers, but digital readings can also be directly annotated for digital instruments, and all scale ranges and the actual numerical value corresponding to the scale the instrument is pointing to can be annotated for pointer instruments.

[0097] It is understandable that the above annotation process can be implemented manually, using pre-developed algorithms (such as a combination of manual and algorithmic annotation for category annotation), or using pre-trained language models. Specifically, the sample skill function operation chain mentioned above can include process functions such as task parsing, visual operation scheduling, and output results. High-order language tasks can be manually decomposed into a series of executable intermediate reasoning operations, with each step clearly described in natural language and organized into a structured operation. Alternatively, pre-trained language models (such as the ChatGPT model) can be used to generate skill function operation chains for solving instrument recognition, positioning, and reading tasks, thereby simulating the step-by-step reasoning process of a visual language model for complex visual language tasks and providing structured data support for training the visual language model.

[0098] The visual language model is trained using the multimodal training dataset containing the sample skill function operation chain, which enables the trained visual language model to improve its ability to recognize complex reading tasks.

[0099] Step S220: Use sample instrument images and sample question texts as training data, and sample answer texts and sample skill function operation chains as training labels to fine-tune the visual language model.

[0100] Optionally, in order to ensure that the sample answer text corresponding to the sample question text conforms to the image annotation data specification answer description format and that the sample answer text has a standard output format, the sample answer text can be structured and its content standardized. For example: the output format of the instrument counting problem is [int], indicating the number of instruments in the image; the output format of the instrument positioning problem is [[x1,y1,x2,y2]], indicating the position of the instrument in the image; the output format of the instrument key point detection problem is [[px1,py1]], indicating the pixel coordinates of the center point or endpoint of the instrument pointer; the output format of the instrument range problem is [[txt1,txt2,...,txtn]], such as [['0.1','0.2',...,'0.8']], indicating the text data of the instrument scale value; the output format of the instrument reading problem is [[float]], indicating a specific instrument reading value; where int represents an integer, float represents a floating-point number, x1, y1, x2, y2 represent the pixel coordinates of the upper left and lower right corners of the instrument positioning rectangle, and px1, py1 represent the pixel coordinates of the pointer key points. For example, a uniform format (such as .json) can be used to perform structured annotation on each training sample. Each training sample in a multimodal training dataset file contains: image path (i.e., the path of sample instrument image I), sample question text Q, sample answer text A, and sample skill function operation chain S.

[0101] For example, the implementation of step S148 above can be carried out by using Low-Rank Adaptation (LoRA) technology to fine-tune the visual language model. Specifically, sample instrument images and sample question texts are used as training data, and sample answer texts and sample skill function operation chains are used as training labels. The visual language model (such as the CogCoM model) is fine-tuned and trained by LoRA technology to optimize the visual language model's perception and skill operation chain generation capabilities in general instrument recognition and reading tasks.

[0102] Furthermore, in some implementations, after automatically generating the skill function operation chain (i.e., after constructing the skill function operation chain corresponding to multiple candidate skill functions based on the input-output dependency relationship of each candidate skill function through a visual language model), a skill function correction wheel can be introduced. The role of the skill function correction wheel is to identify and correct problems such as missing steps, incorrect order, and logical inconsistencies, thereby improving the interpretability and execution stability of the inference chain.

[0103] Before executing the skill functions in the skill function operation chain in the order they appear, a multimodal training dataset for a general instrument can be obtained. This multimodal training dataset includes: sample instrument images, sample question texts, sample answer texts corresponding to the sample question texts, and sample skill function operation chains. The sample skill function operation chains are used for semantic consistency verification based on a visual language model. Specific implementation methods for semantic consistency verification may include: Step S221: Perform semantic consistency verification on the skill function operation chain based on the visual language model, and add the sample skill function operation chain to the multimodal training dataset after correcting it using a general language model. The semantic consistency verification is used to ensure that the description after each skill function operation in the sample skill function operation chain is consistent with the actual collected visual information.

[0104] Step S222: If a missing key step or an abnormal step order is identified in the sample skill function operation chain, the sample skill function operation chain is corrected using a general language model to obtain a corrected skill function operation chain. The corrected skill function operation chain is then added to the multimodal training dataset. The corrected skill function operation chain is a skill function operation chain that conforms to the semantics of the sample problem text.

[0105] The specific implementation of steps S221 to S222 above is as follows: After the initial skill function operation chain is generated, the skill function operation chain and the target problem text can be input into the trained visual language model.

[0106] Then, structured cue words are constructed to instruct the visual language model to verify the semantic consistency of the skill function chain. Other review cue words can also be used to review the completeness and rationality of the skill function chain. The constructed structured cue words include, but are not limited to: "For the task description '[target problem text]', the current skill function chain is '[preliminary generated skill function chain]'. Please check whether the chain contains all the key steps, and adjust or complete the necessary skill functions in a reasonable order to generate a complete skill function chain that meets the task requirements."

[0107] The technical effects of steps S221 to S222 described above can include: the visual language model, through a skill function correction round, identifies missing key skill functions, redundant steps, or sequential errors in the initially generated skill chain, and generates a corrected skill function chain, thereby improving the system's versatility, robustness, and interpretability of reasoning. One example is: the target question text is "Read the current instrument value," and the visual language model initially generates a skill function chain of "reading recognition," skipping operations such as "location" and "scale recognition." After the second round of correction, the visual language model outputs a corrected skill function chain as: "Semantic location → pointer location → OCR recognition of scale value → reading recognition." The system can first complete instrument area detection, pointer center point and endpoint location, and scale value and scale line matching, and then accurately calculate the reading. This results in a more interpretable and accurate target answer.

[0108] Please see Figure 3 The diagram shown is a structural schematic of a general-purpose instrument image processing device provided in an embodiment of this application; this application provides a general-purpose instrument image processing device 300, including: The image problem text acquisition module 310 is used to acquire the target instrument image and the target problem text.

[0109] The image question text input module 320 is used to input the target instrument image and the target question text into the trained visual language model, so as to match multiple candidate skill functions in a predefined skill function set through the visual language model.

[0110] The skill function operation chain generation module 330 is used to construct skill function operation chains corresponding to multiple candidate skill functions based on the input-output dependencies of each candidate skill function through a visual language model.

[0111] The skill function operation chain execution module 340 is used to execute the skill functions in the skill function operation chain in the order of the skill function operation chain to obtain the final answer corresponding to the target question text.

[0112] As an optional implementation of the above-mentioned device, the skill function operation chain execution module includes: The skill function judgment submodule is used to obtain the currently executing skill function in the skill function operation chain according to the order of the skill function operation chain, and to determine whether the current skill function has a corresponding utility function.

[0113] The utility function execution submodule is used to determine the final execution result based on the execution result of the current skill function if a corresponding utility function exists for the current skill function.

[0114] As an optional implementation of the above-mentioned device, the tool function execution submodule includes: The execution result judgment unit is used to determine whether the execution result of the current skill function meets the preset calling conditions.

[0115] The utility function call unit is used to call the utility function corresponding to the current skill function if the execution result of the current skill function meets the preset call conditions, so that the utility function corresponding to the current skill function and the skill function operation chain can work together to obtain the final execution result.

[0116] As an optional implementation of the above-mentioned device, the tool function call unit includes: The skill function parsing unit is used to parse the function operation name and operation parameters of the current skill function from the skill function operation chain generated by the visual language model.

[0117] The utility function call unit is used to pass the function operation name and operation parameters of the current skill function to the interface of the utility function corresponding to the current skill function, so as to call and execute the utility function corresponding to the current skill function.

[0118] As an optional implementation of the above-mentioned device, the tool function execution submodule further includes: The execution result determination unit is used to determine the execution result of the current skill function as the final execution result if the execution result of the current skill function does not meet the preset calling conditions.

[0119] As an optional implementation of the above-mentioned device, the predefined skill function set includes: semantic localization skill function, key point localization skill function, optical character recognition skill function, and reading recognition skill function; the skill function execution module includes: The instrument boundary localization submodule is used to locate the instrument location area from the target instrument image and return the instrument boundary box information when the semantic localization skill function is executed in the order of the skill function operation chain.

[0120] The pointer position determination submodule is used to determine the pointer center point position and pointer endpoint position from the image area of ​​the bounding box where the instrument is located when the key point positioning skill function is executed in the order of the skill function operation chain.

[0121] The instrument scale recognition submodule is used to identify the scale value text and the corresponding scale position from the image area of ​​the bounding box where the instrument is located when the optical character recognition skill function is executed in the order of the skill function operation chain.

[0122] The current reading determination submodule is used to determine the current reading value of the target instrument image based on the pointer center point position, pointer endpoint position, scale value text, and scale corresponding position when the reading recognition skill function is executed in the order of the skill function operation chain.

[0123] As an optional implementation of the above-mentioned device, the skill function operation chain execution module further includes: The scale setting judgment submodule is used to determine whether the scale value text meets the preset verification conditions. The preset verification conditions include: the scale value text contains preset characters, or the ratio between the number of preset characters in the scale value text and the total number of characters in the scale value text is greater than a preset threshold, or the scale value trend in the scale value text does not show a monotonically increasing trend or does not show a monotonically decreasing trend.

[0124] The scale value recognition submodule is used to call the PaddleOCR tool function corresponding to the optical character recognition skill function if the confidence value of the scale value text is lower than the preset confidence threshold. The PaddleOCR tool function is used to identify the scale value text and the corresponding scale position from the image area of ​​the instrument bounding box.

[0125] As an optional embodiment of the above-mentioned device, the general-purpose instrument image processing device further includes: The training data acquisition module is used to acquire a multimodal training dataset, which includes: sample instrument images, sample question texts, sample answer texts corresponding to the sample question texts, and sample skill function operation chains.

[0126] The model fine-tuning training module is used to fine-tune the visual language model using sample instrument images and sample question texts as training data, and sample answer texts and sample skill function operation chains as training labels. The sample skill function operation chain is defined as follows: semantic consistency verification is performed based on a visual language model, and the sample skill function operation chain is then corrected using a general language model before being added to the multimodal training dataset. Semantic consistency verification ensures that the description of each skill function operation in the sample skill function operation chain matches the actual collected visual information. Correcting the sample skill function operation chain using a general language model includes: if a missing key step or an abnormal step order is identified in the sample skill function operation chain, the sample skill function operation chain is corrected using a general language model to obtain a corrected skill function operation chain, which is then added to the multimodal training dataset. The corrected skill function operation chain is a skill function operation chain that conforms to the semantics of the sample question text.

[0127] It should be understood that this device corresponds to the above-described general instrument image processing method embodiment and is capable of performing the various steps involved in the above method embodiment. The specific functions of this device can be found in the description above, and detailed descriptions are appropriately omitted here. The device includes at least one software function module that can be stored in memory or embedded in the device's operating system (OS) in the form of software or firmware.

[0128] Please see Figure 4 The diagram shows a structural schematic of an electronic device provided in an embodiment of this application. An electronic device 400 provided in this application includes a processor 410 and a memory 420. The memory 420 stores machine-readable instructions executable by the processor 410. When the machine-readable instructions are executed by the processor 410, the method described above is performed.

[0129] This application embodiment also provides a computer-readable storage medium 430, on which a computer program is stored. This computer program is executed by a processor 410 to perform the methods described above. The computer-readable storage medium 430 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as Static Random Access Memory (SRAM), Electrically Erasable Programmable Read-Only Memory (EEPROM), Erasable Programmable Read Only Memory (EPROM), Programmable Read-Only Memory (PROM), Read-Only Memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk.

[0130] This application also provides a computer program product, including: a computer program or computer instructions, which are executed by a processor to perform the method described above.

[0131] It should be noted that the various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For apparatus embodiments, since they are basically similar to method embodiments, the description is relatively simple; relevant parts can be referred to the descriptions in the method embodiments.

[0132] It should be understood that the disclosed apparatus and methods can also be implemented in other ways, as provided in the embodiments of this application. The apparatus embodiments described above are merely illustrative. For example, the flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of apparatus, methods, and computer program products according to various embodiments of this application. In this regard, each block in the flowchart or block diagram may represent a module, segment, or portion of code, which contains one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions marked in the blocks may occur in a different order than those marked in the accompanying drawings. For example, two consecutive blocks may actually be executed substantially in parallel, or they may sometimes be executed in reverse order, depending primarily on the functions involved.

[0133] Furthermore, the functional modules of each embodiment in this application can be integrated together to form an independent part, or each module can exist independently, or two or more modules can be integrated to form an independent part. In addition, in the description of this specification, the reference to terms such as "one embodiment," "some embodiments," "example," "specific example," "some examples," etc., means that the specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of this application. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials, or characteristics described can be combined in a suitable manner in any one or more embodiments or examples. Furthermore, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification and the features of different embodiments or examples.

[0134] The above description is only an optional implementation of the embodiments of this application, but the protection scope of the embodiments of this application is not limited thereto. Any changes or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the embodiments of this application should be covered within the protection scope of the embodiments of this application.

Claims

1. A general instrument image processing method, characterized in that, include: Acquire the target instrument image and the target problem text; The target instrument image and the target question text are input into the trained visual language model so that multiple candidate skill functions can be matched in a predefined skill function set through the visual language model; Based on the input-output dependencies of each candidate skill function, the visual language model constructs the skill function operation chain corresponding to the multiple candidate skill functions; The skill functions in the skill function operation chain are executed in the order described to obtain the final answer corresponding to the target question text.

2. The method according to claim 1, characterized in that, The step of executing the skill functions in the skill function operation chain in the order of the skill function operation chain includes: According to the order of the skill function operation chain, obtain the currently executing skill function in the skill function operation chain, and determine whether the current skill function has a corresponding utility function; If so, the final execution result is determined based on the execution result of the current skill function.

3. The method according to claim 2, characterized in that, Determining the final execution result based on the execution result of the current skill function includes: Determine whether the execution result of the current skill function meets the preset calling conditions; If so, the utility function corresponding to the current skill function is called so that the utility function corresponding to the current skill function and the skill function operation chain work together to obtain the final execution result.

4. The method according to claim 3, characterized in that, The invocation of the utility function corresponding to the current skill function includes: The function operation name and operation parameters of the current skill function are parsed from the skill function operation chain generated by the visual language model; The function operation name and operation parameters of the current skill function are passed to the interface of the utility function corresponding to the current skill function, so as to call and execute the utility function corresponding to the current skill function.

5. The method according to claim 3, characterized in that, After determining whether the execution result of the current skill function meets the preset invocation conditions, the method further includes: If the execution result of the current skill function does not meet the preset calling conditions, then the execution result of the current skill function is determined as the final execution result.

6. The method according to claim 1, characterized in that, The predefined skill function set includes: semantic localization skill functions, key point localization skill functions, optical character recognition skill functions, and readout recognition skill functions; executing the skill functions in the skill function operation chain in the order of the skill function operation chain includes: When the semantic localization skill function is executed in the order of the skill function operation chain, the instrument location region is located from the target instrument image, and the instrument bounding box information is returned; When the key point positioning skill function is executed in the order of the skill function operation chain, the position of the pointer center point and the position of the pointer endpoint are determined from the image area of ​​the instrument bounding box; When the optical character recognition skill function is executed in the order of the skill function operation chain, the scale value text and the corresponding scale position are identified from the image area of ​​the instrument boundary box; When the reading recognition skill function is executed in the order of the skill function operation chain, the current reading value of the target instrument image is determined based on the pointer center point position, the pointer endpoint position, the scale value text, and the corresponding scale position.

7. The method according to claim 6, characterized in that, After identifying the scale value text and the corresponding scale position from the image area of ​​the instrument boundary frame, the method further includes: Determine whether the scale value text meets the preset verification conditions. The preset verification conditions include: the scale value text contains preset characters; or, the ratio between the number of preset characters in the scale value text and the total number of characters in the scale value text is greater than a preset threshold; or, the scale value trend in the scale value text does not show a monotonically increasing trend or does not show a monotonically decreasing trend. If so, the PaddleOCR tool function corresponding to the optical character recognition skill function is called. The PaddleOCR tool function is used to identify the scale value text and the corresponding scale position from the image area of ​​the instrument bounding box.

8. The method according to any one of claims 1-7, characterized in that, Before inputting the target instrument image and the target question text into the trained visual language model, the method further includes obtaining a multimodal training dataset for general instruments. The multimodal training dataset includes: sample instrument images, sample question texts, sample answer texts corresponding to the sample question texts, and sample skill function operation chains. The visual language model is fine-tuned using the sample instrument images and sample question texts as training data, and the sample answer texts and sample skill function operation chains as training labels. The sample skill function operation chain is added to the multimodal training dataset after semantic consistency verification based on the visual language model and correction using a general language model. The semantic consistency verification ensures that the description of each skill function operation in the sample skill function operation chain matches the actual collected visual information. Correcting the sample skill function operation chain using a general language model includes: if a missing key step or an abnormal step order is identified in the sample skill function operation chain, the sample skill function operation chain is corrected using the general language model to obtain a corrected skill function operation chain, and the corrected skill function operation chain is added to the multimodal training dataset. The corrected skill function operation chain is a skill function operation chain that conforms to the semantics of the sample question text.

9. A general-purpose instrument image processing device, characterized in that, include: The image and question text acquisition module is used to acquire the target instrument image and the target question text; The image-question-text input module is used to input the target instrument image and the target question text into the trained visual language model, so as to match multiple candidate skill functions in a predefined skill function set through the visual language model; The skill function operation chain generation module is used to construct the skill function operation chain corresponding to the multiple candidate skill functions based on the input-output dependency relationship of each candidate skill function through the visual language model; The skill function operation chain execution module is used to execute the skill functions in the skill function operation chain in the order of the skill function operation chain to obtain the final answer corresponding to the target question text.

10. An electronic device, characterized in that, include: A processor and a memory, the memory storing machine-readable instructions executable by the processor, the machine-readable instructions being executed by the processor to perform the method of any one of claims 1 to 8.

Images