Asynchronous intelligent reporting system and method based on multi-modal large model and intelligent agent

Abstract

The application provides an asynchronous intelligent reporting system and method based on a multi-modal large model and an intelligent agent. The system comprises: a multi-modal acquisition terminal, which is used to obtain original reporting data in response to a trigger event and release the terminal after asynchronous sending; a message queue service module, which is used to receive and cache the original reporting data; a cognitive extraction module, which is used to call a multi-modal large language model to perform semantic understanding and structured information extraction on the original reporting data, and generate initial structured data; an Agent reasoning module, which is used to map text descriptions in the initial structured data to unique entity identifiers of an ERP system through an intelligent Agent, generate a mapping result and a reasoning link record; an adversarial verification module, which is used to perform adversarial auditing on the mapping result and the reasoning link record through a heterogeneous independent auditing large language model, and generate an auditing report; and an application interaction module, which is used to execute a processing strategy according to the auditing report, and feed back artificial correction results to a memory.

Description

Technical Field

[0001] This invention relates to the fields of industrial internet and intelligent manufacturing technology, and in particular to an asynchronous intelligent work reporting system and method based on a multimodal large model and intelligent agent. Background Technology

[0002] In discrete manufacturing and process manufacturing production sites, production reporting is a crucial data collection link connecting the physical production process with the enterprise digital system (ERP / MES). Traditional reporting methods mainly include: manual entry on PCs, importing from Excel spreadsheets, and scanning QR codes / barcodes with handheld PDAs or mobile phones. However, these methods all face serious limitations in practical industrial scenarios.

[0003] 1. The stringent requirements of B2B industrial systems for entity parsing render traditional OCR fundamentally ineffective.

[0004] Traditional OCR technologies (including modern deep learning OCR, such as PaddleOCR) essentially output text transcription, converting characters in an image into numeric text strings. While this output is acceptable in consumer (C-end) scenarios, it has fundamental flaws when integrated with industrial systems (B-end). Enterprise ERP / MES systems require all business operations to be based on unique internal identifiers. For example, work order numbers must be precise 12-digit business codes, operators must be employee IDs, and processes must be standard process codes. There is a fundamental semantic gap between the text recognized by OCR (e.g., Zhang San, polishing, W20231012) and the system entity IDs (e.g., employee ID 202301, process code OP30-surface treatment, work order number WO-2023-10-12-001). Traditional OCR will immediately crash and report an error if a character is misrecognized (e.g., misreading the number 0 as the English letter O) or fails to match precisely with the database, completely failing to meet the data rigor requirements of B-end systems. This problem is particularly prominent in handwritten document scenarios: individual differences in handwriting, degree of illegibility, and colloquial descriptions (such as using polishing instead of the standard procedure "OP30-Surface Treatment") all cause the error rate of the rule engine solution to rise sharply.

[0005] 2. Synchronous interaction mode causes systematic interference to production cycle time.

[0006] The most mainstream digital work reporting solution currently is QR code reporting: After completing a process, workers must actively interrupt production, pick up their mobile phones or workstation terminals, scan the QR code on the transfer document, select the corresponding process (or have it automatically selected by the system), manually enter the quantity of good products, and finally submit the work report. This complete closed loop of stop → scan → select → input → submit is essentially a synchronous interaction mode. The system requires workers to forcibly insert a data entry action unrelated to production itself into the production process, and they must wait for the system's response and confirmation before returning to production. This forced interruption mechanism not only damages production efficiency by the time loss of a single operation, but also systematically disrupts the continuity of workers' operations and their flow state. In high-frequency, short-cycle production lines (such as electronic assembly and precision parts processing), the operation time of each process may only be tens of seconds to several minutes, while the complete interaction of each work report can take more than 30 seconds. The cumulative effect of such interruptions is extremely significant. The existing solution completely shifts the burden of data collection, a management requirement, onto frontline workers, resulting in a fundamental conflict between the management's need for timely and accurate production data and the workers' need for uninterrupted continuous operation.

[0007] 3. Existing digital solutions face difficulties in practical implementation in industrial settings.

[0008] The physical environment and management systems in industrial settings pose multiple obstacles to the implementation of digital work reporting solutions. On the one hand, the prevalent heavy oil stains, metal dust, and high vibrations in workshop environments significantly reduce the usability and durability of touchscreen terminals. Frontline workers have low acceptance of frequent operation of electronic devices, and training costs are high. Some confidential workshops (such as aerospace) even prohibit the carrying of mobile electronic devices such as mobile phones, making it impossible to deploy mobile scanning solutions in such scenarios. On the other hand, many factories have long-established paper-based document circulation systems. Abruptly demanding the complete abolition of paper documents and a shift to purely digital operations often encounters strong resistance from frontline workers, and cases of implementation failure are common.

[0009] 4. Data entry delays and disconnect from management decision-making.

[0010] In many small and medium-sized manufacturing enterprises, even though some processes have deployed QR code-based work reporting systems, paper-based circulation documents are still widely used. A large amount of production data still relies on manual collection and secondary entry into the system after each shift, resulting in a serious lag of several hours or even days in production data. This lag prevents management from scheduling production and responding to anomalies based on real-time data, and the grasp of production progress often relies on verbal communication and experience-based judgment. In addition, manual secondary entry is prone to errors due to factors such as fatigue and difficulty in recognizing handwriting, further reducing data quality and creating a vicious cycle of slow collection → entry errors → delayed decision-making. Summary of the Invention

[0011] The purpose of this invention is to provide an asynchronous intelligent reporting system and method based on a multimodal large model and intelligent agent. Specifically, it proposes an asynchronous reporting architecture that decouples the acquisition and processing time. Through multimodal acquisition, semantic extraction from a large language model, agent intelligent entity mapping with tool invocation and memory mechanisms, and collaboration of heterogeneous dual-model adversarial verification, it aims to systematically solve the above-mentioned problems in the prior art.

[0012] This invention provides an asynchronous intelligent work reporting system based on a multimodal large model and intelligent agents, comprising:

[0013] A multimodal acquisition terminal is deployed at the production workstation to acquire raw work report data in response to a trigger event, encapsulate the raw work report data and send it asynchronously to the message queue service module. After sending, there is no need to wait for the processing result, and the multimodal acquisition terminal is released to continue subsequent production operations.

[0014] The message queue service module is connected to the multimodal acquisition terminal and is used to receive and cache the original work report data;

[0015] The cognitive extraction module, connected to the message queue service module, is used to pull the original work report data from the message queue service module and call the multimodal large language model to perform semantic understanding and structured information extraction on the original work report data to generate initial structured data.

[0016] The Agent reasoning module, connected to the cognitive extraction module, is used to receive the initial structured data, map the text descriptions in the initial structured data to unique entity identifiers of the ERP system through the intelligent Agent, and generate mapping results and reasoning link records.

[0017] The adversarial verification module, connected to the Agent inference module and the message queue service module, is used to perform adversarial verification on the mapping results and the inference link records through a heterogeneous independent auditing large language model, and generate an audit report;

[0018] The application interaction module is connected to the adversarial verification module and the agent inference module. It is used to execute the corresponding processing strategy according to the audit report and to feed back the manual correction results to the memory of the agent inference module.

[0019] This invention provides an asynchronous intelligent reporting method based on a multimodal large model and intelligent agents, comprising:

[0020] The multimodal acquisition terminal deployed at the production workstation responds to the trigger event and acquires the original work report data. The original work report data is then encapsulated and sent asynchronously to the message queue service module. After the sending is completed, the multimodal acquisition terminal is released.

[0021] The original work report data is received and cached through the message queue service module;

[0022] The cognitive extraction module retrieves the original work report data from the message queue service module and calls the multimodal large language model to perform semantic understanding and structured information extraction on the original work report data to generate initial structured data.

[0023] The Agent inference module receives the initial structured data and uses the intelligent Agent to map the text descriptions in the initial structured data to unique entity identifiers of the ERP system, generating mapping results and inference link records.

[0024] The adversarial verification module uses a heterogeneous, independent auditing language model to perform adversarial auditing on the mapping results and the inference link records, and generates an audit report.

[0025] The application interaction module executes the corresponding processing strategy based on the audit report and feeds back the manual correction results to the memory of the Agent inference module.

[0026] This invention also provides an electronic device, including: a memory, a processor, and a computer program stored in the memory and executable on the processor. When the computer program is executed by the processor, it implements the steps of the above-described asynchronous intelligent reporting method based on a multimodal large model and intelligent agent.

[0027] This invention also provides a computer-readable storage medium storing an information transmission implementation program, which, when executed by a processor, implements the steps of the above-described asynchronous intelligent reporting method based on a multimodal large model and intelligent agent.

[0028] The following beneficial effects can be achieved by adopting the embodiments of the present invention: (1) the time decoupling between the collection end and the processing end is realized through the message queue. After the operator completes the provision of the original information, he / she can return to production without waiting for the system to process and confirm, thus eliminating the interference of the traditional synchronous interaction mode on the production cycle; (2) by introducing an Agent with the ability to call tools, retrieve memories and correct itself, the colloquial and ambiguous handwritten text is accurately mapped to the unique entity identifier of the ERP system, fundamentally solving the technical problem that traditional OCR cannot meet the entity parsing requirements of B-end; (3) by introducing an independent audit model with a different source or architecture from the extraction model for adversarial verification, the same source deviation is effectively avoided, the accuracy and interpretability of the system output are improved, and the accuracy is continuously improved through the memory closed-loop mechanism to accumulate experience and self-evolve. Attached Figure Description

[0029] To more clearly illustrate the technical solutions in one or more embodiments of this specification or in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in this specification. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0030] Figure 1 This is a schematic diagram of an asynchronous intelligent work reporting system based on a multimodal large model and intelligent agent according to an embodiment of the present invention;

[0031] Figure 2 This is a schematic diagram of the overall system architecture according to an embodiment of the present invention;

[0032] Figure 3 This is a flowchart of the asynchronous intelligent work reporting method based on multimodal large model and intelligent agent according to an embodiment of the present invention;

[0033] Figure 4(a) shows a partial asynchronous interaction timing process according to an embodiment of the present invention;

[0034] Figure 4(b) shows another part of the asynchronous interaction timing process of an embodiment of the present invention;

[0035] Figure 5(a) shows a partial Agent Loop intelligent search and entity mapping process according to an embodiment of the present invention;

[0036] Figure 5(b) illustrates another part of the Agent Loop intelligent search and entity mapping process in this embodiment of the invention;

[0037] Figure 6 This is a schematic diagram of the dual-mode verification logic according to an embodiment of the present invention. Detailed Implementation

[0038] To enable those skilled in the art to better understand the technical solutions in one or more embodiments of this specification, the technical solutions in one or more embodiments of this specification will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this specification, and not all of the embodiments. Based on one or more embodiments of this specification, all other embodiments obtained by those skilled in the art without creative effort should fall within the protection scope of this document.

[0039] System Implementation Examples

[0040] According to embodiments of the present invention, an asynchronous intelligent work reporting system based on a multimodal large model and intelligent agents is provided. Figure 1 This is a schematic diagram of an asynchronous intelligent work reporting system based on a multimodal large model and intelligent agent, according to an embodiment of the present invention. Figure 1 As shown, the asynchronous intelligent reporting system based on a multimodal large model and intelligent agent according to an embodiment of the present invention specifically includes:

[0041] A multimodal data acquisition terminal 10, deployed at the production workstation, is used to acquire raw work report data in response to a trigger event, encapsulate the raw work report data, and asynchronously send it to the message queue service module. After sending, without waiting for processing results, the multimodal data acquisition terminal is released to continue subsequent production operations; specifically including:

[0042] An image acquisition unit includes a camera and a trigger controller, used to respond to a trigger event and control the camera to acquire image data. The trigger controller includes at least one of a video stream detection subunit, a physical button, and a manual photo capture interface. The video stream detection subunit receives the video stream acquired by the camera and performs differential calculations on consecutive frames in the video stream. When a new stationary object is detected, a rectangular outline is detected in the object's region. If the outline size matches a preset document size and the stabilization time exceeds a preset threshold, automatic photo capture is triggered. The physical button is an industrial-grade button located at the workstation, used to trigger photo capture in response to a press operation. The manual photo capture interface communicates with a mobile device to receive images captured and uploaded by the mobile device.

[0043] The voice acquisition unit includes a microphone and a voice activity detection subunit, which is used to acquire workers' oral work reports through the microphone and perform voice start and end point recognition, environmental noise suppression and echo cancellation processing on the oral work reports through the voice activity detection subunit to generate voice data;

[0044] A text input unit, including an industrial-grade terminal device or a mobile input interface, is used to receive direct text input through the industrial-grade terminal device or the mobile input interface and generate text data.

[0045] An environment adaptive unit, connected to the image acquisition unit and the voice acquisition unit, is used to detect environmental conditions and adjust the operating parameters of the image acquisition unit and the voice acquisition unit according to the detection results.

[0046] Message queue service module 11 is connected to the multimodal acquisition terminal and is used to receive and cache the original work reporting data;

[0047] The cognitive extraction module 12, connected to the message queue service module, is used to retrieve the original work report data from the message queue service module and call a multimodal large language model to perform semantic understanding and structured information extraction on the original work report data to generate initial structured data; specifically including:

[0048] A multimodal data parser is used to pull raw work report data from the message queue service module, identify the type of the raw work report data, call the corresponding multimodal large language model interface according to the identification result, and output the raw work report data and its type identification result to the prompt word construction unit.

[0049] The prompt word construction unit, connected to the multimodal data parser, is used to receive the original work report data and its type identification result, generate prompt words for extracting structured information based on the type identification result and the business entity field to be extracted, and pass the prompt words and the original work report data to the invoked multimodal large language model;

[0050] The field extraction unit is used to receive the structured output results returned by the multimodal large language model, parse the structured output results and extract at least one business entity field and the first confidence level of each field, and output the extraction results;

[0051] The confidence assessment unit, connected to the field extraction unit, is used to receive the extraction results and comprehensively calculate the confidence score of each field based on the self-assessment results of the multimodal large language model, the field format standardization, and the completeness of the extraction results. The business entity fields extracted by the field extraction unit are merged with the confidence scores to generate initial structured data and output to the Agent inference module.

[0052] Agent reasoning module 13, connected to the cognitive extraction module, is used to receive the initial structured data, map the text descriptions in the initial structured data to unique entity identifiers of the ERP system through the intelligent agent, and generate mapping results and reasoning link records; specifically including:

[0053] The precise matching unit, connected to the cognitive extraction module, is used to receive the initial structured data, call the precise query interface of the ERP system to match the text description in the initial structured data with the entity identifier in the ERP system. If the match is successful, the mapping result is directly output to the inference output unit, and the Agent inference process ends. If the match fails, the field that failed to match and the corresponding text description are output to the memory retrieval unit and the tool calling unit.

[0054] The memory retrieval unit, connected to the precise matching unit, is used to receive the field that failed to match and its corresponding text description, and retrieve the historical records related to the text description from the memory bank before the tool is launched, and output the retrieval results to the inference engine unit.

[0055] The tool calling unit, connected to the precise matching unit, is used to receive the fields that failed to match and their corresponding text descriptions, call the corresponding external query tool according to the type of the field that failed to match, obtain the candidate entity list from the ERP system, and output the candidate entity list to the inference engine unit;

[0056] The inference engine unit, connected to the memory retrieval unit and the tool invocation unit, is used to perform comprehensive inference based on the retrieval results and the candidate entity list, determine the final entity identifier, generate mapping results and inference link records, and output the mapping results and inference link records to the self-correction unit.

[0057] A self-correction unit, connected to the inference engine unit and the tool invocation unit, is used to receive the mapping results and inference link records. When the confidence level of the mapping result is lower than a preset threshold, it triggers the tool invocation unit to adjust the invocation strategy and re-execute the tool invocation, or triggers the inference engine unit to re-perform comprehensive inference. The adjustment of the invocation strategy includes at least one of expanding the search parameter range, switching from precise query to fuzzy query, or switching to a backup query tool.

[0058] The inference output unit, connected to the inference engine unit and the self-correction unit, is used to receive the final mapping result and inference link record, and output them to the adversarial verification module.

[0059] The adversarial verification module 14, connected to the Agent inference module and the message queue service module, is used to perform adversarial verification on the mapping results and the inference link records through a heterogeneous independent auditing language model, and generate an audit report; specifically including:

[0060] The data cross-validation unit is used to receive the final mapping result and inference link record, obtain the corresponding original work report data from the message queue service module, and use a heterogeneous independent auditing language model to perform independent cross-validation on the final mapping result and the original work report data to generate a first verification result.

[0061] The reasoning link review unit is used to review the logical rationality of the reasoning steps in the final reasoning link record and generate a second verification result.

[0062] The consistency verification unit is used to verify the consistency of business logic among the work order, process, and personnel in the final mapping result and generate a third verification result.

[0063] An adversarial audit engine, connected to the data cross-validation unit, inference link review unit, and consistency verification unit, is used to receive the first verification result, the second verification result, and the third verification result, generate an audit report by integrating the verification results, and transmit the audit report to the application interaction module.

[0064] The multimodal large language model called by the cognitive extraction module 12 and the independent auditing large language model called by the adversarial verification module 14 are heterogeneous models from different sources or with different architectures. For example, the cognitive extraction module 12 uses Tongyi Qianwen-VL, while the adversarial verification module 14 uses GPT-4V or DeepSeek-VL to avoid homology bias caused by a single model.

[0065] Application interaction module 15, connected to the adversarial verification module and the Agent inference module, is used to execute corresponding processing strategies based on the review report and feed back the manual correction results to the memory of the Agent inference module; specifically including:

[0066] An automatic posting unit is used to receive the audit report. When the audit report meets the preset automatic posting conditions, the unit writes the mapping result into the ERP system through the ERP system's application programming interface and generates a posting record.

[0067] An exception handling unit is used to receive the audit report. When the audit report does not meet the preset automatic posting conditions, it pushes the mapping result, the audit report and the corresponding original work report data to the exception handling queue and displays the exception items to be processed on the management interface.

[0068] The memory feedback unit, connected to the exception handling unit, is used to receive the manual correction results of the exception items to be processed by the manager, and write the manual correction results into the memory bank of the Agent inference module;

[0069] The report statistics unit, connected to the automatic posting unit and the exception handling unit, is used to count the posting records and the exception handling records of the exception handling unit, and generate system operation indicator reports.

[0070] The system further includes:

[0071] The priority scheduling module, connected to the message queue service module and the cognitive extraction module, is used to obtain tasks to be processed from the message queue service module, prioritize the tasks to be processed according to the urgency of the work order and the first confidence level of each field in the initial structured data, and push the tasks to the cognitive extraction module in priority order.

[0072] A progressive feedback module, connected to the cognitive extraction module or the agent inference module, is used to generate a partial confirmation status notification when the cognitive extraction module completes partial field extraction or the agent inference module completes partial field mapping, and to feed back the partial confirmation status notification to the multimodal acquisition terminal.

[0073] The following describes in detail the above-mentioned technical solutions of the present invention with reference to the specific circumstances of the asynchronous intelligent reporting system based on multimodal large model and intelligent agent in the embodiments of the present invention.

[0074] This invention proposes an asynchronous intelligent reporting system based on a multimodal large model and agent proxy, such as... Figure 2 As shown, it includes:

[0075] The multimodal data acquisition terminal, deployed at the production workstation, is used to acquire raw work report data in response to triggered events, encapsulate the raw work report data and send it to the message queue. After the acquisition is completed, it returns immediately without on-site confirmation of the acquisition results. The multimodal data acquisition terminal supports at least one acquisition mode, including an image acquisition unit (such as automatically taking pictures of paper documents through a fixed camera), a voice acquisition unit (such as workers dictating their work report information), and a text input unit. The acquisition method can be flexibly selected according to the on-site conditions.

[0076] The message queue service module is used to receive and cache raw work reporting data from the multimodal acquisition terminal, thereby achieving time decoupling between the acquisition end and the processing end.

[0077] The cognitive extraction module is used to pull raw reporting data from the message queue, call a multimodal large language model to perform semantic understanding and structured information extraction on the raw reporting data, and generate initial structured data; wherein the initial structured data contains at least one business entity field and its corresponding text description and first confidence level;

[0078] The Agent reasoning module receives initial structured data and, through an intelligent agent with tool invocation, memory retrieval, and self-correction capabilities, maps the text descriptions in the initial structured data to unique entity identifiers in the ERP system, generating mapping results and reasoning chain records. When the extracted text cannot precisely match the business entities in the ERP system, the Agent autonomously constructs a tool invocation chain and uses fuzzy search, semantic vector matching, and contextual exclusive inference to map non-standard handwritten information to system entity IDs. The Agent has a built-in long-term memory mechanism that continuously accumulates historical work report records, manual review and correction records, and contextual information (current shift, time period, work order, etc.). During the reasoning process, it retrieves relevant memories to assist decision-making. For example, when encountering fuzzy input, it can prioritize inferring the most likely work order and personnel based on the recent work report history of that workstation. Here, "Agent" refers to an intelligent agent program with autonomous decision-making, tool invocation, memory retrieval, and self-correction capabilities.

[0079] The adversarial verification module is used to perform adversarial verification on the mapping results and inference link records through heterogeneous independent auditing large language models, and generate an audit report including confidence scores and risk descriptions.

[0080] The application interaction module is used to execute corresponding processing strategies based on the audit report and feed back the manual correction results to the memory of the Agent inference module.

[0081] Specifically, the multimodal acquisition terminal in this embodiment of the invention includes:

[0082] The image acquisition unit includes a fixed camera and a trigger controller. The trigger controller includes at least one of a video stream detection subunit, a physical button, and a manual photo capture interface. The video stream detection subunit is used to automatically identify document placement behavior and trigger image capture via frame difference and rectangular contour detection. Specifically, it performs differential calculation on consecutive frames in the video stream. When a new stationary object is detected, it performs rectangular contour detection on the object's area. If the contour size matches the preset document size and the stabilization time exceeds a preset threshold (e.g., 0.5 seconds), it automatically triggers image capture. The physical button is an industrial-grade waterproof and dustproof button located at the workstation. Pressing the button after completing paper recording triggers image capture. The manual photo capture interface is used to communicate with mobile devices and receive images captured and uploaded by the mobile devices.

[0083] The voice acquisition unit includes a microphone and a voice activity detection subunit, which is used to acquire workers' spoken work information and perform endpoint detection. The voice activity detection subunit adopts a dual-threshold detection algorithm based on energy threshold and zero-crossing rate to automatically identify the start and end points of the voice, and performs environmental noise suppression and echo cancellation processing on the acquired voice.

[0084] The text input unit includes a simple terminal or mobile input interface for receiving direct text input. The simple terminal is an industrial-grade terminal device with a physical keyboard or touch screen.

[0085] An environment adaptive unit is used to automatically adjust the exposure time, gain, and white balance parameters of the image acquisition unit according to ambient lighting conditions, and to automatically adjust the noise reduction parameters and gain coefficient of the voice acquisition unit according to ambient noise levels.

[0086] The message queue service module of this embodiment of the invention specifically includes:

[0087] The message receiving interface is used to receive raw work reports sent by the multimodal acquisition terminal and to verify the data format.

[0088] The priority queue manager is used to sort messages based on the urgency of work orders and preset priority rules. The urgency of work orders is obtained synchronously from the ERP system, and the priority rules include prioritizing urgent work orders, prioritizing high-confidence estimated tasks, and first-in-first-out (FIFO) rules.

[0089] Message persistent storage is used to persistently store messages to disk to prevent message loss;

[0090] The message dispatcher is used to push messages to the cognitive extraction module according to priority, and supports message retry mechanism and dead letter queue processing.

[0091] The cognitive extraction module of this invention specifically includes:

[0092] The multimodal data parser is used to identify the type of the original reporting data. If it is image data, it calls the visual understanding interface of the multimodal large language model. If it is voice data, it first calls the voice recognition interface to convert it into text and then calls the text understanding interface. If it is text data, it directly calls the text understanding interface.

[0093] The prompt word engineering unit is used to construct prompt word templates for structured information extraction. The prompt word template includes field definitions, output format requirements, and domain knowledge constraints. The domain knowledge constraints include common formats for work order numbers, common ways of expressing process names, and common patterns for personnel names.

[0094] The field extractor is used to parse the output of the multimodal large language model and extract at least one business entity field from the following: work order number text, process description text, operator name text, completed quantity, reporting time, equipment number, and material batch number, and record the first confidence level of each field.

[0095] The confidence evaluator is used to evaluate the confidence of each extracted field. The confidence evaluation is a weighted calculation based on the self-rated confidence of the multimodal large language model output, the format standardization of the extracted results, the completeness of the fields, and the consistency of the context.

[0096] For example, let the confidence level of field f be... It can be calculated by weighting the following dimensions:

[0097] ;

[0098] In the formula, The model's self-assessment confidence level. For format specification compliance, For field integrity, For context consistency, For each dimension's weight coefficients, satisfying As an optional example, the initial default values ​​can be set to 0.4, 0.2, 0.2, and 0.2 respectively.

[0099] The Agent inference module of this invention specifically includes:

[0100] The exact match unit is used to directly call the exact query interface of the ERP system to attempt a match using the text description in the initial structured data. If the work order number, process code, and employee number are all exactly matched, the mapping result is directly output without triggering subsequent search logic.

[0101] A memory retrieval unit, connected to a memory bank, is used to retrieve historical information to aid decision-making when an exact match fails; the memory bank includes:

[0102] A. Historical work report memory sub-database, used to store historical work report records for workstations, including timestamps, work order numbers, process codes, operator employee numbers, completed quantities, and work report image feature vectors;

[0103] B. Correction record memory sub-database, used to store manual review and correction records, including the original fuzzy input text, the corrected entity identifier, correction time, correction personnel identifier, and correction context;

[0104] C. Contextual Memory Sub-library, used to store the current shift's scheduling information, workstation supervisor list, work-in-process information, production plan, and equipment status.

[0105] The memory retrieval unit uses a combination of vector similarity retrieval and keyword retrieval. Based on the currently input text description and workstation identifier, it retrieves the K most similar historical records from the memory bank and calculates the similarity score of each record as an auxiliary inference weight.

[0106] The tool invocation unit is used to autonomously plan and execute tool invocation chains according to priority based on the reasons for matching failures of fields of each business entity. The tool invocation chain includes:

[0107] A. Work order query tool, used to perform fuzzy search based on at least one of the identified work order number fragments, product model, batch number, and production date, and return a list of candidate work orders and the matching score of each candidate work order;

[0108] B. Process query tool, which calculates the similarity between the identified process description text and the standard process name through semantic vector matching, and returns the standard process code with the highest similarity and the similarity score;

[0109] C. Personnel query tool, which is used to filter based on the identified personnel name or employee number fragment, combined with the current team information and work station supervisor list in the contextual memory sub-bank, to make an exclusive inference among multiple candidates with the same name and return a unique employee number;

[0110] D. Process route query tool, used to obtain the standard process route of candidate work orders and constrain the process matching results within the scope of the process route.

[0111] The inference engine unit performs comprehensive inference based on memory retrieval results and tool call results. It adopts a confidence-weighted voting mechanism to integrate candidate results from various sources. The calculation formula is as follows (as an optional example):

[0112] ;

[0113] In the formula, For the final similarity, For memory similarity, Match scores to the tool. For context-related, The preset weighting coefficients satisfy... ;

[0114] A self-correction unit is used to perform a self-correction operation when the final confidence level output by the inference engine is lower than a preset threshold. This self-correction operation includes:

[0115] A. Backtrack to the previous reasoning step and check for any missing tool call paths;

[0116] B. Expand the range of search parameters to increase the tolerance of fuzzy matching, for example, by adjusting the exact matching mode to prefix matching or inclusion matching;

[0117] C. Switch tool invocation strategies, such as switching from API queries to vector database semantic retrieval, or from precise field queries to full-text search;

[0118] D. Lower the matching threshold to allow output of results with lower confidence levels and mark them as pending review.

[0119] The inference output unit is used to output the mapping results and a complete inference chain record. The inference chain record includes the input parameters, return results, intermediate decision-making processes, and final confidence scores for each tool call.

[0120] The anti-countermeasures module in this embodiment of the invention specifically includes:

[0121] The data cross-validation unit is used to independently analyze the original collected data and verify its consistency with the mapping results. For image data, it calls an independent multimodal large language model to re-identify key visual information and compares it with the Agent's mapping results. For speech data, it calls an independent speech recognition model to re-transcribe the data and compares it with the transcribed text used by the Agent.

[0122] The reasoning chain review unit is used to review the logical rationality of the Agent's reasoning steps and whether there is excessive inference. Specifically, it includes checking whether the reasoning steps are skipped, whether the intermediate conclusions are supported by sufficient evidence, and whether there is circular dependency.

[0123] The consistency verification unit is used to verify the consistency of business logic among work orders, processes, and personnel in the mapping results. Specifically, it includes: verifying whether the process belongs to the standard process route of the work order, whether the personnel have the authority to execute the process, whether the current status of the work order allows reporting work, and whether the number of reported work is within a reasonable range.

[0124] An adversarial auditing engine is used to synthesize the validation results of each unit and generate an audit report; the audit report includes:

[0125] A. Confidence score, a value ranging from 0 to 100%, is calculated by weighting the pass rate and severity of each verification unit;

[0126] B. Natural Language Risk Statement, describing specific risk points and recommended issues to be addressed;

[0127] C. Review suggestions, including three types: automatic posting, triggering manual review, and rejection and return;

[0128] D. List of discrepancies: This section lists the fields in the Agent output that are inconsistent with the independent audit results, along with details of the discrepancies.

[0129] The application interaction module of this invention specifically includes:

[0130] The automatic posting unit is used to write the mapping result through the ERP system's API interface when the confidence score of the audit report exceeds a preset threshold (such as 85%) and the audit suggestion is automatic posting, thus completing the automatic entry of work report data.

[0131] The exception handling unit is used to push the mapping results, audit report, and original collected data to the exception handling queue and display them on the management interface when the confidence score of the audit report falls below a preset threshold or the audit suggestion is to trigger manual review.

[0132] A. High-risk field highlighting area, used to highlight fields with low confidence levels;

[0133] B. Risk Description Display Area, used to display risk descriptions in natural language;

[0134] C. Candidate Correction Suggestion Area, used to display other candidate entities recommended by the Agent and their confidence levels;

[0135] D. Correction operation area, used to receive field correction operations from administrators.

[0136] The memory feedback unit is used to write the correction results of the administrator into the memory of the Agent inference module. The correction results include the original fuzzy input text, the corrected entity identifier, the correction context, and the timestamp. The memory feedback unit adopts an incremental writing method, does not overwrite existing historical records, and periodically deduplicates and compresses the memory.

[0137] The report statistics unit is used to collect statistics on the system's operational metrics, including the total number of reported tasks, automatic posting rate, manual review rate, average processing time, recognition accuracy of each field, and confidence distribution of Agent inference.

[0138] Preferably, the system in this embodiment of the invention further includes:

[0139] The priority scheduling module connects the message queue service module and the cognitive extraction module. It is used to dynamically adjust the processing order of tasks based on the urgency of the work order and the first confidence level. This module adopts a multi-level queue scheduling algorithm to divide tasks into high-priority queues, medium-priority queues and low-priority queues. Tasks in the high-priority queue are pulled and processed by the cognitive extraction module first.

[0140] The progressive feedback module is used to send partial confirmation status notifications to the multimodal acquisition terminal after the cognitive extraction module or agent inference module has completed the processing of some fields. This includes a list of confirmed fields, a list of fields to be confirmed, and the estimated completion time, so that operators or managers can understand the processing progress in real time without waiting for all fields to be processed.

[0141] It is worth noting that the multimodal large language model called in the cognitive extraction module of this embodiment and the independent auditing large language model in the adversarial verification module are heterogeneous models from different sources or with different architectures to avoid homogeneity bias. Specifically, the cognitive extraction module uses a first multimodal large language model, and the adversarial verification module uses a second multimodal large language model from a different source than the first multimodal large language model, or a third multimodal large language model based on a different technical architecture.

[0142] Preferably, the system also supports multiple deployment modes, including:

[0143] In the cloud deployment mode, the cognitive extraction module, agent inference module, adversarial verification module, and application interaction module are deployed on public or private cloud servers, and the multimodal acquisition terminal communicates with the cloud via the Internet or dedicated line.

[0144] The edge deployment mode deploys at least some of the cognitive extraction module, agent inference module, adversarial verification module and application interaction module at the workstation edge computing node to achieve local data processing and low-latency response.

[0145] Hybrid deployment modes, such as deploying the cognitive extraction module on edge nodes to achieve rapid structured extraction, and deploying the agent inference module and adversarial verification module in the cloud to achieve complex inference and verification, can utilize cloud computing power to process complex tasks while ensuring real-time performance.

[0146] It should also be noted that the ERP system in the embodiments of the present invention should be interpreted broadly, including enterprise resource planning systems, manufacturing execution systems, and other enterprise-level business management systems with similar functions.

[0147] In summary, the asynchronous intelligent work reporting system based on a multimodal large model and agent provided in this invention can be deployed and applied in the production sites of various discrete manufacturing and process manufacturing enterprises. It is particularly suitable for manufacturing scenarios with paper-based document circulation, handwritten records, and tight worker operation rhythms, such as electronic assembly, precision parts machining, and automotive parts manufacturing. The system is compatible with existing work habits, reducing implementation resistance. At the same time, it eliminates interference with production rhythm through asynchronous architecture and solves the entity parsing problem through agent intelligent mapping, demonstrating good industrial practical value and application prospects.

[0148] Method Implementation Examples

[0149] According to embodiments of the present invention, an asynchronous intelligent reporting method based on a multimodal large model and intelligent agents is provided. Figure 3 This is a flowchart of the asynchronous intelligent work reporting method based on a multimodal large model and intelligent agent according to an embodiment of the present invention, as follows: Figure 3As shown, the asynchronous intelligent reporting method based on a multimodal large model and intelligent agent according to an embodiment of the present invention specifically includes:

[0150] Step S301: The multimodal acquisition terminal deployed at the production workstation responds to the trigger event and acquires the original work report data. The original work report data is encapsulated and sent asynchronously to the message queue service module. After the sending is completed, the multimodal acquisition terminal is released.

[0151] Step S302: Receive and cache the original work report data through the message queue service module;

[0152] Step S303: The original work report data is retrieved from the message queue service module through the cognitive extraction module, and the multimodal large language model is called to perform semantic understanding and structured information extraction on the original work report data to generate initial structured data.

[0153] Step S304: Receive the initial structured data through the Agent inference module, and use the intelligent Agent to map the text descriptions in the initial structured data to unique entity identifiers of the ERP system, generating mapping results and inference link records;

[0154] Step S305: The adversarial verification module uses a heterogeneous independent auditing language model to perform adversarial auditing on the mapping results and the inference link records, and generates an audit report.

[0155] Step S306: The application interaction module executes the corresponding processing strategy based on the audit report and feeds back the manual correction result to the memory of the Agent inference module.

[0156] The following describes in detail the above-mentioned technical solution of the present invention with reference to the specific details of the asynchronous intelligent reporting method based on multimodal large model and intelligent agent in the embodiments of the present invention, as shown in Figures 4(a) and (b).

[0157] Step 1: Multimodal Information Acquisition (Acquisition Layer)

[0158] This system supports multiple information acquisition modes, which can be flexibly selected or combined according to different industrial site environmental conditions and management requirements:

[0159] 1. Image Acquisition Mode

[0160] Automatic data collection: Fixed cameras are deployed at workstations to automatically identify document placement behavior and trigger shooting through video stream detection (such as frame difference method + rectangular contour detection), without requiring any operation from workers;

[0161] Button trigger: After completing the paper record, the worker presses the physical button on the workstation to trigger the shooting, which is simple and direct to operate;

[0162] Manual photography: Workers use their mobile phones or other mobile devices to take photos and upload them, which is suitable for scenarios where there is no fixed equipment.

[0163] 2. Voice acquisition mode: Workers verbally report work information (such as work order A, grinding process, 50 pieces completed) through workstation microphones or mobile devices, and the system automatically transcribes the speech.

[0164] 3. Text Input Mode: Input work information directly through a simple terminal or mobile device. Suitable for scenarios where text input is possible.

[0165] All collected modal data are encapsulated in a unified format and then asynchronously uploaded to the cloud for processing via a message queue. The acquisition terminal does not display any recognition results, and workers can return to production without waiting for further processing.

[0166] Step 2: Structured Information Extraction Based on Multimodal LLM (Cognitive Extraction Layer)

[0167] The cloud retrieves data collection tasks (which could be images, speech-to-text transcription, or direct text input) from a message queue and calls a multimodal large language model (such as Tongyi Qianwen) for semantic understanding. Since the structured fields required for production reports are relatively fixed, the model can extract a unified structured JSON from any input modality, containing the following fields:

[0168] Work order related fields: identified work order number text, product model, batch number;

[0169] Process-related fields: Identified process descriptions (possibly in colloquial language);

[0170] Personnel-related fields: Name or employee number of the identified operator;

[0171] Quantity and Time fields: Number of completed items, Time of reporting work;

[0172] Identify confidence fields: The model's self-assessed confidence level for the identification results of each field.

[0173] The output at this stage is the text transcription result extracted from the original input, which has not yet been associated with the ERP system entity and will be used as the input for the Agent Loop.

[0174] Step 3: Agent Loop Intelligent Search and Entity Mapping (Autonomous Reasoning Layer)

[0175] The system initializes the Agent, whose goal is to map the text fields output in step two to unique entity IDs in the ERP system. As shown in Figures 5(a) and (b), the Agent Loop executes the following multi-step inference and self-correction process:

[0176] 1. Exact match attempt

[0177] The Agent first attempts to directly call the ERP precise query interface using the extracted fields. If all fields match precisely, the mapping result is output directly without triggering subsequent search logic.

[0178] 2. Memory retrieval and context-aware inference

[0179] If an exact match fails, the Agent first searches the long-term memory (containing historical work reports and manual review / correction records for the workstation) before invoking the tool, and obtains contextual information such as the current shift and production plan for subsequent exclusive inference of candidate entities. For example, if the memory shows that the workstation has been continuously executing work order WO-2023-10-001 recently, the Agent will prioritize including this work order in the high-weight candidate list, even if the current work report identification result is ambiguous. Similarly, if a previous fuzzy input was manually corrected to a specific entity ID, the Agent can directly retrieve the correction record to assist in the judgment when encountering similar inputs again.

[0180] 3. Construction of dynamic tool call chain

[0181] Based on the failure reasons for each field, the Agent autonomously plans and executes the tool call chain in order of priority, for example:

[0182] Scenario A (work order number recognition error): The Agent calls the search_work_order tool, passes in the recognized number fragments, product model and other relevant fields to perform fuzzy search, and returns a list of candidate work orders;

[0183] Scenario B (Mismatched process names): The Agent calls the get_routing_info tool to obtain the standard process route of the candidate work order, and uses semantic vector matching to map the identified colloquial description (such as grinding) to the most similar standard process code (such as OP30-Surface Treatment).

[0184] Scenario C (unclear personnel identification / duplicate names): The Agent calls the search_employee tool and uses contextual information (current shift, list of workstation supervisors, time period) as filtering conditions to make an exclusive inference among multiple candidates with the same name, narrowing it down to a uniquely identified employee number;

[0185] Scenario D (Multiple fields are fuzzy): The agent parses the work order, process, and personnel level by level according to their priority. The parsing result of the previous step is used as the constraint condition for the next step (e.g., first determine the work order, and then match the process within the process route of the work order).

[0186] 4. Self-correction mechanism

[0187] If a tool call returns an empty result or the result confidence is too low, the Agent will not directly report an error and terminate. Instead, it will execute self-correction logic: backtrack to the previous inference step, expand the range of search parameters (such as increasing the tolerance of fuzzy matching), switch to a backup tool (such as switching from API query to vector database semantic retrieval), or rebuild the query strategy in combination with the confirmed fields, until a mapping result that meets the confidence threshold is found or it is confirmed that mapping is not possible.

[0188] 5. Physical anchoring output

[0189] The Agent ultimately outputs a set of verified system entity IDs and a complete inference chain record, the latter serving as the reviewer LLM's input.

[0190] Step 4: Heterogeneous dual-model adversarial verification and confidence assessment (review layer)

[0191] This invention introduces an independent Reviewer LLM (a heterogeneous model with a different source / architecture from the extracted Agent) to perform adversarial review of the Agent's output, such as... Figure 6 As shown. The Reviewer receives the following input:

[0192] A. Raw acquired data (images, speech-to-text transcriptions, or text input);

[0193] B. Complete inference chain record of the agent (including the input and output of each tool call);

[0194] C.Agent is the final set of entity IDs mapped.

[0195] The Reviewer LLM proactively questions the Agent's output from a quality inspector's perspective, executing the following verification logic:

[0196] 1. Cross-validation of raw data: The Reviewer independently analyzes the raw collected data (such as re-examining visual features in images and key information in speech-transcribed text) to verify whether it is consistent with the work order information mapped by the Agent.

[0197] 2. Reasoning Link Review: The Reviewer reviews whether there are logical jumps or excessive inferences in the Agent's reasoning steps (e.g., the Agent believes it is work order A, but the features in the original data are closer to work order B).

[0198] 3. Consistency check: Verify the consistency of business logic among the mapped work order, process, and personnel (e.g., whether the personnel have the authority to execute the process).

[0199] The reviewer outputs a structured review report:

[0200] A. Confidence score (0-100%)

[0201] B. Natural Language Risk Description (e.g., Confidence level 65%, Risk point: Work process name depends on semantic inference, and the person has multiple work orders in progress this month, which poses a risk of work order confusion).

[0202] C. Review suggestions (automatic posting / triggering manual review / rejection and return).

[0203] Step 5: Priority-based dynamic scheduling and incremental feedback (processing scheduling layer)

[0204] The backend processing system introduces a priority queue mechanism to dynamically adjust the task processing order based on the following dimensions:

[0205] A. Work order urgency level (priority marker from ERP);

[0206] B. Identify confidence level (prioritize scheduling tasks with high confidence level to reduce manual intervention and waiting time).

[0207] It also supports phased processing and progressive feedback: the backend can prioritize and quickly complete the confirmation of high-confidence fields (such as work order number) and write back part of the status first, and then process low-confidence fields. The frontend can obtain partial confirmation status notifications in real time without waiting for all fields to be processed.

[0208] Step Six: Human-Machine Collaborative Review and Closed-Loop Feedback (Application Layer)

[0209] High-confidence tasks (those exceeding a threshold, such as 85%) are automatically posted to the ERP system without manual intervention.

[0210] Low-confidence tasks are placed in an exception handling queue and processed by back-office administrators (rather than front-line workers) in the office environment. The system highlights risk fields and the reviewer's natural language risk descriptions on the review interface to help administrators quickly locate and correct discrepancies.

[0211] Closed-loop memory writing: The results of manual corrections are written into the Agent's long-term knowledge base through a memory mechanism, recording structured information including the original fuzzy input, the corrected entity ID, and the correction context. The Agent can retrieve these historical correction records in subsequent inference, directly assisting in judgment when encountering the same or similar fuzzy inputs, forming a complete closed loop of acquisition → extraction → inference → verification → correction → memory → inference optimization.

[0212] The embodiments of the present invention are method embodiments corresponding to the system embodiments described above. The specific operations of each step can be understood by referring to the description of the system embodiments, and will not be repeated here.

[0213] In summary, compared with the prior art, the embodiments of the present invention have the following beneficial effects:

[0214] 1. Decoupling the time of data collection and processing to eliminate interference with production rhythm: This embodiment of the invention achieves asynchronous separation between the data collection end and the processing end through a message queue. After the operator provides the original information, they can return to production without waiting for the system to process and confirm. This completely eliminates the systemic interference of the synchronous interaction mode of "stop work → scan code → select → input → submit" in the traditional work reporting scheme on the production rhythm, and truly achieves zero perception of data collection on the production process.

[0215] 2. Achieving Intelligent Mapping of Non-Standard Text to System Entities: This embodiment of the invention introduces an Agent with tool invocation, memory retrieval, and self-correction capabilities to accurately map colloquial and ambiguous text identified by OCR to unique entity identifiers in the ERP system. The Agent can autonomously plan tool invocation chains and utilize fuzzy search, semantic vector matching, and contextual exclusiveness inference to map polishing to OP30-Surface Processing, and Zhang San combined with team information to a unique employee ID, fundamentally solving the technical challenge that traditional OCR cannot meet the stringent entity parsing requirements of B-end systems.

[0216] 3. Enhance the reliability and credibility of system output: This embodiment of the invention introduces a heterogeneous dual-model adversarial verification architecture, where an independent auditing model cross-validates the agent's reasoning process and output, effectively avoiding homology bias and significantly improving the accuracy and interpretability of the system output. The auditing model not only outputs confidence scores but also generates natural language risk statements, providing managers with clear auditing criteria.

[0217] 4. Achieving system self-evolution capability: This embodiment of the invention feeds back the results of manual corrections to the Agent's memory bank, enabling the system to continuously accumulate experience. When encountering similar fuzzy inputs in the future, it can directly retrieve historical correction records to assist in judgment, forming a complete closed loop of collection → extraction → reasoning → verification → correction → memory → reasoning optimization. The system's accuracy continues to improve with the time of use.

[0218] 5. Compatible with existing paper-based work processes, reducing implementation difficulty: This embodiment of the invention supports automatic shooting of paper documents through a fixed camera, allowing workers to maintain their existing paper-based recording habits without being forced to switch to purely digital operations, greatly reducing on-site implementation resistance; at the same time, the system supports multiple interaction methods such as physical button triggering and voice command, adapting to the different operating preferences of workers.

[0219] 6. Supports diverse industrial site deployments: This invention provides multiple acquisition modes and triggering methods, which can be flexibly selected or combined according to workshop environmental conditions and management requirements. The fixed camera solution can circumvent the restrictions on carrying mobile phones in confidential workshops, and the environmental adaptive unit ensures the acquisition quality in harsh environments such as oil and dust, meeting the deployment needs of diverse industrial scenarios.

[0220] 7. Implementing priority scheduling and progressive feedback: This embodiment of the invention uses a priority scheduling module to dynamically adjust the processing order based on the urgency of the work order and the confidence level of identification, ensuring that urgent tasks are processed first; the progressive feedback module enables the front end to obtain partial confirmation status in real time without waiting for all fields to be processed, thus improving the user experience.

[0221] 8. Supports multiple deployment modes: The embodiments of this invention support cloud, edge and hybrid deployment modes, and can flexibly choose deployment schemes according to the enterprise's data security requirements, network conditions and computing resources to meet the actual needs of enterprises of different sizes.

[0222] Device Example 1

[0223] This invention provides an electronic device, including: a memory, a processor, and a computer program stored in the memory and executable on the processor. When the computer program is executed by the processor, it performs the steps described in the method embodiment.

[0224] Device Example 2

[0225] This invention provides a computer-readable storage medium storing an information transmission implementation program, which, when executed by a processor, performs the steps described in the method embodiment.

[0226] The computer-readable storage media described in this embodiment include, but are not limited to, ROM, RAM, disk, or optical disk.

[0227] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. An asynchronous intelligent work reporting system based on a multimodal large model and intelligent agents, characterized in that, include: A multimodal acquisition terminal is deployed at the production workstation to acquire raw work report data in response to a trigger event, encapsulate the raw work report data and send it asynchronously to the message queue service module. After sending, there is no need to wait for the processing result, and the multimodal acquisition terminal is released to continue subsequent production operations. The message queue service module is connected to the multimodal acquisition terminal and is used to receive and cache the original work report data; The cognitive extraction module, connected to the message queue service module, is used to pull the original work report data from the message queue service module and call the multimodal large language model to perform semantic understanding and structured information extraction on the original work report data to generate initial structured data. The Agent reasoning module, connected to the cognitive extraction module, is used to receive the initial structured data, map the text descriptions in the initial structured data to unique entity identifiers of the ERP system through the intelligent Agent, and generate mapping results and reasoning link records. The adversarial verification module, connected to the Agent inference module and the message queue service module, is used to perform adversarial verification on the mapping results and the inference link records through a heterogeneous independent auditing language model, and generate an audit report; The application interaction module is connected to the adversarial verification module and the agent inference module. It is used to execute the corresponding processing strategy according to the audit report and feed back the manual correction result to the memory of the agent inference module.

2. The system according to claim 1, characterized in that, The multimodal acquisition terminal specifically includes: An image acquisition unit includes a camera and a trigger controller, used to respond to a trigger event and control the camera to acquire image data. The trigger controller includes at least one of a video stream detection subunit, a physical button, and a manual photo capture interface. The video stream detection subunit receives the video stream acquired by the camera and performs differential calculations on consecutive frames in the video stream. When a new stationary object is detected, a rectangular outline is detected in the object's region. If the outline size matches a preset document size and the stabilization time exceeds a preset threshold, automatic photo capture is triggered. The physical button is an industrial-grade button located at the workstation, used to trigger photo capture in response to a press operation. The manual photo capture interface communicates with a mobile device to receive images captured and uploaded by the mobile device. The voice acquisition unit includes a microphone and a voice activity detection subunit, which is used to acquire workers' oral work reports through the microphone and perform voice start and end point recognition, environmental noise suppression and echo cancellation processing on the oral work reports through the voice activity detection subunit to generate voice data; A text input unit, including an industrial-grade terminal device or a mobile input interface, is used to receive direct text input through the industrial-grade terminal device or the mobile input interface and generate text data. An environment adaptive unit, connected to the image acquisition unit and the voice acquisition unit, is used to detect environmental conditions and adjust the operating parameters of the image acquisition unit and the voice acquisition unit according to the detection results.

3. The system according to claim 1, characterized in that, The cognitive extraction module specifically includes: A multimodal data parser is used to pull raw work report data from the message queue service module, identify the type of the raw work report data, call the corresponding multimodal large language model interface according to the identification result, and output the raw work report data and its type identification result to the prompt word construction unit. The prompt word construction unit, connected to the multimodal data parser, is used to receive the original work report data and its type identification result, generate prompt words for extracting structured information based on the type identification result and the business entity field to be extracted, and pass the prompt words and the original work report data to the invoked multimodal large language model; The field extraction unit is used to receive the structured output results returned by the multimodal large language model, parse the structured output results and extract at least one business entity field and the first confidence level of each field, and output the extraction results; The confidence assessment unit, connected to the field extraction unit, is used to receive the extraction results and comprehensively calculate the confidence score of each field based on the self-assessment results of the multimodal large language model, the field format standardization, and the completeness of the extraction results. The business entity fields extracted by the field extraction unit are merged with the confidence scores to generate initial structured data and output to the Agent inference module.

4. The system according to claim 1, characterized in that, The Agent reasoning module specifically includes: The precise matching unit, connected to the cognitive extraction module, is used to receive the initial structured data, call the precise query interface of the ERP system to match the text description in the initial structured data with the entity identifier in the ERP system. If the match is successful, the mapping result is directly output to the inference output unit, and the Agent inference process ends. If the match fails, the field that failed to match and the corresponding text description are output to the memory retrieval unit and the tool calling unit. The memory retrieval unit, connected to the precise matching unit, is used to receive the field that failed to match and its corresponding text description, retrieve the historical records related to the text description from the memory bank, and output the retrieval results to the inference engine unit. The tool calling unit, connected to the precise matching unit, is used to receive the fields that failed to match and their corresponding text descriptions, call the corresponding external query tool according to the type of the field that failed to match, obtain the candidate entity list from the ERP system, and output the candidate entity list to the inference engine unit; The inference engine unit, connected to the memory retrieval unit and the tool invocation unit, is used to perform comprehensive inference based on the retrieval results and the candidate entity list, determine the final entity identifier, generate mapping results and inference link records, and output the mapping results and inference link records to the self-correction unit. A self-correction unit, connected to the inference engine unit and the tool invocation unit, is used to receive the mapping results and inference link records. When the confidence level of the mapping result is lower than a preset threshold, it triggers the tool invocation unit to adjust the invocation strategy and re-execute the tool invocation, or triggers the inference engine unit to re-perform comprehensive inference. The adjustment of the invocation strategy includes at least one of expanding the range of search parameters, switching from precise query to fuzzy query, or switching to a backup query tool. The inference output unit, connected to the inference engine unit and the self-correction unit, is used to receive the final mapping result and inference link record, and output them to the adversarial verification module.

5. The system according to claim 4, characterized in that, The adversarial verification module specifically includes: The data cross-validation unit is used to receive the final mapping result and inference link record, obtain the corresponding original work report data from the message queue service module, and use a heterogeneous independent auditing language model to perform independent cross-validation on the final mapping result and the original work report data to generate a first verification result. The reasoning link review unit is used to review the logical rationality of the reasoning steps in the final reasoning link record and generate a second verification result. The consistency verification unit is used to verify the consistency of business logic among the work order, process, and personnel in the final mapping result and generate a third verification result. An adversarial audit engine, connected to the data cross-validation unit, inference link review unit, and consistency verification unit, is used to receive the first verification result, the second verification result, and the third verification result, generate an audit report by integrating the verification results, and transmit the audit report to the application interaction module.

6. The system according to claim 1, characterized in that, The application interaction module specifically includes: An automatic posting unit is used to receive the audit report. When the audit report meets the preset automatic posting conditions, the unit writes the mapping result into the ERP system through the ERP system's application programming interface and generates a posting record. An exception handling unit is used to receive the audit report. When the audit report does not meet the preset automatic posting conditions, it pushes the mapping result, the audit report and the corresponding original work report data to the exception handling queue and displays the exception items to be processed on the management interface. The memory feedback unit, connected to the exception handling unit, is used to receive the manual correction results of the exception items to be processed by the manager, and write the manual correction results into the memory bank of the Agent inference module; The report statistics unit, connected to the automatic posting unit and the exception handling unit, is used to count the posting records and the exception handling records of the exception handling unit, and generate system operation indicator reports.

7. The system according to claim 1, characterized in that, The system further includes: The priority scheduling module, connected to the message queue service module and the cognitive extraction module, is used to obtain tasks to be processed from the message queue service module, prioritize the tasks to be processed according to the urgency of the work order and the first confidence level of each field in the initial structured data, and push the tasks to the cognitive extraction module in priority order. A progressive feedback module, connected to the cognitive extraction module or the agent inference module, is used to generate a partial confirmation status notification when the cognitive extraction module completes partial field extraction or the agent inference module completes partial field mapping, and to feed back the partial confirmation status notification to the multimodal acquisition terminal.

8. An asynchronous intelligent reporting method based on a multimodal large model and intelligent agents, used in the system according to any one of claims 1 to 7, characterized in that, include: The multimodal acquisition terminal deployed at the production workstation responds to the trigger event and acquires the original work report data. The original work report data is then encapsulated and asynchronously sent to the message queue service module. After the sending is completed, the multimodal acquisition terminal is released. The original work report data is received and cached through the message queue service module; The cognitive extraction module retrieves the original work report data from the message queue service module and calls the multimodal large language model to perform semantic understanding and structured information extraction on the original work report data to generate initial structured data. The Agent inference module receives the initial structured data and uses the intelligent Agent to map the text descriptions in the initial structured data to unique entity identifiers of the ERP system, generating mapping results and inference link records. The adversarial verification module uses a heterogeneous, independent auditing language model to perform adversarial auditing on the mapping results and the inference link records, and generates an audit report. The application interaction module executes the corresponding processing strategy based on the audit report and feeds back the manual correction results to the memory of the Agent inference module.

9. An electronic device, characterized in that, include: The memory, the processor, and the computer program stored in the memory and executable on the processor, wherein the computer program, when executed by the processor, implements the steps of the asynchronous intelligent reporting method based on a multimodal large model and intelligent agent as described in claim 8.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores an implementation program for information transmission, which, when executed by a processor, implements the steps of the asynchronous intelligent reporting method based on a multimodal large model and intelligent agent as described in claim 8.

Images