Production field digital human AI assistant collaborative operation system based on multi-modal interaction

Abstract

The invention discloses a production field digital human AI assistant collaborative operation system based on multi-modal interaction, and relates to the technical field of production field digital human collaborative operation, and the system comprises the steps: obtaining the voice data of a worker, and carrying out the recognition of a voice based on a voice recognition model, and generating a text; analyzing the generated text based on a natural language processing model to generate a worker intention; acquiring a gesture image of a worker, and performing gesture recognition on the gesture image based on a gesture recognition model to generate a gesture instruction; the digital human control center is responsible for receiving the intention of the worker, judging the received intention of the worker and sending a control signal according to a judgment result; the interactive interface generation module receives the control signal and generates a non-contact interactive interface in combination with the gesture instruction; and the intelligent quality inspection module receives the control signal, plans an optimal quality inspection path, and performs quality inspection on the product by means of the target inspection model to generate a quality inspection report. According to the invention, non-contact operation of workers is realized, the labor cost of product quality inspection is reduced, and the quality inspection efficiency is improved.

Description

Technical Field

[0001] This invention relates to the field of digital human collaborative operation technology in production sites, specifically a digital human AI assistant collaborative operation system for production sites based on multimodal interaction. Background Technology

[0002] Under the general trend of intelligent and digital transformation in manufacturing, my country's manufacturing industry is facing a critical period of transition from traditional production models to efficient, precise, and intelligent production models. As the core link in manufacturing, the production site's operational efficiency and management level directly affect product quality, production costs, and the market competitiveness of enterprises. However, many problems still urgently need to be addressed in current manufacturing production sites.

[0003] In daily operations in the production workshop, workers frequently need to check product process flows and their own task assignments. However, because workers' hands easily become stained with oil, dust, and other contaminants during operation, they inevitably soil the equipment when directly touching the large-screen computers and keyboards to retrieve information. This not only affects the lifespan and operational sensitivity of the computers and keyboards but also requires additional time and effort for cleaning and maintenance. Furthermore, workers operating the equipment with dirty hands may experience inconvenience or obstructed vision, leading to inefficient searches for process flows and task information, wasting significant time and ultimately impacting production progress.

[0004] Product quality is the lifeline of the manufacturing industry, and quality inspection and patrol inspection play a crucial role in ensuring product quality. Traditional product quality inspection and patrol inspection mainly rely on manpower, requiring a large number of quality inspectors to check each product individually. This not only consumes a significant amount of human and material resources, but manual inspection also has certain subjectivity and limitations, potentially leading to missed or incorrect inspections, affecting the stability of product quality. With the continuous rise in labor costs, enterprises face enormous pressure from labor costs and urgently need a more efficient and accurate method of quality inspection and patrol inspection. Summary of the Invention

[0005] To address the aforementioned technical problems, the present invention aims to provide a collaborative work system for digital human AI assistants in production sites based on multimodal interaction. This system includes: a speech recognition module, a text parsing module, a gesture recognition module, a digital human control center module, an interactive interface generation module, and an intelligent quality inspection module. The modules described above are connected via wired and / or wireless means to enable data transmission between them. The speech recognition module acquires workers' speech data, recognizes the speech based on the speech recognition model, and generates text. The text parsing module, based on a natural language processing model, parses the generated text to obtain the worker's intent; The gesture recognition module acquires images of worker gestures, performs gesture recognition based on the gesture recognition model, and generates gesture commands. The digital human control center module is responsible for receiving worker intentions, judging the received worker intentions, and sending control signals to the interactive interface generation module or intelligent quality inspection module through the PLC controller based on the judgment results. The interactive interface generation module receives control signals sent by the digital human control center and combines them with gesture commands to generate a contactless large-screen digital human interactive interface, enabling the digital human to freely interact with workers through voice and gestures. The intelligent quality inspection module receives control signals from the digital human control center, plans the optimal quality inspection path, and uses the target detection model to inspect the products, generate quality inspection reports, and feed them back to the digital human, which then presents them to the operators.

[0006] Furthermore, the process of generating text is as follows: Mel-frequency cepstral coefficients are extracted from worker speech data as speech features. The import torch command is used to call the PyTorch deep learning framework to load a pre-trained speech recognition model, convert the speech features into PyTorch tensor form, input them into the pre-trained speech recognition model, perform speech recognition on the speech features, and output text. The speech recognition model includes deep neural networks (DNN), recurrent neural networks (RNN), and Transformer.

[0007] Furthermore, the worker's intent includes: querying the process flow, querying equipment parameters, querying task assignments, and product quality inspection.

[0008] Furthermore, the process of obtaining worker intentions is as follows: Using HuggingFace's transformers library to load pre-trained natural language processing models, the Spacy library is used to perform part-of-speech tagging, stop word removal, and named entity recognition on the text. A text classification pipeline is created, and classification algorithms are used to call the knowledge graph database API interface to obtain worker intent. The classification algorithms include support vector machines, Naive Bayes, decision trees, and neural network algorithms. The natural language processing models include GPT, LSTM, and T5.

[0009] Furthermore, the process of generating gesture commands is as follows: The worker's gesture image is normalized to keep the pixel values ​​between [-1, 1]. The worker's gesture image is then input into a pre-trained gesture recognition model. The gesture recognition model analyzes and processes the input image, outputs the recognized gesture category, matches the gesture category with standard gestures in the gesture library, finds the gesture with the highest similarity, and obtains the instruction corresponding to the gesture with the highest similarity as the gesture instruction. The gesture instructions include: swiping left or right with the palm to switch pages, tapping in the air to select a menu, clenching a fist to stop, and raising a thumb to start.

[0010] Furthermore, the process of judging the worker's intention and generating the judgment result is as follows: If the digital human control center receives a worker's intention to query the process flow, query equipment parameters, or query task arrangement, it will send a query process flow, query equipment parameters, or query task arrangement control signal to the interactive interface generation module through the PLC controller. If a worker intends to perform product quality inspection, the PLC controller sends a product quality inspection control signal to the intelligent quality inspection module.

[0011] Furthermore, the process of planning the optimal inspection path is as follows: A quality inspection scenario model is constructed, a 3D map of the production site is generated, and the product location and obstacle location are marked on the 3D map. An improved algorithm combined with Dijkstra's algorithm is used to plan the quality inspection path and generate the optimal quality inspection path.

[0012] Furthermore, the process of generating a quality inspection report is as follows: The intelligent quality inspection module receives product quality inspection control signals from the digital human control center and develops a robot engine. The robot receives the product quality inspection control signals, acquires product image data along the generated optimal quality inspection path step by step, and sends it to the pre-trained target detection model. The target detection model generates a pass or fail result and feeds it back to the robot. The robot integrates the results and transmits them to the intelligent quality inspection module. Based on the integrated results and combined with pre-set product quality inspection standards and report templates, the intelligent quality inspection module comprehensively evaluates various indicators of the product and generates a quality inspection report.

[0013] Compared with the prior art, the present invention has the following beneficial effects: By acquiring multimodal worker voice data from the production site and using a speech recognition model to parse the worker's voice data to generate text, the text is then analyzed using a natural language processing model to interpret the worker's intentions. Simultaneously, multimodal worker gesture data from the production site is acquired, and gesture commands are generated. The digital human control center judges the interpreted worker intentions and sends control signals to the interactive interface generation module based on the judgment results. Combined with the gesture commands, a contactless large-screen digital human interactive interface is generated, enabling a free interactive experience between the digital human and the worker through voice and gestures. The intelligent quality inspection module receives the control signals sent by the digital human control center, plans the optimal quality inspection path, and uses a target detection model to perform quality inspection on the products, generating a quality inspection report, which is then fed back to the digital human and presented to the operators.

[0014] On the one hand, it enables workers to interact with digital humans in a non-contact manner through multimodal data interaction such as voice and gestures, avoiding the impact of dirty hands on equipment and improving the efficiency of information query and operation; on the other hand, by integrating intelligent quality inspection technology, it realizes the automatic detection and analysis of products on the production site, generates quality inspection reports, and feeds them back to the digital human, which then presents them to the operators; reducing labor input costs while improving the accuracy and efficiency of quality inspection. Attached Figure Description

[0015] Figure 1 This is a flowchart illustrating the connection of each module in the multimodal interactive production site digital human AI assistant collaborative operation system, as described in this application embodiment. Detailed Implementation

[0016] like Figure 1 As shown, a multimodal interactive digital human AI assistant collaborative operation system for production sites includes: a speech recognition module, a text parsing module, a gesture recognition module, a digital human control center module, an interactive interface generation module, and an intelligent quality inspection module. The modules described above are connected via wired and / or wireless means to enable data transmission between them. The speech recognition module acquires workers' speech data, recognizes the speech based on the speech recognition model, and generates text. The method for acquiring worker voice data is as follows: industrial-grade noise-canceling microphone arrays deployed in the AI ​​digital human system, such as Sennheiser MKE416 (explosion-proof) and Audio-Technica AT899 (dustproof and splashproof), can focus on worker voices and suppress background noise (such as machine noise and workshop voices).

[0017] The process of recognizing speech based on a speech recognition model and generating text is as follows: Extract the Mel-frequency cepstral coefficients from the worker's speech data as speech features, use the "import torch" command to call the PyTorch deep learning framework to load the pre-trained speech recognition model, convert the speech features into the form of PyTorch tensors, input them into the pre-trained speech recognition model, perform speech recognition on the speech features, and output the text; The speech recognition model includes deep neural networks DNN, recurrent neural networks RNN, Transformer, etc.

[0018] On the basis of the above embodiment, it should be supplemented that the Mel-frequency cepstral coefficients can convert the speech signal into a feature representation consistent with human auditory perception. By using a Mel filter bank to perform weighted processing on the spectrum, the frequency components sensitive to the human ear can be highlighted, and at the same time, the frequency components insensitive to the human ear can be suppressed, which enables the speech recognition model to pay more attention to the speech features related to human auditory perception and improve the recognition effect.

[0019] The text parsing module, based on a natural language processing model, parses the generated text to obtain the worker's intention, and the worker's intention includes: querying the process flow, querying device parameters, querying task arrangements, and product quality inspection.

[0020] The process of obtaining the worker's intention is as follows: Load the pre-trained natural language processing model with the transformers library of HuggingFace. The model performs word tagging, stop word removal, and named entity recognition on the text through the Spacy library, and creates a text classification pipeline. Using classification algorithms, call the api interface of the knowledge graph database to obtain the worker's intention; The classification algorithms include: support vector machine, naive Bayes, decision tree, and neural network algorithms.

[0021] On the basis of the above embodiment, it should be supplemented that the transformers library of HuggingFac is a powerful open-source library for natural language processing (NLP), which provides support for various pre-trained models; Word tagging can provide detailed grammatical information for the text, and it marks each word in the text as specific word classes such as nouns, verbs, adjectives, adverbs, etc., which helps us deeply understand the grammatical structure of the text; Stop words (such as "de", "le", "he", etc.) appear frequently in the text, but usually contribute less to the main semantics of the text. Removing stop words can reduce the scale of the text and speed up the text processing; Named entity recognition can identify key entities in the text, such as person names, place names, organization names, device names, time, etc., which is very useful in text processing tasks.

[0022] Knowledge graphs primarily represent structured knowledge content such as product information, process flow, task assignment, and inspection process, enabling digital humans to better understand worker issues.

[0023] The gesture recognition module acquires images of worker gestures, performs gesture recognition on the images based on the gesture recognition model, and generates gesture commands.

[0024] The method for acquiring worker gesture images is as follows: they are acquired by a depth camera mounted on an AI digital human system; The process of generating gesture commands is as follows: The worker's gesture image is normalized to ensure that the pixel values ​​remain within the range [-1, 1]. The worker's gesture image is then input into a pre-trained gesture recognition model. The model analyzes and processes the input image, outputs the recognized gesture category, matches the gesture category with standard gestures in the gesture library, finds the gesture with the highest similarity, and obtains the command corresponding to the gesture with the highest similarity as the gesture command. The gesture commands include, but are not limited to: swiping left or right with the palm to switch pages, tapping in the air to select a menu, clenching a fist to stop, and raising a thumb to start.

[0025] Based on the above embodiments, it is necessary to add that the gesture recognition model includes neural network models such as CNN, LeNet, AlexNet, ResNet, etc. The gesture library contains standard gesture images and corresponding gesture instructions. For example, if the gesture category of a worker's gesture image is that the hands are crossed in front of the body to form a cross, then the corresponding gesture instruction is to turn off the main power of the equipment. Swiping left indicates switching to the previous page, and swiping right indicates switching to the next page.

[0026] The digital human control center module is responsible for receiving the worker's intentions obtained by the text parsing module, judging the received worker's intentions, and sending control signals to the interactive interface generation module or intelligent quality inspection module through the PLC controller based on the judgment result.

[0027] The process of judging the worker's intention and generating the judgment result is as follows: If the digital human control center receives a worker's intention to query the process flow, query equipment parameters, or query task arrangement, it will send a query process flow, query equipment parameters, or query task arrangement control signal to the interactive interface generation module through the PLC controller. If a worker intends to perform product quality inspection, the PLC controller sends a product quality inspection control signal to the intelligent quality inspection module.

[0028] The interactive interface generation module receives control signals sent by the digital human control center and combines them with gesture commands to generate a contactless large-screen digital human interactive interface, enabling the digital human to freely interact with workers through voice and gestures.

[0029] Building upon the aforementioned embodiments, it is worth noting that this contactless large-screen digital human interactive interface features a simple and intuitive layout, making it easy for workers to operate and quickly displaying worker needs based on their intentions. Simultaneously, it boasts a high-resolution display, ensuring that the information displayed by the digital human is clearer and more accurate, perfectly presenting both text descriptions and graphic displays. Furthermore, this contactless large-screen digital human interactive interface is optimized for different gesture commands, with extremely fast recognition and response speeds, ensuring that worker gestures are captured instantly and responded to accordingly, further enhancing the smoothness and convenience of the interaction.

[0030] The intelligent quality inspection module receives control signals from the digital human control center, plans the optimal quality inspection path, and uses the target detection model to inspect the products, generate quality inspection reports, and feed them back to the digital human, which then presents them to the operators.

[0031] The process of planning the optimal quality inspection path involves: constructing a quality inspection scenario model, generating a 3D map of the production site, and marking product and obstacle locations on the 3D map; and adopting improved... The algorithm combines Dijkstra's algorithm to plan the quality inspection path and generate the optimal quality inspection path.

[0032] Based on the above embodiments, it is necessary to add that, while Dijkstra's algorithm can guarantee the optimal solution by using LiDAR and visual SLAM technology to construct a quality inspection scene model and generate a 3D map of the production site, its computational complexity is high; an improved version is needed. The algorithm reduces computational complexity by optimizing the heuristic function, but at the cost of some accuracy. Combining both approaches achieves a balance between computational resources and path accuracy. In the intelligent quality inspection module, the computational resources can be adjusted appropriately based on the scale of the production site and real-time conditions when computational resources are scarce. The weights of the algorithm's heuristic function reduce computation while ensuring high accuracy. When resources are sufficient, Dijkstra's algorithm is used to ensure the absolute optimal path, so that path planning can meet accuracy requirements without excessively consuming computational resources, thereby improving the overall performance of the system.

[0033] The process of generating a quality inspection report is as follows: The intelligent quality inspection module receives product quality inspection control signals sent by the digital human control center, develops a robot engine, and the robot receives the product quality inspection control signals. Based on the generated optimal quality inspection path, the robot gradually acquires product image data along the inspection path and sends it to the pre-trained target detection model. The target detection model generates a pass or fail result and feeds the result back to the robot. The robot integrates the results and transmits them to the intelligent quality inspection module. Based on the integrated results, combined with the pre-set product quality inspection standards and report templates, the intelligent quality inspection module comprehensively evaluates the various indicators of the product and generates a quality inspection report.

[0034] Based on the above embodiments, it should be added that the product image data acquisition method is acquired by a depth camera or a high-speed camera, and the target detection model includes, but is not limited to, deep learning models such as YOLOv5, Mask R-CNN, and DETR. The report not only covers the conclusion of whether a product is qualified, but also includes specific problems with non-conforming products, their exact location on the quality inspection path, and image data of the non-conforming products as evidence. The intelligent quality inspection module sends the quality inspection report to the digital human control center, where the digital human presents it to the operators in a user-friendly and intuitive way, allowing them to understand the product quality status and take timely appropriate measures.

[0035] This embodiment acquires voice data from workers on the production site, uses a speech recognition model to parse the voice data and generate text, then uses a natural language processing model to analyze the workers' intentions from the text. Simultaneously, it acquires workers' gesture data and generates gesture commands. The digital human control center judges the analyzed worker intentions and, based on the judgment result, sends control signals to the interactive interface generation module. Combined with the gesture commands, it generates a contactless large-screen digital human interactive interface, enabling free interaction between the digital human and workers through voice and gestures. The intelligent quality inspection module receives control signals from the digital human control center, plans the optimal quality inspection path, and uses a target detection model to inspect products, generating a quality inspection report, which is then fed back to the digital human. The digital human presents the report to the operator, who analyzes it and saves the analyzed report in the digital human collaborative operation system database, providing assistance for the digital human's future learning.

[0036] On the one hand, it solves the problem of equipment contamination caused by workers' dirty hands operating equipment on the production site, as well as the need for subsequent equipment cleaning, and the problem of wasted time when workers operate equipment while working; on the other hand, workers can interact with digital humans in a non-contact manner through multimodal data interaction such as voice and gestures, avoiding the impact of dirty hands on equipment and improving the efficiency of information query and operation.

[0037] On the other hand, it solves the problem that the need for manual quality inspection and patrol of products on the production site increases the company's labor costs; by integrating intelligent quality inspection technology, it realizes automatic detection and analysis of products, reduces labor input, and improves the accuracy and efficiency of quality inspection.

[0038] It should be understood that in the various embodiments of this application, the order of the above-mentioned processes does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0039] The above embodiments are only used to illustrate the technical method of the present invention and are not intended to limit the present invention. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical method of the present invention may be modified or replaced by equivalents without departing from the spirit and scope of the technical method of the present invention.

[0040] In conclusion, the above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A collaborative work system for digital human AI assistants in production sites based on multimodal interaction, characterized in that: The system includes: a speech recognition module, a text parsing module, a gesture recognition module, a digital human control center module, an interactive interface generation module, and an intelligent quality inspection module; The speech recognition module acquires workers' speech data, recognizes the speech based on the speech recognition model, and generates text. The text parsing module, based on a natural language processing model, parses the generated text to obtain the worker's intent; The gesture recognition module acquires images of worker gestures, performs gesture recognition based on the gesture recognition model, and generates gesture commands. The digital human control center module is responsible for receiving worker intentions, judging the received worker intentions, and sending control signals to the interactive interface generation module or intelligent quality inspection module through the PLC controller based on the judgment results. The interactive interface generation module receives control signals sent by the digital human control center and combines them with gesture commands to generate a contactless large-screen digital human interactive interface, enabling the digital human to freely interact with workers through voice and gestures. The intelligent quality inspection module receives control signals from the digital human control center, plans the optimal quality inspection path, and uses the target detection model to inspect the products, generate quality inspection reports, and feed them back to the digital human, which then presents them to the operators.

2. The production site digital human AI assistant collaborative operation system based on multimodal interaction according to claim 1, characterized in that, The process of generating text is as follows: Mel-frequency cepstral coefficients are extracted from worker speech data as speech features. The import torch command is used to call the PyTorch deep learning framework to load a pre-trained speech recognition model, convert the speech features into PyTorch tensor form, input them into the pre-trained speech recognition model, perform speech recognition on the speech features, and output text. The speech recognition model includes deep neural networks (DNN), recurrent neural networks (RNN), and Transformer.

3. The production site digital human AI assistant collaborative operation system based on multimodal interaction according to claim 1, characterized in that, The worker's intentions include: querying process flow, querying equipment parameters, querying task assignments, and product quality inspection.

4. The production site digital human AI assistant collaborative operation system based on multimodal interaction according to claim 1, characterized in that, The process of obtaining worker intentions is as follows: By using HuggingFace's transformers library to load a pre-trained natural language processing model, and using the Spacy library to perform part-of-speech tagging, stop word removal, and named entity recognition on the text, a text classification pipeline is created. A classification algorithm is then used to call the knowledge graph database API interface to obtain the worker's intent. The classification algorithms include: support vector machine, Naive Bayes, decision tree and neural network algorithms, and the natural language processing models include GPT, LSTM and T5.

5. The production site digital human AI assistant collaborative operation system based on multimodal interaction according to claim 1, characterized in that, The process of generating gesture commands is as follows: The worker's gesture image is normalized to keep the pixel values ​​between [-1, 1]. The worker's gesture image is then input into a pre-trained gesture recognition model. The gesture recognition model analyzes and processes the input image, outputs the recognized gesture category, matches the gesture category with standard gestures in the gesture library, finds the gesture with the highest similarity, and obtains the instruction corresponding to the gesture with the highest similarity as the gesture instruction. The gesture instructions include: swiping left or right with the palm to switch pages, tapping in the air to select a menu, clenching a fist to stop, and raising a thumb to start.

6. The production site digital human AI assistant collaborative operation system based on multimodal interaction according to claim 1, characterized in that, The process of judging the worker's intention and generating the judgment result is as follows: If the digital human control center receives a worker's intention to query the process flow, query equipment parameters, or query task arrangement, it will send a query process flow, query equipment parameters, or query task arrangement control signal to the interactive interface generation module through the PLC controller. If a worker intends to perform product quality inspection, the PLC controller sends a product quality inspection control signal to the intelligent quality inspection module.

7. The production site digital human AI assistant collaborative operation system based on multimodal interaction according to claim 1, characterized in that, The process of planning the optimal inspection path is as follows: A quality inspection scenario model is constructed, a 3D map of the production site is generated, and product and obstacle locations are marked on the 3D map. An improved [method / approach] is then employed. The algorithm combines Dijkstra's algorithm to plan the quality inspection path and generate the optimal quality inspection path.

8. The production site digital human AI assistant collaborative operation system based on multimodal interaction according to claim 1, characterized in that, The process of generating a quality inspection report is as follows: The intelligent quality inspection module receives product quality inspection control signals from the digital human control center and develops a robot engine. The robot receives the product quality inspection control signals, acquires product image data along the generated optimal quality inspection path step by step, and sends it to the pre-trained target detection model. The target detection model generates a pass or fail result and feeds it back to the robot. The robot integrates the results and transmits them to the intelligent quality inspection module. Based on the integrated results and combined with pre-set product quality inspection standards and report templates, the intelligent quality inspection module comprehensively evaluates various indicators of the product and generates a quality inspection report.

Images