A duty recording terminal and server

Abstract

The present application relates to the field of digital image processing technology, and in particular to a duty recording terminal and server. When receiving a recording instruction sent by a user, the duty recording terminal controls the camera to collect video data of the duty operation, and controls the microphone to collect audio data of the operation instruction issued by the user, and sends the collected video data and audio data to the server, so that the server analyzes the operation information included in the video data and audio data based on a pre-trained model, determines the inspection result corresponding to each preset inspection item, and receives the inspection result sent by the server, thereby prompting the user based on the inspection result, which effectively improves the efficiency of duty inspection compared with the traditional manual inspection of the duty process.

Description

Technical Field

[0001] The present application relates to the technical field of digital image processing, and in particular to a duty recording terminal and a server. Background Art

[0002] With the rapid development of the railway transportation industry, the compliance of railway crew members' on-duty process has become increasingly important. Therefore, it is necessary to inspect the on-duty process of crew members. The traditional inspection method mainly relies on manual work, which has the problem of low efficiency.

[0003] Therefore, how to improve the efficiency of on-duty inspections has become an urgent problem to be solved. Summary of the invention

[0004] The embodiments of the present application provide a duty recording terminal and a server to solve the problem of low duty inspection efficiency in the prior art.

[0005] The present application provides a duty recording terminal, which includes a camera, a microphone, a processor, and a communication module:

[0006] The processor is configured to: if a recording instruction sent by a user is received, control the camera to collect video data of the on-duty operation, and control the microphone to collect audio data of the operation instruction issued by the user;

[0007] The collected video data and the audio data are sent to the server through the communication module, so that the server analyzes the operation information included in the video data and the audio data based on the pre-trained model to determine the inspection result corresponding to each preset inspection item;

[0008] The inspection result sent by the server is received through the communication module, and the user is prompted based on the inspection result.

[0009] The above technical solution has the following advantages or beneficial effects: when the duty recording terminal receives a recording instruction sent by the user, it controls the camera to collect video data of the duty operation, and controls the microphone to collect audio data of the operation instruction issued by the user, and sends the collected video data and audio data to the server, so that the server analyzes the operation information included in the video data and audio data based on the pre-trained model, determines the inspection result corresponding to each preset inspection item, and receives the inspection result sent by the server, so as to prompt the user based on the inspection result. Compared with the traditional reliance on manual inspection of the duty process, the efficiency of duty inspection is effectively improved.

[0010] Furthermore, the processor is further configured to: if a photo taking instruction sent by a user is received, control the camera to take image data of the on-duty process;

[0011] The video data, the audio data and the image data are sent to the server through the communication module, so that the server analyzes the duty information included in the video data, the audio data and the image data based on a pre-trained model to determine the inspection result corresponding to each preset inspection item.

[0012] The above technical solution has the following advantages or beneficial effects: users can also take images through the duty recording terminal, so that the server can analyze based on video data, audio data and image data, further improving the accuracy of duty inspection.

[0013] The present application provides a server configured as follows:

[0014] If video data of the on-duty operation and audio data of the operation instruction are received, the video data is analyzed based on the first multimodal large model to obtain the operation description text contained in the video data;

[0015] Performing speech recognition on the audio data to obtain a description text of the operation instruction described in the audio data;

[0016] The operation description text, operation instruction description text and prompt words containing each preset detection item are input into the first large language model, and the inspection result corresponding to each preset inspection item output by the first large voice model is obtained and sent to the duty recording terminal.

[0017] The above technical solution has the following advantages or beneficial effects: first, the first multimodal large model is used to analyze the content contained in the video data, and the content included in the audio data is converted into text. Finally, the first language large model combines the two parts to determine the inspection results corresponding to each preset inspection item, so that duty inspections can be carried out accurately.

[0018] Furthermore, if image data of the on-duty process is received, it is also configured to:

[0019] Analyzing the image data based on the first multimodal large model to obtain other description texts corresponding to the content recorded in the image data;

[0020] The operation description text, the operation instruction description text, the other description texts and the prompt words containing each preset detection item are input into the first large language model to obtain the inspection result corresponding to each preset inspection item output by the first large speech model.

[0021] The above technical solution has the following advantages or beneficial effects: if image data is received, the first multimodal large model is used to analyze the image data to obtain text describing the content recorded in the image data, and then the first language large model is used to analyze the text of the content in the video, the text of the content in the audio, and the text of the content in the image, thereby further improving the accuracy of duty inspections.

[0022] The present application provides a server configured as follows:

[0023] If video data of on-duty operation and audio data of operation instructions are received, the video data, the audio data and the prompt words containing each preset detection item are input into the second multimodal large model, and the encoder of the second multimodal large model extracts features of the video data, the audio data and the prompt words to obtain a first feature matrix of the video data, a second feature matrix of the audio data and a third feature matrix of the prompt words;

[0024] The language large model of the second multimodal large model analyzes the first feature matrix, the second feature matrix and the third feature matrix to determine the inspection result corresponding to each preset inspection item.

[0025] The above technical solution has the following advantages or beneficial effects: the received video data and audio data are processed using the second multimodal large model completed by fine-tuning training, thereby further improving the accuracy of duty inspection.

[0026] Furthermore, if image data of the on-duty process is received, it is specifically configured as follows:

[0027] The video data, the audio data, the image data and the prompt words containing each preset detection item are input into a second multimodal large model, and an encoder of the second multimodal large model extracts features of the video data, the audio data, the image data and the prompt words to obtain a first feature matrix of the video data, a second feature matrix of the audio data, a third feature matrix of the prompt words and a fourth feature matrix of the image data;

[0028] The language large model of the second multimodal large model analyzes the first feature matrix, the second feature matrix, the third feature matrix and the fourth feature matrix to determine the inspection result corresponding to each preset inspection item.

[0029] The above technical solution has the following advantages or beneficial effects: if image data is received, the encoder of the second multimodal large model extracts features of the received image, video, and audio, and then uses the language large model in the second multimodal large model to analyze the content in the video, the audio, and the image, thereby further improving the accuracy of duty inspections.

[0030] Furthermore, it is specifically configured as follows:

[0031] The image encoder of the second multimodal large model performs feature extraction on the video data and the image data to obtain a first feature matrix of the video data and a fourth feature matrix of the image data;

[0032] The audio encoder of the second multimodal large model performs feature extraction on the audio data to obtain a second feature matrix of the audio data;

[0033] The text encoder of the second multimodal large model performs feature extraction on the prompt word to obtain a third feature matrix of the prompt word.

[0034] The above technical solution has the following advantages or beneficial effects: different types of encoders are provided for different types of data, and one encoder is dedicated to extracting features of one data type, thereby further improving the accuracy of duty inspection.

[0035] Furthermore, it is configured as follows:

[0036] Identify the targets included in the video data based on the target recognition model, and determine the video segment corresponding to each target according to the image frame where each identified target is located;

[0037] According to the detection target corresponding to each preset detection item, the video data is updated using the video segment corresponding to the detection target.

[0038] The above technical solution has the following advantages or beneficial effects: each target is identified based on the target recognition model, and then the video segment corresponding to each collected target is determined; according to the detection target corresponding to each preset detection item, the video data is updated using the video segment corresponding to the detection target, thereby removing redundant information in the video data and improving detection efficiency.

[0039] Furthermore, it is configured as follows:

[0040] Recognize the operation instructions included in the initial audio data, and determine the audio segment corresponding to each operation content according to the time corresponding to each recognized operation instruction;

[0041] According to the target operation content corresponding to each preset detection item, the audio data is updated using the audio segment corresponding to the target operation content.

[0042] The above technical solution has the following advantages or beneficial effects: the audio data is recognized, and the audio segment corresponding to each job content is determined according to the time corresponding to each job instruction, and the redundant information in the audio is eliminated, thereby improving the detection efficiency.

[0043] Furthermore, the second multimodal large model is obtained based on fine-tuning training; the training process of the second multimodal large model includes:

[0044] Obtain a preset number of sample pair data in a training set, wherein any sample pair data includes text data and non-text data, and the text data is an image, audio, or video;

[0045] Input each of the sample pair data into the original multimodal large model, and respectively determine a feature matrix corresponding to the text data and a feature matrix corresponding to the non-text data in the sample pair data;

[0046] Determine the loss value based on each feature matrix and loss function corresponding to each sample pair data;

[0047] According to the loss value and the current number of training iterations, the parameters in the original multimodal large model are adjusted to obtain the second multimodal large model.

[0048] The above technical solution has the following advantages or beneficial effects: fine-tuning and training of the second multimodal large model improves the accuracy of the second multimodal large model in inspecting the duty process.

[0049] The present application provides a method for inspecting the on-duty process based on a multimodal large model, the method comprising:

[0050] If video data of the on-duty operation and audio data of the operation instruction are received, the video data is analyzed based on the first multimodal large model to obtain the operation description text contained in the video data;

[0051] Performing speech recognition on the audio data to obtain a description text of the operation instruction described in the audio data;

[0052] The operation description text, operation instruction description text and prompt words containing each preset detection item are input into the first large language model, and the inspection result corresponding to each preset inspection item output by the first large voice model is obtained and sent to the duty recording terminal.

[0053] The present application provides a method for inspecting the on-duty process based on a multimodal large model, the method comprising:

[0054] If video data of on-duty operation and audio data of operation instructions are received, the video data, the audio data and the prompt words containing each preset detection item are input into the second multimodal large model, and the encoder of the second multimodal large model extracts features of the video data, the audio data and the prompt words to obtain a first feature matrix of the video data, a second feature matrix of the audio data and a third feature matrix of the prompt words;

[0055] The language large model of the second multimodal large model analyzes the first feature matrix, the second feature matrix and the third feature matrix to determine the inspection result corresponding to each preset inspection item.

[0056] The present application provides a duty process inspection device based on a multimodal large model, the device comprising:

[0057] A content extraction module is used for, if receiving video data of on-duty operation and audio data of operation instructions, analyzing the video data based on the first multimodal large model to obtain the operation description text contained in the video data; performing speech recognition on the audio data to obtain the operation instruction description text described in the audio data;

[0058] The analysis module is used to input the operation description text, the operation instruction description text and the prompt words containing each preset detection item into the first large language model, obtain the inspection result corresponding to each preset inspection item output by the first large voice model, and send it to the duty recording terminal.

[0059] The present application provides a duty process inspection device based on a multimodal large model, the device comprising:

[0060] A feature extraction module, for, if video data of on-duty operation and audio data of operation instructions are received, inputting the video data, the audio data and the prompt words containing each preset detection item into a second multimodal large model, wherein an encoder of the second multimodal large model extracts features of the video data, the audio data and the prompt words to obtain a first feature matrix of the video data, a second feature matrix of the audio data and a third feature matrix of the prompt words;

[0061] An analysis module is used for the language large model of the second multimodal large model to analyze the first feature matrix, the second feature matrix and the third feature matrix to determine the inspection result corresponding to each preset inspection item.

[0062] The present application also provides an electronic device, which includes a processor, and the processor is used to implement the steps of any of the above-mentioned methods for checking the on-duty process based on a multimodal large model when executing a computer program stored in a memory.

[0063] The present application also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps of any of the above-mentioned methods for inspecting the on-duty process based on a multimodal large model. BRIEF DESCRIPTION OF THE DRAWINGS

[0064] In order to more clearly illustrate the technical solution of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0065] Figure 1 A schematic diagram of the structure of a duty recording terminal provided in an embodiment of the present application;

[0066] Figure 2 A schematic diagram of a duty inspection process provided in an embodiment of the present application;

[0067] Figure 3 A schematic diagram of a duty inspection process provided in an embodiment of the present application;

[0068] Figure 4 A schematic diagram of a duty inspection process provided in an embodiment of the present application;

[0069] Figure 5 A schematic diagram of the structure of a second multimodal large model provided in an embodiment of the present application;

[0070] Figure 6 A schematic diagram of the structure of a duty process inspection device based on a multimodal large model provided in an embodiment of the present application;

[0071] Figure 7 A schematic diagram of the structure of a duty process inspection device based on a multimodal large model provided in an embodiment of the present application;

[0072] Figure 8 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0073] In order to make the purpose, technical solution and advantages of the present application clearer, the technical solution of the embodiment of the present application will be clearly and completely described below in conjunction with the accompanying drawings. Obviously, the described embodiment is only a part of the embodiment of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field belong to the scope of protection of this application.

[0074] In order to make the purpose, technical solutions and advantages of the present application clearer, the present application will be further described in detail below in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present application.

[0075] It should be noted that the terms "including" and "having" and their variations involved in the documents of this application are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0076] The terms "first" and "second" in this document are used for descriptive purposes only and should not be understood as explicitly or implicitly indicating relative importance or implicitly indicating the number of technical features indicated. Therefore, features defined as "first" and "second" may explicitly or implicitly include one or more of the features, and in the description of the embodiments of this application, unless otherwise specified, "plurality" means two or more.

[0077] The word “exemplary” is used hereinafter to mean “serving as an example, example, or illustration.” Any embodiment described as “exemplary” is not necessarily to be construed as preferred or advantageous over other embodiments.

[0078] An embodiment of the present application provides a duty recording terminal and a server, wherein the duty recording terminal includes a camera, a microphone, a processor, and a communication module: the processor is configured to: if a recording instruction sent by a user is received, control the camera to collect video data of the duty operation, and control the microphone to collect audio data of the operation instruction issued by the user; send the collected video data and audio data to the server through the communication module, so that the server analyzes the operation information included in the video data and the audio data based on a pre-trained model, and determines the inspection result corresponding to each preset inspection item; receive the inspection result sent by the server through the communication module, and prompt the user based on the inspection result.

[0079] Exemplarily, the duty recording terminal used by the user during the duty process can be a mobile phone or a wearable mobile terminal. The user can wear the duty recording terminal on the body during the duty process, for example, on the chest, head, etc. During operation, the duty recording terminal can realize functions such as sound collection, image collection, video collection, sound playback, and screen display through hardware support. It should be noted that the duty recording terminal described in the embodiment of the present application is not limited to mobile phones, but can also be other devices, such as computers, tablets, etc. The inspection process of the duty process is described below in conjunction with each embodiment. Figure 1 A structural diagram of a duty recording terminal provided in an embodiment of the present application, the duty recording terminal 10 includes a camera 101, a microphone 102, a processor 103, and a communication module 104, wherein the processor 103 is configured to: if a recording instruction sent by a user is received, control the camera 101 to collect video data of duty operations, and control the microphone 102 to collect audio data of operation instructions issued by the user; send the collected video data and the audio data to the server through the communication module 104, so that the server analyzes the operation information included in the video data and the audio data based on a pre-trained model, and determines the inspection result corresponding to each preset inspection item; receive the inspection result sent by the server through the communication module 104, and prompt the user based on the inspection result.

[0080] When the user is on duty, he can wear the duty recording terminal 10 as required, and send a recording instruction to the duty recording terminal 10 when starting the duty according to the operation requirements. In an embodiment of the present application, the user can send the recording instruction by pressing the physical button on the duty recording terminal 10, or by clicking a virtual button on the visual page of the duty recording terminal 10. After receiving the recording instruction, the duty recording terminal 10 can be considered that the user is ready to start the duty, and the processor 103 of the duty recording terminal 10 can control the camera 101 to collect video data of the duty operation. In order to accurately check the duty process, it can be stipulated that the user should capture the complete operation process as much as possible when performing each step of the operation. Since the user will not only perform corresponding actions with his hands during the duty process, but also explain each step, or it may involve communication between multiple people during the duty process, therefore, in an embodiment of the present application, the processor 103 can also control the microphone 102 to collect audio data of the operation instruction issued by the user. That is, when the processor 103 receives the recording instruction, it controls the camera 101 and the microphone 102 to collect audio data and video data.

[0081] In order to inspect the duty process, in an embodiment of the present application, the processor 103 can send the collected video data and audio data to the server through the communication module 104. In an embodiment of the present application, the collected video data and audio data can be sent to the server in real time, or the complete video data and audio data corresponding to this duty can be sent to the server after the duty task is completed. In other words, the video data and audio data received by the server can be real-time data or offline data. In an embodiment of the present application, the server can analyze the operation information included in the video data and audio data based on a pre-trained model to determine the inspection results corresponding to each preset inspection item. After obtaining the inspection results, the inspection results with abnormalities can be sent to the duty recording terminal 10, or the inspection results corresponding to all preset inspection items can be sent to the duty recording terminal 10.

[0082] In the embodiment of the present application, the duty record terminal 10 can receive the inspection results sent by the server through the communication module 104, and prompt the user based on the inspection results. In the embodiment of the present application, the user can be reminded of the existing problems in the form of voice broadcast, or the inspection results corresponding to each preset inspection item can be displayed on the display screen in the form of screen display.

[0083] In order to improve the accuracy of duty inspection, on the basis of the above embodiments, in the embodiment of the present application, the processor 103 is further configured to: if a photo taking instruction sent by the user is received, control the camera 101 to capture image data of the duty process; send the video data, the audio data and the image data to the server through the communication module 104, so that the server analyzes the duty information included in the video data, the audio data and the image data based on the pre-trained model, and determines the inspection result corresponding to each preset inspection item.

[0084] During the work process, users can record some important scenes by taking photos. For example, some jobs do not allow the use of mobile phones. Therefore, it is necessary to collect the mobile phones of each staff member before starting work. After collecting the mobile phones of all staff members, all mobile phones can be put together to take an image, so as to facilitate the subsequent counting of the number of mobile phones and retain evidence. Therefore, in the embodiment of the present application, if the processor 103 receives a photo taking instruction sent by the user, it can control the camera 101 to capture image data of the work process. After obtaining the image data, the image data can be sent to the server together with the video data and the audio data, so that the server analyzes the work information included in the video data, audio data and image data based on the pre-trained model, and determines the inspection result corresponding to each preset inspection item.

[0085] In order to accurately check the on-duty process, based on the above embodiments, in an embodiment of the present application, a server is further provided, which is configured to: if video data of on-duty operation and audio data of operation instructions are received, analyze the video data based on the first multimodal large model to obtain the operation description text contained in the video data;

[0086] Performing speech recognition on the audio data to obtain a description text of the operation instruction described in the audio data;

[0087] The operation description text, operation instruction description text and prompt words containing each preset detection item are input into the first large language model, and the inspection result corresponding to each preset inspection item output by the first large voice model is obtained and sent to the duty recording terminal.

[0088] In order to be able to accurately and quickly inspect the duty process, in an embodiment of the present application, if the server receives video data of the duty operation and audio data of the operation instructions sent by the duty recording terminal 10, it can analyze the video data and audio data to determine the inspection results corresponding to each preset inspection item.

[0089] Specifically, the video data can be analyzed based on the first multimodal large model to obtain the operation description text contained in the video data. That is, the content included in the video data is described using the first multimodal large model. The first multimodal large model can be any multimodal large model with video understanding capabilities. At the same time, speech recognition is performed on the audio data to obtain the operation instruction description text described in the audio data. Exemplarily, the audio data can be input into a speech recognition model to convert the audio data into text, which is the instruction description text.

[0090] After obtaining the operation description text and the instruction description text, the operation description text, the operation instruction description text and the prompt words containing each preset detection item can be input into the first language large model, so that the first language large model analyzes the overview text and the operation instruction description text based on the requirements in the prompt words, and then obtains the inspection results corresponding to each preset inspection item. In other words, the prompt words are prompt texts used to prompt the first language large model to analyze the video data and audio data, and determine the inspection results corresponding to each preset inspection item. The first language large model can be any large model with semantic understanding capabilities. Exemplarily, the prompt words can be "Please analyze the following two paragraphs of text to determine whether the on-duty content of the walkie-talkie test, alcohol test, and collection of staff mobile phones has been completed during the duty process." Technical personnel in this field can configure the content of the prompt words as needed.

[0091] In order to let the user know the current inspection result, after obtaining the inspection result corresponding to each preset inspection item, the server can send the inspection result to the duty recording terminal 10.

[0092] In the embodiment of the present application, the video data content is structured and output in the form of text through the video text multimodal model, and the structured content includes but is not limited to the behavior information that you want to detect. For example, collecting mobile phones, alcohol testing, etc. At the same time, the audio information is also obtained through speech recognition. Then, the language model is used to comprehensively judge whether the behavior specifications are met based on the structured information extracted from the video and the text information extracted from the audio.

[0093] Figure 2 A schematic diagram of a duty inspection process provided in an embodiment of the present application, such as Figure 2 As shown, the acquired video is input into the multimodal large model to obtain the operation description text. The audio is input into the speech recognition model to obtain the operation instruction description text. The operation description text, the operation instruction description text and the prompt words containing each preset inspection item are input into the language large model to obtain the inspection result corresponding to each preset inspection item. Among them, the preset inspection items include walkie-talkie status, alcohol test, and receiving mobile phones.

[0094] In order to further improve the accuracy of the previous inspection, based on the above embodiments, in the embodiment of the present application, if the image data of the duty process is received, it is also configured as follows:

[0095] Analyzing the image data based on the first multimodal large model to obtain other description texts corresponding to the content recorded in the image data;

[0096] The operation description text, the operation instruction description text, the other description texts and the prompt words containing each preset detection item are input into the first large language model to obtain the inspection result corresponding to each preset inspection item output by the first large speech model.

[0097] If the image data sent by the duty recording terminal 10 is also received, in the embodiment of the present application, the image data can also be analyzed based on the first multimodal large model to obtain other description texts corresponding to the content recorded in the image data. After obtaining the other description texts corresponding to the image data, the operation description text, the operation instruction description text, the other description texts and the prompt words containing each preset detection item can be input into the first language large model, so that the first language large model analyzes the content of the operation description text, the operation instruction description text, and the other description text based on the requirements of the prompt words, and obtains the inspection results corresponding to each preset inspection item output by the first large voice model.

[0098] In order to further accurately inspect the duty process, on the basis of the above embodiments, in the embodiment of the present application, a server is further provided, which is configured to, if video data of duty operation and audio data of operation instructions are received, input the video data, the audio data and the prompt words containing each preset detection item into the second multimodal large model, and the encoder of the second multimodal large model extracts the features of the video data, the audio data and the prompt words to obtain a first feature matrix of the video data, a second feature matrix of the audio data and a third feature matrix of the prompt words;

[0099] The language large model of the second multimodal large model analyzes the first feature matrix, the second feature matrix and the third feature matrix to determine the inspection result corresponding to each preset inspection item.

[0100] Because the first multimodal large model trained based on general data may have a low accuracy rate when analyzing video data or audio data in a proprietary field. In an embodiment of the present application, in order to further accurately inspect the on-duty process, the received video data and audio data can be processed based on the second multimodal large model. The second multimodal large model can be after fine-tuning training is completed, and the second multimodal large model can have a higher accuracy rate when inspecting the on-duty process.

[0101] Specifically, if video data of on-duty operation and audio data of operation instructions are received, the video data, audio data and prompt words containing each preset detection item can be input into the second multimodal large model. The second multimodal large model can use an encoder to extract the features of the video data, audio data and prompt words to obtain a first feature matrix of video data, a second feature matrix of audio data and a third feature matrix of prompt words. In an embodiment of the present application, the encoder for feature extraction can be a general encoder, which can extract features of video data, audio data and prompt words. In other words, an encoder is deployed in the second multimodal large model, and the encoder can extract features from the received data. Since the data characteristics of different types of data are different, in an embodiment of the present application, when each data is input into the encoder, the identifier of the data type of the corresponding data can be input into the encoder together, so that the encoder can select the corresponding feature extraction method according to the identifier to extract features from the corresponding data.

[0102] After obtaining the first feature matrix, the second feature matrix, and the third feature matrix, the first feature matrix, the second feature matrix, and the third feature matrix can be input into the language model of the second multimodal large model, so that the language model analyzes the first feature matrix, the second feature matrix, and the third feature matrix to obtain the inspection result corresponding to each preset inspection item. How the language model in the multimodal large model analyzes the received data is a prior art, and the embodiment of the present application will not repeat this process.

[0103] In order to further improve the accuracy of duty inspection, based on the above embodiments, in the embodiment of the present application, if image data of the duty process is received, it is specifically configured as follows:

[0104] The video data, the audio data, the image data and the prompt words containing each preset detection item are input into a second multimodal large model, and an encoder of the second multimodal large model extracts features of the video data, the audio data, the image data and the prompt words to obtain a first feature matrix of the video data, a second feature matrix of the audio data, a third feature matrix of the prompt words and a fourth feature matrix of the image data;

[0105] The language large model of the second multimodal large model analyzes the first feature matrix, the second feature matrix, the third feature matrix and the fourth feature matrix to determine the inspection result corresponding to each preset inspection item.

[0106] If the image data sent by the duty recording terminal 10 is also received, in the embodiment of the present application, when the second multimodal large model is used for analysis, the image data can be input into the second multimodal large model together with the video data, audio data and prompt words. The encoder of the second multimodal large model can extract the features of the frequency data, audio data, image data and prompt words to obtain the first feature matrix of the video data, the second feature matrix of the audio data, the third feature matrix of the prompt words and the fourth feature matrix of the image data. After obtaining the feature matrix corresponding to each data, the language large model of the second multimodal large model can analyze the first feature matrix, the second feature matrix, the third feature matrix and the fourth feature matrix to determine the inspection results corresponding to each preset inspection item.

[0107] In order to further improve the accuracy of duty inspection, based on the above embodiments, in the embodiment of the present application, it is specifically configured as follows:

[0108] The image encoder of the second multimodal large model performs feature extraction on the video data and the image data to obtain a first feature matrix of the video data and a fourth feature matrix of the image data;

[0109] The audio encoder of the second multimodal large model performs feature extraction on the audio data to obtain a second feature matrix of the audio data;

[0110] The text encoder of the second multimodal large model performs feature extraction on the prompt word to obtain a third feature matrix of the prompt word.

[0111] In order to more accurately extract the features of the received data, in an embodiment of the present application, multiple encoders can be deployed in the second multimodal large model, which are image encoders, audio encoders, and text encoders. When the encoder extracts the received data, the image encoder of the second multimodal large model can extract features from the video data and the image data to obtain a first feature matrix of the video data and a fourth feature matrix of the image data. The audio encoder of the second multimodal large model extracts features from the audio data to obtain a second feature matrix of the audio data. The text encoder of the second multimodal large model extracts features from the received prompt words to obtain a third feature matrix of the prompt words. How the encoder extracts features from the received data is a prior art, and the embodiment of the present application will not go into details about the word process.

[0112] In a possible implementation, some images or image frames may contain text information, which is helpful for inspection. For example, the text information is a car number, a key event, etc. Therefore, in the embodiment of the present application, the character information in the image can be recognized by optical character recognition (OCR), and after the recognition is completed, the text information and the corresponding image can be input into the second multimodal large model.

[0113] Figure 3 A schematic diagram of a duty inspection process provided in an embodiment of the present application, such as Figure 3 As shown, after acquiring the video data and audio data, the video data and audio data are encoded (patch embedding) to obtain content that can be recognized by the electronic device, and the video encoder is used to extract features from the video data to obtain a first feature matrix. The audio encoder is then used to extract features from the audio data to obtain a second feature matrix. After receiving the prompt word, the prompt word is encoded (embedding) for feature extraction to obtain content that can be recognized by the electronic device, and the text encoder is used to extract features from the received prompt word to obtain a third feature matrix of the prompt word. After obtaining the first feature matrix, the second feature matrix, and the third feature matrix, modal alignment is performed, that is, feature mapping is performed to obtain a feature matrix that meets the input requirements of the language model.

[0114] In order to further improve the efficiency of the duty process inspection, based on the above embodiments, in the embodiment of the present application, it is also configured as follows:

[0115] Identify the targets included in the video data based on the target recognition model, and determine the video segment corresponding to each target according to the image frame where each identified target is located;

[0116] According to the detection target corresponding to each preset detection item, the video data is updated using the video segment corresponding to the detection target.

[0117] Since the video content is lengthy, in order to speed up the processing of video data, the timing information of the occurrence of events is accurately located. And the content involved in different duty stages is different. For example, the core targets involved in the business inspection before departure are: walkie-talkies, alcohol testers, and mobile phones. For the inspection of in-warehouse operations, the core targets involved in the main inspection items are: car doors, windows, fire extinguishers, water heaters, emergency equipment, medicine boxes, etc. Therefore, in an embodiment of the present application, the targets included in the video data can be identified based on the target recognition model. After each target is detected, each detected target can be used as a segmentation node. When segmenting the video data, the video segment corresponding to each target can be determined based on the image frame where each identified target is located.

[0118] Specifically, assuming that a car door appears from the 20th second to the 35th second in the video data, and a water heater appears from the 38th second to the 55th second in the video data, then the video segment corresponding to the 20th second to the 35th second in the video data can be determined as the video segment corresponding to the car door, and the video segment corresponding to the 38th second to the 55th second in the video data can be determined as the video segment corresponding to the water heater.

[0119] After obtaining the video segments corresponding to each target, the video data can be updated according to the detection targets corresponding to each preset inspection item using the video segments corresponding to the detection targets. In other words, the redundant information in the video data is deleted, and the subsequent multimodal large model only processes the video segments of the detection targets corresponding to the detection items, thereby improving the detection efficiency.

[0120] In the embodiment of the present application, by using the core elements located by the small model, the main video clips where the event occurred can be quickly located, and then the large model is used to analyze the standardization and rationality of the on-duty behavior to achieve operational inspection of the scene.

[0121] In order to further improve the efficiency of the duty process inspection, based on the above embodiments, in the embodiment of the present application, it is also configured as follows:

[0122] Recognize the operation instructions included in the initial audio data, and determine the audio segment corresponding to each operation content according to the time corresponding to each recognized operation instruction;

[0123] According to the target operation content corresponding to each preset detection item, the audio data is updated using the audio segment corresponding to the target operation content.

[0124] During the duty process, relevant personnel will also explain the duty process. For example, the train conductor will generally issue work instructions before special operations, such as "XX station is here, and the sleeper car is now on gate duty", "XXXX car transaction completed, collect the transition plate and warning tape, XXXXX." In an embodiment of the present application, the work instructions included in the initial audio data can be identified, and the audio segment corresponding to each work content can be determined according to the time corresponding to each identified work instruction. Then, according to the target work content corresponding to each preset detection item, the audio data is updated using the audio segment corresponding to the target work content. In other words, the redundant information in the audio data is deleted, and the subsequent multimodal large model only processes the audio segment of the detection target corresponding to the detection item, thereby improving the detection efficiency.

[0125] In a possible implementation, the train arrival time can be accurately located according to the train arrival schedule, corresponding to the recorder shooting time, to achieve the video node positioning of the train arrival operation. The train arrival schedule can include a stop schedule and a departure and arrival schedule. Many operations are completed before and after arrival, and the video clip can be quickly located in combination with the stop schedule. Similarly, many operations are completed before or after departure and arrival, and the departure and arrival schedule can be used to locate the video clip more quickly.

[0126] By using a small model, audio, and prior knowledge positioning method, and inputting video segments containing core elements, audio, and arrival time into a large model for understanding, accurate response to business scenarios can be achieved quickly.

[0127] In the embodiment of the present application, the time information is obtained through OCR recognition through a small model, audio, prior knowledge (timetable) and image time watermark, and the video clip is quickly located to judge the corresponding behavior norms. For example, the instructions issued by the train conductor can be quickly located in combination with the audio information, and the video screen that needs to complete a certain safety inspection can be quickly located according to the instructions to judge whether the relevant operation has been completed.

[0128] Figure 4 A schematic diagram of a duty inspection process provided in an embodiment of the present application, such as Figure 4As shown in the figure, after obtaining the video data and audio data, the small model is used to locate the key elements in the initial video data, split the initial video into multiple video segments, and identify the initial audio data to locate the key elements. Subsequently, the video segments, audio segments, and image data corresponding to the preset inspection items are input into the multimodal large model. The multimodal large model processes the received data in the same way as Figure 3 The processing process is similar and will not be repeated in the embodiments of the present application.

[0129] In order to further improve the accuracy of the duty process inspection, on the basis of the above embodiments, in the embodiment of the present application, the second multimodal large model is obtained based on fine-tuning training; the training process of the second multimodal large model includes:

[0130] Obtain a preset number of sample pair data in a training set, wherein any sample pair data includes text data and non-text data, and the text data is an image, audio, or video;

[0131] Input each of the sample pair data into the original multimodal large model, and respectively determine a feature matrix corresponding to the text data and a feature matrix corresponding to the non-text data in the sample pair data;

[0132] Determine the loss value based on each feature matrix and loss function corresponding to each sample pair data;

[0133] According to the loss value and the current number of training iterations, the parameters in the original multimodal large model are adjusted to obtain the second multimodal large model.

[0134] In order to accurately inspect the duty process, in the embodiment of the present application, the second multimodal large model used can be obtained based on fine-tuning training. In the embodiment of the present application, video, audio, and text information can be encoded into a unified feature space, and the alignment of video and text, audio and text in the feature space can be achieved through comparative learning, and the alignment of audio and video in the feature space can be indirectly achieved, and then the audio, video and text tokens are combined and input into the language large model for judgment of relevant behavioral norms. In order to fine-tune the second multimodal large model, in the embodiment of the present application, a training set can be pre-configured, and the training set can include text data and non-text data, wherein the text data can be an image, audio or video. That is, the training set includes video-text pairs, image-text pairs, and audio-text pairs. Among them, the text in the video-text pair mainly describes the content in the video in detail, focusing on the description of the environment, the behavior of the train attendant captured by the screen, and the action. The text in the image-text pair mainly describes the content in the image in detail, focusing on the description of the environment, the behavior of the train attendant captured by the screen, and the action. The text in the audio-text pair mainly describes the content included in the audio, and exemplarily, the text can be an overview text. For example, the conductor issues an order, "We have arrived at XX station, sleeper cars are now on guard duty", and the corresponding text for this audio can be "Original operation inspection - conductor sends and receives instructions - arrives at the station to notify the conductor to get off the station to guard duty". "The transaction of XXXX'' car is completed, collect the transition board and warning tape, XXXXX." The corresponding text for this audio can be "Original operation inspection - conductor sends and receives instructions - the transaction is completed and the conductor is notified to collect the warning tape and safety pedal".

[0135] During training, a preset number of sample pair data can be obtained in the training set. Among them, the preset number can be any integer such as 10, 20, 50, 200, etc. After obtaining the preset number of sample pair data, each sample pair data can be input into the original multimodal large model to determine the feature matrix corresponding to the text data in the sample pair data and the feature matrix corresponding to the non-text data. After obtaining the feature matrix of each data, the loss value will be determined according to each feature matrix corresponding to each sample pair data and the loss function. After obtaining the loss value, the parameters in the original multimodal large model can be adjusted according to the loss value and the current number of training iterations to obtain the second multimodal large model. For example, during the first 500 trainings, the parameters in the encoder in the second multimodal large model are adjusted according to the loss value, and all parameters in the second multimodal large model are adjusted according to the loss value from the 501st to the 1000th time. It should be noted that technicians in this field can configure parameter adjustment strategies as needed.

[0136] Specifically, assume that the image modality data is I and the text modality data is T. At this time, a preset number of sample pairs (I, T) are obtained, for example, the preset number is 10, that is, 10 image-text pairs. After obtaining the feature matrix corresponding to each text data and non-text data in the 10 image-text pairs, the loss value is determined based on the following InfoNCE loss function:

[0137] L=L I，T +L T，I

[0138] in, Among them, i≠j;

[0139] Among them, i≠j.

[0140] Among them, t, represents the temperature coefficient, q i represents the feature matrix corresponding to the non-text data of any target sample pair in the above 10 sample pairs, k i Represents the feature matrix corresponding to the text data of the target sample pair, k j Represents the feature matrix corresponding to the text data in the non-target sample pair data; q j Represents the feature matrix corresponding to the non-text data in the non-target sample pair data; n represents the preset number.

[0141] This loss function makes the embedding q i and k i Closer in the joint embedding space, thus aligning I and T. In practice, a symmetric loss function L is used I，T +L T，I Construct samples of (I, T), (V, T) and (A, T) to train the data, so that the video, image and audio are aligned to the text space, and the image, video and audio are indirectly aligned.

[0142] In this embodiment of the present application, a modality-specific linear projection head is added to each encoder to obtain a fixed-size d-dimensional embedding, which is normalized and used for the above-mentioned InfoNCE loss function. In addition to being easy to learn, this setting also allows the use of pre-trained CLIP or OpenCLIP image and text encoders.

[0143] The construction process of the training set is described below in conjunction with a specific embodiment. When constructing the training set, the railway scene data can be crawled and annotated through the network. At present, the Audioset dataset contains naturally paired (video, audio) data. In an embodiment of the present application, image frames can be extracted from the video of the data set as images in the database. In an embodiment of the present application, GPT4o can be used to analyze the content of each video (video) respectively to obtain the text corresponding to the video, thereby obtaining a video-text pair. GPT4o can also be used to analyze the content of the image frame extracted from the video to obtain the text corresponding to the image frame, thereby obtaining an image-text pair. The audio can also be processed using speech recognition technology to obtain the text corresponding to the audio, thereby obtaining an audio-text pair. In an embodiment of the present application, 128 mel-spectrogram bins can be used to convert n seconds of audio sampled at 16kHz.

[0144] In the embodiment of the present application, the training set may also include training data for instruction fine-tuning tasks. Based on these training data, the second multimodal large model may be trained to have the ability to search in text, audio, video, and images. The training data is as follows:

[0145] a. Find the corresponding video clip based on the audio. For example: given the audio detected by the recorder and multiple end videos, find the corresponding video clip based on the given audio.

[0146] b. Find the corresponding audio based on the video clip. For example: given the video detected by the recorder and the multi-end audio, find the corresponding audio clip based on the given video.

[0147] c. Find the corresponding audio based on the text.

[0148] d. Find the appropriate video clips based on the text.

[0149] E. Find the corresponding text based on the audio.

[0150] F. Which of the following audio clips are about alcohol testing?

[0151] G. The following video clips are of walkie-talkie testing.

[0152] H. Image retrieval: During the inspection of doors and windows, the driver cannot speak and the car environment is noisy, so audio is not very useful. However, the inspection action characteristics of doors and windows are very obvious, which can be realized through image retrieval. We evenly extract 2 frames per second for image retrieval, and retrieve the action of checking the door lock in the video.

[0153] The second multimodal large model is trained to have the ability to perform inspection based on audio data. For example:

[0154] Given a voice: Detect the person on duty wearing a red vest in the picture. Answer: [25,26,86,78].

[0155] Given a voice: Detect the person holding the alcohol test in the picture. Answer: [25,26,86,78].

[0156] The second multimodal large model is trained to have the ability to generate references based on audio. For example:

[0157] Given the audio: Describe what the on-duty personnel [25,26,86,78] are doing in the picture?

[0158] Given an audio file: Describe what happens in the [25,26,86,78] region of the image?

[0159] The above [25, 26, 86, 78] can be understood as the coordinate information of the corresponding rectangular box.

[0160] The second multimodal large model is trained to have the ability to describe based on vision. For example:

[0161] What happened in this video? Answer: The on-duty staff in the video is conducting a pre-departure inspection, including testing the intercom, alcohol testing, and collecting the staff's mobile phones.

[0162] By using the above tasks to fine-tune the large multimodal model, the model can be equipped with the ability to recognize and understand the on-duty behavior in the scenario.

[0163] The following describes the on-duty inspection process in conjunction with a specific embodiment. Figure 5 A schematic diagram of the structure of a second multimodal large model provided in an embodiment of the present application is shown in FIG. Figure 5As shown, one or more data of images, videos, audios and texts can be input into the second multimodal large model. When encoding the received data, the second multimodal large model can use an independent modality encoder for encoding or a modality universal encoder for encoding. The independent modality encoder can be understood as different types of data being encoded using different encoders. For example, image encoders can be used for encoding image and video data, audio encoders can be used for encoding audio data, and text encoders can be used for encoding text data. The modality universal encoder can be understood as using the same encoder for encoding regardless of the type of data. In order to allow the encoder to know the type of data currently being processed, the data can be identified by a modality marker. After the encoder obtains the feature matrix corresponding to each data, modality alignment can be performed. Modality alignment can be understood as feature mapping the feature matrix obtained by the encoder to obtain a feature matrix that meets the input requirements of the language large model. When performing modality alignment, linear mapping, cross attention, or Q-Former and other methods can be used for processing, and of course, a general projection module can also be used for processing. After obtaining the feature matrix after modal alignment, the prompt words and features will be processed using the language model. Among them, the language model can be understood as a language model backbone network, which can be a Transformer architecture or a Mamba structure, and the embodiment of the present application does not limit this. The language model can process the received features according to the requirements in the prompt words, so as to obtain the inspection results corresponding to each preset inspection item. For example, check whether the conductor's pre-departure operations, in-depot operations, initial operations, post-start operations, stop operations, and final arrival operations meet the requirements. In the embodiment of the present application, by integrating video, voice, and text information, a more comprehensive intelligent inspection of the on-duty work of railway crew members is achieved, thereby improving the accuracy and efficiency of the inspection.

[0164] Taking the train conductor's duty scenario as an example, the above-mentioned duty process inspection method can inspect the train conductor's work conditions before departure, work conditions in the depot, work conditions at the start, work conditions after starting, work conditions at the stop station, and work conditions at the final (return) station.

[0165] The pre-departure inspection may include the following:

[0166] Intercom status check: Identify the status of the intercom through video data, and identify the test audio of the intercom with voice data to determine whether the intercom is working properly.

[0167] Alcohol test: Use video data to identify the use process of the alcohol tester, and combine voice data to identify the test results to determine whether the crew member has passed the alcohol test.

[0168] Put away the mobile phone: Use video data to identify whether the crew has put away the mobile phone and record the relevant time information.

[0169] Inspections of warehouse operations may include the following:

[0170] Carriage patrol: Use video data to identify the crew's patrol path in the carriage to ensure that the patrol covers all areas.

[0171] Check doors and windows: Use video data to identify crew members checking doors and windows to ensure they are closed.

[0172] Spot check of fire extinguishers: Use video data to identify crew members’ actions of checking fire extinguishers to ensure that the fire extinguishers are within the validity period.

[0173] Water heater inspection: Use video data to identify crew members’ actions in inspecting water heaters to ensure that the water heaters are functioning properly.

[0174] Emergency equipment (flashlight): Use video data to identify crew members checking emergency flashlights to ensure that the flashlights are working properly.

[0175] Emergency medicine box inspection: Use video data to identify crew members checking emergency medicine boxes to ensure that all medicines are in the box.

[0176] Inspection of the originating operation may include the following:

[0177] Whether the conductor has issued an instruction: The conductor's instructions are identified through audio data, and the conductor's actions are identified through video data to confirm whether the instruction has been issued.

[0178] Whether the warning tape is retracted: Identify the status of the warning tape through video data to determine whether it is retracted.

[0179] Whether the safety pedal is retracted: The status of the safety pedal is identified through video data to determine whether it is retracted.

[0180] The inspection of the operation after driving may include the following:

[0181] Door closed status: Identify the status of the door through video data to ensure that the door is closed.

[0182] Lock and plug sliding doors: Use video data to identify the crew's actions of using keys to lock and plug sliding doors to ensure that the doors are locked.

[0183] Photographing at the end of a carriage: The camera uses video data to identify the crew member’s action of taking a photo at the end of a carriage, and records the carriage number and the crew member’s facial information. This is performed every 2 hours.

[0184] Inspections of stop operations may include the following:

[0185] Check whether the conductor has issued an instruction: Use voice data to identify the conductor's instructions, and combine video data to identify the conductor's actions to confirm whether the instruction has been issued.

[0186] Whether the warning tape is retracted: Identify the status of the warning tape through video data to determine whether it is retracted.

[0187] Whether the safety pedal is retracted: Identify the status of the safety pedal through video data to determine whether it is retracted.

[0188] Inspection of terminal (return) operations may include the following:

[0189] Is the dining car door locked? Use video data to identify the crew member's action of pulling the dining car door to ensure that the dining car door is locked.

[0190] Induction cooker off: The crew's action of turning off the induction cooker is identified through video data to ensure that the induction cooker is turned off.

[0191] Is the refrigerator empty? Use video data to identify the crew's actions of opening the refrigerator to ensure that the refrigerator is empty.

[0192] It should be noted that the above-described detection process is applicable to all scenarios where the behavioral norms of personnel need to be supervised. This application only uses the duty inspection in the railway scenario as an example to illustrate the idea of ​​the solution.

[0193] This application has the following advantages:

[0194] 1. Deep integration with railway scenario business.

[0195] This application takes the on-duty inspection work of railway crew members as an example, and designs special inspection items and algorithm strategies to ensure the applicability and effectiveness of the method. Through an in-depth understanding of the business needs and operating procedures of the railway scenario, detailed inspection steps have been formulated, including pre-departure operation inspection, in-depot operation inspection, departure operation, post-start operation, stop operation and final (return) station operation. Each inspection item is combined with the specific requirements of the railway scenario, ensuring the comprehensiveness and accuracy of the inspection process, and improving the safety and efficiency of railway operations.

[0196] 2. Integrate video, voice and text information.

[0197] This application achieves a comprehensive inspection of railway crew members' on-duty work by integrating multimodal data, including video, voice and text information. Video information is used to identify crew members' movements and equipment status, voice information is used to identify instructions and conversation content, and text information is used to record important information such as key times and car numbers. The fusion of multimodal data makes the inspection process more comprehensive and accurate, can effectively capture and analyze various information, and improves the robustness and reliability of the inspection.

[0198] 3. Accurate, efficient and safe.

[0199] Through the automated intelligent inspection method, the accuracy and efficiency of the inspection are greatly improved. Traditional manual inspection methods are prone to omissions and errors, but this application uses intelligent algorithm strategies to make real-time judgments and verifications on each step of the inspection, ensuring the accuracy of the inspection results. At the same time, the automated inspection process reduces manual operations, improves work efficiency, and allows crew members to focus more on other important tasks. In addition, the system can output inspection results in real time, discover and solve problems in a timely manner, and further improve the safety and reliability of railway operations. The safety of railway operations is ensured through real-time monitoring and verification of key inspection items.

[0200] 4. The camera is shaking.

[0201] The mobile camera's perspective causes severe shaking in the picture, making scene recognition more difficult, and it is difficult for small models to achieve accurate recognition. The information fusion solution of multiple models can accurately identify target behaviors and events.

[0202] Based on the same inventive concept, the embodiment of the present application also provides a duty process inspection device based on a multi-modal large model. Figure 6 A schematic diagram of the structure of a duty process inspection device based on a multimodal large model provided in an embodiment of the present application, the device includes:

[0203] The content extraction module 601 is used for, if the video data of the on-duty operation and the audio data of the operation instruction are received, analyzing the video data based on the first multimodal large model to obtain the operation description text contained in the video data; performing speech recognition on the audio data to obtain the operation instruction description text described in the audio data;

[0204] The analysis module 602 is used to input the operation description text, operation instruction description text and the prompt words containing each preset detection item into the first large language model, obtain the inspection result corresponding to each preset inspection item output by the first large voice model, and send it to the duty recording terminal.

[0205] In a possible implementation manner, if image data of the on-duty process is received, the content extraction module 601 is further used to analyze the image data based on the first multimodal large model to obtain other descriptive texts corresponding to the content recorded in the image data;

[0206] The analysis module 602 is also used to input the operation description text, the operation instruction description text, the other description texts and the prompt words containing each preset detection item into the first large language model, and obtain the inspection result corresponding to each preset inspection item output by the first large speech model.

[0207] Figure 7 A schematic diagram of the structure of a duty process inspection device based on a multimodal large model provided in an embodiment of the present application, the device includes:

[0208] The feature extraction module 701 is used for, if the video data of the on-duty operation and the audio data of the operation instruction are received, inputting the video data, the audio data and the prompt words containing each preset detection item into the second multimodal large model, and the encoder of the second multimodal large model extracts the features of the video data, the audio data and the prompt words to obtain a first feature matrix of the video data, a second feature matrix of the audio data and a third feature matrix of the prompt words;

[0209] The analysis module 702 is used for the language large model of the second multimodal large model to analyze the first feature matrix, the second feature matrix and the third feature matrix to determine the inspection result corresponding to each preset inspection item.

[0210] In a possible implementation, if image data of a duty process is received, the feature extraction module 701 is specifically used to input the video data, the audio data, the image data and the prompt words containing each preset detection item into a second multimodal large model, and the encoder of the second multimodal large model extracts features of the video data, the audio data, the image data and the prompt words to obtain a first feature matrix of the video data, a second feature matrix of the audio data, a third feature matrix of the prompt words and a fourth feature matrix of the image data;

[0211] The analysis module 702 is specifically used for the language large model of the second multimodal large model to analyze the first feature matrix, the second feature matrix, the third feature matrix and the fourth feature matrix to determine the inspection result corresponding to each preset inspection item.

[0212] In a possible implementation, the feature extraction module 701 is specifically used for the image encoder of the second multimodal large model to perform feature extraction on the video data and the image data to obtain a first feature matrix of the video data and a fourth feature matrix of the image data; the audio encoder of the second multimodal large model to perform feature extraction on the audio data to obtain a second feature matrix of the audio data; and the text encoder of the second multimodal large model to perform feature extraction on the prompt word to obtain a third feature matrix of the prompt word.

[0213] In a possible implementation, the device further includes:

[0214] The screening and updating module 703 is used to identify the targets included in the video data based on the target recognition model, and determine the video segment corresponding to each target according to the image frame where each identified target is located; according to the detection target corresponding to each preset detection item, the video data is updated using the video segment corresponding to the detection target.

[0215] In a possible implementation, the screening and updating module 703 is also used to identify the job instructions included in the initial audio data, and determine the audio segment corresponding to each job content based on the time corresponding to each identified job instruction; and based on the target job content corresponding to each preset detection item, the audio data is updated using the audio segment corresponding to the target job content.

[0216] In a possible implementation, the device further includes:

[0217] The training module 704 is used to obtain a preset number of sample pair data in the training set, any sample pair data includes text data and non-text data, and the text data is an image, audio or video; each of the sample pair data is input into the original multimodal large model, and the feature matrix corresponding to the text data and the feature matrix corresponding to the non-text data in the sample pair data are determined respectively; the loss value is determined according to each feature matrix corresponding to each sample pair data and the loss function; the parameters in the original multimodal large model are adjusted according to the loss value and the current number of training iterations to obtain the second multimodal large model.

[0218] Based on the same inventive concept, an embodiment of the present application further provides an electronic device, Figure 8 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application is shown in FIG. Figure 8 As shown, it includes: a processor 801, a communication interface 802, a memory 803 and a communication bus 804, wherein the processor 801, the communication interface 802, and the memory 803 communicate with each other through the communication bus 804;

[0219] The memory 803 stores a computer program. When the program is executed by the processor 801, the processor 801 executes the steps of the on-duty process inspection method based on the multimodal large model described in the above embodiments. The on-duty process inspection method based on the multimodal large model is similar to the process executed by the above servers, and the embodiments of this application will not be repeated.

[0220] The communication bus mentioned in the above electronic device can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The communication bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, only one thick line is used in the figure, but it does not mean that there is only one bus or one type of bus. The communication interface 802 is used for communication between the above electronic device and other devices. The memory may include a random access memory (RAM) and may also include a non-volatile memory (NVM), such as at least one disk storage. Optionally, the memory may also be at least one storage device located away from the aforementioned processor.

[0221] The above-mentioned processor can be a general-purpose processor, including a central processing unit, a network processor (Network Processor, NP), etc.; it can also be a digital signal processing processor (Digital Signal Processing, DSP), an application-specific integrated circuit, a field programmable gate array or other programmable logic device, a discrete gate or transistor logic device, a discrete hardware component, etc.

[0222] On the basis of the above embodiments, an embodiment of the present invention further provides a computer-readable storage medium, which stores a computer program executable by a processor. When the program runs on the processor, the processor implements the steps of the duty process inspection method based on a multimodal large model described in the above embodiments.

[0223] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that include computer-usable program code.

[0224] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0225] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0226] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process in the computer or other programmable device. Figure 1 A process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0227] Obviously, those skilled in the art can make various changes and modifications to the present application without departing from the spirit and scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the claims of the present application and their equivalents, the present application is also intended to include these modifications and variations.

Claims

1. A duty recording terminal, characterized in that: The duty recording terminal includes a camera, a microphone, a processor, and a communication module: The processor is configured to: if a recording instruction sent by a user is received, control the camera to collect video data of the on-duty operation, and control the microphone to collect audio data of the operation instruction issued by the user; The collected video data and the audio data are sent to the server through the communication module, so that the server analyzes the operation information included in the video data and the audio data based on the pre-trained model to determine the inspection result corresponding to each preset inspection item; The inspection result sent by the server is received through the communication module, and the user is prompted based on the inspection result.

2. The terminal according to claim 1, characterized in that The processor is further configured to: if a photo taking instruction sent by a user is received, control the camera to take image data of the on-duty process; The video data, the audio data and the image data are sent to the server through the communication module, so that the server analyzes the duty information included in the video data, the audio data and the image data based on a pre-trained model to determine the inspection result corresponding to each preset inspection item.

3. A server, characterized in that: is configured as: If video data of the on-duty operation and audio data of the operation instruction are received, the video data is analyzed based on the first multimodal large model to obtain the operation description text contained in the video data; Performing speech recognition on the audio data to obtain a description text of the operation instruction described in the audio data; The operation description text, operation instruction description text and prompt words containing each preset detection item are input into the first large language model, and the inspection result corresponding to each preset inspection item output by the first large voice model is obtained and sent to the duty recording terminal.

4. The server according to claim 3, characterized in that: If image data of the on-duty process is received, it is also configured as follows: Analyzing the image data based on the first multimodal large model to obtain other description texts corresponding to the content recorded in the image data; The operation description text, the operation instruction description text, the other description texts and the prompt words containing each preset detection item are input into the first large language model to obtain the inspection result corresponding to each preset inspection item output by the first large speech model.

5. A server, characterized in that: is configured as: If video data of on-duty operation and audio data of operation instructions are received, the video data, the audio data and the prompt words containing each preset detection item are input into the second multimodal large model, and the encoder of the second multimodal large model extracts features of the video data, the audio data and the prompt words to obtain a first feature matrix of the video data, a second feature matrix of the audio data and a third feature matrix of the prompt words; The language large model of the second multimodal large model analyzes the first feature matrix, the second feature matrix and the third feature matrix to determine the inspection result corresponding to each preset inspection item.

6. The server according to claim 5, characterized in that: If image data of the on-duty process is received, it is specifically configured as follows: The video data, the audio data, the image data and the prompt words containing each preset detection item are input into a second multimodal large model, and an encoder of the second multimodal large model extracts features of the video data, the audio data, the image data and the prompt words to obtain a first feature matrix of the video data, a second feature matrix of the audio data, a third feature matrix of the prompt words and a fourth feature matrix of the image data; The language large model of the second multimodal large model analyzes the first feature matrix, the second feature matrix, the third feature matrix and the fourth feature matrix to determine the inspection result corresponding to each preset inspection item.

7. The server according to claim 6, characterized in that: Specifically configured as: The image encoder of the second multimodal large model performs feature extraction on the video data and the image data to obtain a first feature matrix of the video data and a fourth feature matrix of the image data; The audio encoder of the second multimodal large model performs feature extraction on the audio data to obtain a second feature matrix of the audio data; The text encoder of the second multimodal large model performs feature extraction on the prompt word to obtain a third feature matrix of the prompt word.

8. The server according to claim 5, characterized in that: Also configured as: Identify the targets included in the video data based on the target recognition model, and determine the video segment corresponding to each target according to the image frame where each identified target is located; According to the detection target corresponding to each preset detection item, the video data is updated using the video segment corresponding to the detection target.

9. The server according to claim 5, characterized in that: Also configured as: Recognize the operation instructions included in the initial audio data, and determine the audio segment corresponding to each operation content according to the time corresponding to each recognized operation instruction; According to the target operation content corresponding to each preset detection item, the audio data is updated using the audio segment corresponding to the target operation content.

10. The server according to claim 5, characterized in that The second multimodal large model is obtained based on fine-tuning training; the training process of the second multimodal large model includes: Obtain a preset number of sample pair data in a training set, wherein any sample pair data includes text data and non-text data, and the text data is an image, audio, or video; Input each of the sample pair data into the original multimodal large model, and respectively determine a feature matrix corresponding to the text data and a feature matrix corresponding to the non-text data in the sample pair data; Determine the loss value based on each feature matrix and loss function corresponding to each sample pair data; According to the loss value and the current number of training iterations, the parameters in the original multimodal large model are adjusted to obtain the second multimodal large model.

Images