Embodied intelligence-based mechanical engineering practical training scoring system, method and device

Abstract

The application relates to the field of artificial intelligence technology, and provides a mechanical engineering practical training scoring system, method and device based on embodied intelligence. The device comprises an image data acquisition and preprocessing module, a human target position detection module, a human joint recognition and posture estimation module, a human action recognition module and a report generation module; the image data acquisition and preprocessing module is used for standardizing original image and video data, and outputs action sequence data with unified formats; the human target position detection module is used for marking target frames of a human body and components; the human joint recognition and posture estimation module is used for recognizing key points of a trunk and hands of the human body; the human action recognition module is used for classifying actions; and the report generation module is used for generating a mechanical engineering practical training scoring table. Through the design of a modular stacking structure, the device supports flexible adjustment of model depth and multi-task expansion, and can meet the requirements of edge device deployment and high-performance computing.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of artificial intelligence, and provides a mechanical engineering practical training scoring system, method and device based on embodied intelligence. BACKGROUND

[0002] In the current field of vocational education, practical teaching has always been an important way to cultivate students' operation skills, but the traditional teaching mode generally has problems such as too high dependence on teachers, lagging feedback information and scattered data processing, which to some extent restricts the teaching efficiency and the improvement of students' skills. In the traditional mode, since the teaching process mainly relies on the on-site guidance of teachers, it is difficult to timely and comprehensively grasp the state of students in the operation process of complex equipment, so that the teaching feedback often cannot meet the requirements of real-time and comprehensiveness, thereby affecting the teaching effect and learning quality.

[0003] At the same time, the existing teaching equipment has problems such as insufficient intelligence, and most systems still stay at the level of basic information recording or one-way demonstration, lacking multi-dimensional perception and dynamic decision-making ability for operation behavior. For example, although some practical training platforms are equipped with video monitoring function, due to the lagging nature of data processing architecture, they can only realize after-playback review and cannot provide immediate feedback during operation. In addition, there is still a lack of unified data integration and cooperation mechanism among the scattered practical training equipment, which makes it difficult for teachers to simultaneously control the real-time practical training state of all students, and the management of practical training teaching still relies on the fragmented manual inspection mode.

[0004] In recent years, with the continuous development of intelligent technology and information means, edge computing, multi-modal perception and large models provide new ideas for the intelligent upgrading of teaching equipment. However, the existing technology still has the following shortcomings:

[0005] (1) The existing target detection model has large calculation amount and slow detection speed, which cannot meet the real-time requirements of practical training scenes.

[0006] (2) The traditional action recognition method is difficult to handle the alignment and calculation redundancy of multi-modal features.

[0007] (3) The automatic practical training report generation technology still cannot meet the requirements of high accuracy, multi-dimensional evaluation and personalized evaluation, so how to efficiently and comprehensively evaluate and feedback the practical training process of each student is still a technical difficulty to be broken through. SUMMARY

[0008] The present application aims to at least solve one of the technical problems in the related art. To this end, the present application provides a mechanical engineering practical training scoring system, method and device based on embodied intelligence, which realizes fast detection speed and meets the real-time requirements of practical training scenes.

[0009] The application provides a mechanical engineering practical training scoring system based on embodied intelligence, comprising the following modules: an image data acquisition and preprocessing module, a human target position detection module, a human joint recognition and posture estimation module, a human action recognition module and a report generation module.

[0010] The image data acquisition and preprocessing module is used for standardizing the original image and video data, and outputting action sequence data with uniform format.

[0011] The human target position detection module is used for marking the human body and component target frame according to the action sequence data to obtain a human body frame diagram.

[0012] The human joint recognition and posture estimation module is used for recognizing the key points of the trunk and hands of the human body frame diagram to obtain a human body key point diagram.

[0013] The human action recognition module is used for classifying the action of the human body key point diagram to obtain action classification data.

[0014] The report generation module is used for generating a mechanical engineering practical training scoring table according to the action classification data and the action sequence data.

[0015] According to the mechanical engineering practical training scoring system based on embodied intelligence, the image data acquisition and preprocessing module comprises an image data acquisition module and an image preprocessing module.

[0016] The image data acquisition module is used for acquiring the images and videos of the mechanical engineering practical training.

[0017] The image preprocessing module is used for synchronizing the images and videos of the mechanical engineering practical training by using the NTP protocol, extracting key frames and obtaining action sequence data.

[0018] According to the mechanical engineering practical training scoring system based on embodied intelligence, the human target position detection module uses a CGYOLO human target detection network to detect the action sequence data and obtain a human body frame diagram.

[0019] According to the mechanical engineering practical training scoring system based on embodied intelligence, the CGYOLO human target detection network comprises a backbone network, a neck network and a detection head.

[0020] The backbone network comprises a C3k2 module and a C2PSA module, and the input features are split into two branches, one of which is processed by grouped convolution and C2PSA attention mechanism, and the other branch retains the original features, and finally realizes cross-stage information reuse through splicing.

[0021] The neck network replaces a traditional bilinear interpolation with a CARAFE light-weight up-sampling operator, and replaces an original EIoU loss function with a WIoU loss function.

[0022] The detection head integrates a CSPRGB (CSPNet Red Green Blue) module to perform feature fusion.

[0023] According to the mechanical engineering practical training scoring system based on embodied intelligence provided by the application, the working process of the human joint recognition and posture estimation module is:

[0024] S31: input the human body frame diagram into the SRTMPose model, classify the video action according to the space-time change characteristics of the action posture, and obtain classified video data;

[0025] S32: calculate the classified video data, generate an attention mask according to the SimAM attention mechanism, and obtain a human key point diagram by suppressing the attention mask region.

[0026] According to the mechanical engineering practical training scoring system based on embodied intelligence provided by the application, the SRTMPose model is a multi-scale feature fusion Inception module and a parameter-free attention optimization SimAM module added in the backbone network.

[0027] According to the mechanical engineering practical training scoring system based on embodied intelligence provided by the application, the working process of the human action recognition module is:

[0028] S41: input the human key point diagram into the STAR-Transformer model, classify the video action according to the space-time change characteristics of the action posture, and obtain classified action video data;

[0029] S42: input the classified action video data into an encoder to obtain encoded data, and the encoder includes a full attention FAttn module and a zigzag attention ZAttn module;

[0030] S43: input the encoded data into a decoder to obtain action classification data, and the decoder includes a full attention FAttn module and a binary attention BAttn module.

[0031] According to the mechanical engineering practical training scoring system based on embodied intelligence provided by the application, the report generation module includes a text affine generator sub-module, an image prompt word cross affine generator sub-module and a fusion module, and the working process of the report generation module is:

[0032] S51: input the action classification data and the action sequence data into an image prompt word cross- affine ware submodule to generate image fusion data, the image prompt word cross- affine ware submodule comprising a self-attention module, a cross-attention module, a feedforward neural network module and a residual connection module;

[0033] S52: input a text prompt word into a text affine ware submodule to obtain text fusion data, the text affine ware submodule comprising a self-attention module, a normalization module, a feedforward neural network module and a residual connection module;

[0034] S53: input the action classification data, the image fusion data and the text fusion data into a fusion module to generate an experiment score table.

[0035] The application also provides a mechanical engineering training scoring method based on embodied intelligence, comprising:

[0036] S1: a mechanical engineering training person performs mechanical engineering training in an image data acquisition and preprocessing module, and the image data acquisition and preprocessing module performs image acquisition;

[0037] S2: the image data acquisition and preprocessing module performs standardized processing on original image and video data, and outputs action sequence data with uniform format; a human target position detection module marks a human body and a target frame of an assembly according to the action sequence data to obtain a human body frame diagram; a human joint recognition and posture estimation module performs key point recognition on the human body frame diagram to obtain a human body key point diagram; a human action recognition module classifies actions of the human body key point diagram to obtain action classification data; and a report generation module is used to generate a scoring table of mechanical engineering training according to the action classification data and the action sequence data.

[0038] The application also provides an electronic device comprising a memory, a processor and a computer program stored in the memory and capable of running on the processor, wherein the processor implements the steps of the mechanical engineering training scoring method based on embodied intelligence according to any one of the above when executing the program.

[0039] The one or more technical solutions described above in the embodiments of the application have at least one of the following technical effects:

[0040] The mechanical engineering training scoring system, method and device based on embodied intelligence provided by the application realize efficient release of teachers from repetitive teaching tasks, and simultaneously improve the standardization and individualized ability training efficiency of students in the training process, through the multi-modal perception (image recognition, target detection) and large model decision fusion technology driven by edge computing, to construct a full-process automatic guidance and evaluation system for training operation.

[0041] Additional aspects and advantages of the present application will be made apparent from the following description. BRIEF DESCRIPTION OF DRAWINGS

[0042] In order to more clearly illustrate the technical solutions of the present application or the prior art, the following will briefly introduce the drawings needed to be used in the embodiments or prior art description. Obviously, the drawings in the following description are some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0043] Figure 1 is a structural schematic diagram of a mechanical engineering practical training scoring device based on embodied intelligence provided by the present application.

[0044] Figure 2 is a flowchart of a mechanical engineering practical training scoring method based on embodied intelligence provided by the present application.

[0045] Figure 3 is a system device deployment schematic diagram.

[0046] Figure 4 is a flowchart of image preprocessing by the image preprocessing module.

[0047] Figure 5 is a human joint recognition and posture estimation module structure.

[0048] Figure 6 is a structural schematic diagram of an electronic device provided by the present application.

[0049] Reference signs:

[0050] 101, image data acquisition and preprocessing module; 102, human target position detection module; 103, human joint recognition and posture estimation module; 104, human action recognition module; 105, report generation module; 810, processor; 820, communication interface; 830, memory; 840, communication bus. DETAILED DESCRIPTION

[0051] In order to make the purpose, technical solutions and advantages of the present application more clear, the technical solutions in the present application will be clearly and completely described below. Obviously, the described embodiments are some embodiments of the present application, not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor belong to the scope of protection of the present application. The following embodiments are used to illustrate the present application, but cannot be used to limit the scope of the present application.

[0052] In the description of the present specification, the description of the terms "one embodiment", "some embodiments", "an example", "a specific example", or "some examples" and the like means that the specific features, structures, materials or characteristics described in connection with the embodiment or example are included in at least one embodiment or example of the present application. In the present specification, the illustrative description of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in any appropriate manner in any one or more embodiments or examples. In addition, the person skilled in the art can combine and combine the different embodiments or examples described in the present specification and the features of the different embodiments or examples without contradiction.

[0053] The application will be described below in conjunction with Figures 1 to 6 The application is described.

[0054] Embodiments

[0055] As Figure 1 shown, Figure 1 The structure schematic diagram of the mechanical engineering practical training scoring device based on embodied intelligence provided by the embodiment of the present application includes the following modules: image data acquisition and preprocessing module 101, human target position detection module 102, human joint recognition and posture estimation module 103, human action recognition module 104 and report generation module 105:

[0056] The image data acquisition and preprocessing module 101 is used for standardizing the processing of original image and video data, and outputting action sequence data with uniform format;

[0057] The human target position detection module 102 is used for marking the target frame of human body and components;

[0058] The human joint recognition and posture estimation module 103 is used for recognizing the key points of the trunk and hands of the human body;

[0059] The human action recognition module 104 is used for classifying actions;

[0060] The report generation module 105 is used for generating the scoring table of mechanical engineering practical training.

[0061] Specifically, the image data acquisition and preprocessing module 101 includes an image data acquisition module and an image preprocessing module:

[0062] The image data acquisition module is used for acquiring images and videos of mechanical engineering practical training;

[0063] The image preprocessing module is used for synchronizing the images and videos of the mechanical engineering practical training by using NTP protocol, extracting key frames, and obtaining action sequence data.

[0064] The goal of this module is to collect and preprocess the motion image data of students' practical operation, build a multi-view visual data collection system for mechanical engineering practical training scenarios, and achieve high-fidelity recording of the operation process and standardized data conversion through heterogeneous camera deployment and edge-side preprocessing. As shown in Figure 3 The system is deployed around each practical training station in a ring shape with 4 high-resolution three-dimensional motion cameras to cover the details of the operator's hand movements and machine components. Each camera synchronously collects RGB images with a resolution of 640x640 or above at a frame rate of 30fps or above, focusing on capturing two types of key data: one is the dynamic action sequence during disassembly and assembly operations, such as moving and aligning components; the other is the static contrast image before and after the installation of machine components, such as screw alignment and tightening. At the same time, 6 wide-angle monitoring cameras are installed on the top of the training room, which can record 1080p panoramic videos at least at 25fps, used to record the student's operation position movement trajectory and the global spatial distribution state of tools and components.

[0065] The collected image and video data are transmitted in real time to the edge server cluster through 5G or gigabit Ethernet, and multi-level preprocessing is performed to adapt to subsequent analysis requirements. As shown in Figure 4 First, the Network Time Protocol (NTP) network transmission protocol is used to achieve millisecond-level time synchronization of all cameras, eliminating the time offset of cross-device data streams. Subsequently, the video stream is analyzed for motion saliency by the Normalized Cross-Correlation (NCC) key frame dynamic extraction module: full frame rate data is retained during the disassembly and assembly action change stage, while key frames are extracted at 1fps during the static stage such as waiting for guidance, greatly reducing the data volume without losing effective information to obtain the extracted key frame video. The picture quality optimization module performs adaptive correction for the fluctuation of the training room lighting conditions such as top light glare and metal component reflection: a local contrast enhancement algorithm based on the Retinex theory is used to suppress high light overexposure, while a non-local mean denoising algorithm is used to eliminate image grain noise in low light environments, obtaining a multi-view action sequence and component disassembly state contrast image set that are time-synchronized. The above two constitute the action sequence data.

[0066] Specifically, the human target position detection module 102 uses the CGYOLO human target detection network to detect the human body frame diagram from the action sequence data. The CGYOLO human target detection network includes a backbone network, a neck network, and a detection head:

[0067] The backbone network comprises a C3k2 module and a C2PSA module, input features are split into two branches, one branch is processed by grouped convolution and C2PSA attention mechanism, and the other branch retains the original features, and finally cross-stage information multiplexing is realized through splicing;

[0068] The neck network adopts a CARAFE lightweight up-sampling operator to replace the traditional bilinear interpolation, and a WIoU loss function is used to replace the original EIoU loss function; WIoU introduces a quality-aware weight factor to impose higher penalties on low-quality prediction boxes (such as occluded or motion-blurred human targets), thereby reducing the missed detection rate of dense crowds. The calculation expression is as follows:

[0069]

[0070] wherein, WIoU is the loss result, is a dynamic weight factor, is an intersection over union, is a center distance, which represents the Euclidean distance between the center points of the prediction box and the real box, is a diagonal length, defined as the diagonal length of the minimum bounding box of the prediction box and the real box.

[0071] The detection head integrates a CSPRGB module for feature fusion.

[0072] CSPRGB Block is a high-efficiency visual feature extraction module, which combines the cross-stage partial network (CSPNet) structure and RGB channel fine-grained optimization. Firstly, the input features are divided into two paths, the main path convolution extracts local features, and the bypass retains global information. Then, the two are fused to complement each other, while introducing channel attention, channel shuffle or grouped convolution to strengthen RGB color interaction, and supporting color space conversion to decouple brightness and chrominance information. Finally, it can improve the model's ability to capture details such as color and texture while keeping low computational cost, especially suitable for visual tasks with high real-time and accuracy requirements.

[0073] After the above series of improvements to the YOLOv11 network, this module can realize fast and accurate labeling of student positions and component positions, such as judging whether the component is correctly taken out through the change of component position.

[0074] Specifically, the working process of the human joint recognition and pose estimation module 103 is as follows:

[0075] S31: input the human frame diagram into the SRTMPose model, classify the video action according to the spatiotemporal change characteristics of the action pose, and obtain classified video data;

[0076] S32: Calculate the classified video data, generate an attention mask for the classified video data according to a SimAM attention mechanism, and obtain a human key point map by suppressing the attention mask region.

[0077] Wherein, the SRTMPose model is an improvement for the RTMPose human pose estimation model, and a multi-scale feature fusion Inception module and a parameter-free attention optimization SimAM module are added to the backbone network of the RTMPose.

[0078] First, the human frame map is input into the backbone network of the SRTMPose model, and the backbone network of the SRTMPose is an improved CSPNeXt structure. In this structure, the input picture is first subjected to initial convolution and down-sampling operation by the backbone layer, and then the features are extracted in the CSPNeXt block pair stack. In this layer model, a plurality of Inception modules are introduced to capture features of different scales (such as large joints of the body and small joints of the fingers), alleviate the scale difference between the body and the hand, and a plurality of SimAM attention modules are introduced to dynamically enhance the key area (such as the hand contour) through the parameter-free energy function and suppress the background interference (such as clothing shielding or complex lighting). Specifically, the data of each CSPNeXt module is first subjected to the SimAM attention module, then subjected to the Inception module, and finally the results of all Inception modules are added element by element to obtain an addition result. The addition result and the feature extraction result of the CSPNeXt block pair stack are added element by element to obtain the final extracted features.

[0079] Finally, the extracted features are reduced in dimension and passed to the neck network. The neck network is only responsible for simple feature splicing, and the spliced features are output by the detection head through double-task output, including 17 body key points and 42 hand key points.

[0080] Wherein, the core idea of the Inception module is parallel multi-scale convolution, which extracts features through convolution kernels of different sizes at the same time, enhances the network's ability to capture multi-scale targets, and improves the model's adaptability to large and small targets. At the same time, through convolution dimension reduction, the calculation amount of large convolution kernels is reduced.

[0081] SimAM is a parameter-free attention mechanism that dynamically allocates feature weights through an energy function. The energy function can calculate the importance score of each position in the feature map:

[0082]

[0083] Wherein, is the energy function value, is the total number of feature values, is the eigenvalue of the first feature, is the eigenvalue of the first feature, is the eigenvalue of the first feature, is the eigenvalue of the first feature, is the average value of the eigenvalue, is the standard deviation of the eigenvalue, is the smoothing term.

[0084] According to the above formula, the lower the energy, the less important the position is, and the attention mask is generated by normalizing the energy value to suppress the low energy area. By introducing the SimAM attention mechanism, the memory occupation can be reduced, while the response of the key area is enhanced, and the anti-interference ability of the model is improved.

[0085] In the present application, the Inception module solves the scale difference between the body and the hand through multi-scale convolution parallel processing, and the SimAM attention enhances the anti-interference ability through non-parametric dynamic weighting. The cooperative design of the two in the SRTMPose backbone network significantly improves the precision and robustness of hand key point detection, while maintaining real-time performance, providing an efficient solution for fine gesture estimation in complex scenes.

[0086] Specifically, the human action recognition module 104 workflow is:

[0087] S41: input the human key point map into the STAR-Transformer model, and classify the video action according to the spatio-temporal change feature of the action posture to obtain classified video data;

[0088] S42: input the classified video data into the encoder to obtain encoded data, the encoder comprising a full attention FAttn module and a zigzag attention ZAttn module;

[0089] S43: input the encoded data into the decoder to obtain action classification data, the decoder comprising a full attention FAttn module and a binary attention BAttn module.

[0090] The human action recognition module 104 inputs the video frame and the key point sequence of the video action into the STAR-Transformer model, and classifies the video action according to the spatio-temporal change feature of the action posture.

[0091] STAR-Transformer is a deep neural network model for video action classification task, and its architecture efficiently processes video frame sequence and human skeleton sequence input through multi-modal feature fusion and spatio-temporal attention mechanism. Figure 5As shown, the human action recognition module 104 as a whole adopts an encoder-decoder structure. The left side of the dashed line in the figure is the overall workflow of the STAR-Transformer model, and the right side of the dashed line represents the detailed composition of the encoder and decoder of the model, including the full space-time attention mechanism, residual link, layer normalization, token decoupling, feedforward network, etc. The arrow indicates the data flow. In the input and feature preprocessing stage, the model input is two types of multi-modal data, video frame sequence and human skeleton sequence, which are preliminarily extracted by the token decoupling module. Then, it is processed by the encoder, and the core components of the encoder include the full attention FAttn and the zigzag attention ZAttn.

[0092] wherein the full attention FAttn refers to the standard self-attention mechanism in the Transformer, that is, the formula for each token paying attention to all other position tokens is:

[0093]

[0094]

[0095]

[0096]

[0097] wherein, is an attention function, is an activation function, is a query matrix, is a key matrix, is a value matrix, is a transpose of the key matrix, is a key vector dimension, is a video space-time feature tensor, is a query matrix weight matrix, represents matrix splicing, is a human skeleton sequence feature tensor, is a key matrix weight matrix, is a value matrix weight matrix.

[0098] The zigzag attention ZAttn refers to limiting the attention range through a specific "zigzag" path. In the image Transformer, adjacent regions are alternately covered according to the spatial coordinates, and the computational complexity is reduced. The calculation formula is as follows:

[0099]

[0100] wherein, is a sparse mask, is the position at the time step, is the position at the time step, is the two-dimensional spatial coordinate of the position is the two-dimensional spatial coordinate of the position is the field radius.

[0101] Further, the processing result of the encoder is transmitted into the decoder, and the decoder generates a classification result through cross-modal interaction. The decoder also contains a full attention module FAttn, and the difference is that it replaces the ZAttn module with a binary attention module BAttn. The BAttn module binarizes the attention weight (0 or 1), and only the most critical position relationship is retained, which can greatly reduce the memory occupation. The formula is:

[0102]

[0103] wherein, is the binarized value, is a binarization function, is a threshold value.

[0104] After the above processing, the model outputs the obtained action category probability result through the Softmax activation. In summary, STAR-Transformer realizes the efficient fusion of video and skeleton sequence through the collaborative work of multiple groups of parameters.

[0105] Specifically, the report generation module 105 includes a text affine sub-module, an image prompt cross-affine sub-module, and a fusion module. The working process of the report generation module 105 is as follows:

[0106] S51: input the action classification data and the action sequence data into the image prompt cross-affine sub-module to generate image fusion data. The image prompt cross-affine sub-module includes a self-attention module, a cross-attention module, a feedforward neural network module, and a residual connection module.

[0107] S52: input the text prompt into the text affine sub-module to obtain text fusion data. The text affine sub-module includes a self-attention module, a normalization module, a feedforward neural network module, and a residual connection module.

[0108] S53: input the action classification data, the image fusion data, and the text fusion data into the fusion module to generate an experimental score table.

[0109] The main purpose of this module is to use large model technology to judge the action classification results obtained by the human action recognition module, score the standardization of each student's operation in the practical training process according to the pre-set scoring standard, and finally output the practical training score table of each student. The format of the practical training score is shown in Table 1:

[0110] Table 1 Practical training score table

[0111]

[0112] The total score of the experiment is 100 points, and it is divided into 8 sub-projects, including the complete process from taking out the components for installation to disassembling and putting back the components after the experiment ends. Each sub-project is given different scores according to its importance in the experiment.

[0113] Specifically, at the beginning of the practical training, student A enters the practical training room for preparation, and the multi-view video monitoring camera can capture the real-time position of each student in real time. During the practical training, student A performs the experiment according to the steps described in the experimental score table.

[0114] For example, taking the installation of the Y-axis module by student A as an example. First, student A puts on gloves and takes out the Y-axis module from the drawer. At this time, the video monitoring camera records the comparison chart before and after the module is taken out, and automatically judges whether the Y-axis module is correctly taken out.

[0115] Further, student A places the taken-out Y-axis module on the appropriate position on the disassembly tool table and performs operations such as installation alignment, screw tightening, etc. At this time, the three-dimensional motion camera deployed around the tool table will capture student A's hand movements in real time and upload them to the edge server for analysis to determine which step student A is performing and whether the action conforms to the standard.

[0116] Further, after detecting that the module installation is completed, the system will automatically identify whether the installed module meets the standard, whether it lacks screws and how many screws are missing, and finally automatically score this step according to the score table.

[0117] Further, after student A completes a sub-step, he does not need to pause and directly proceed to the next step operation. The system will automatically identify which step student A is completing and perform corresponding evaluation and scoring. After the system identifies that student A has completed all the practical training operations, it automatically summarizes the scores of all sub-steps and generates a final experimental score table for student A. The score representation is shown in Table 2, for example.

[0118] Specifically, Table 2 shows in detail the final score of each step of student A in the practical training process and the deduction reason, such as student A in the operation process of step 2, the direction of X-axis module is installed reversely, according to the scoring standard, 5 points shall be deducted, and the score of this step is 5 points. Finally, the total score of 64 points is obtained by summarizing all the steps.

[0119] Table 2 Student A experimental score table

[0120]

[0121] Further, the automatic report generation based on the multi-modal large model proposed by the module includes three sub-modules, and the model accepts three inputs, which are the classification results obtained by the previous module, the image representation, that is, various video frame data collected in the practical training process, and the text representation, that is, the specific content and prompt words of each item in the score table. The image representation is input into the image prompt cross-attentive sub-module, the text representation is input into the text attender sub-module, and finally the two are fused with the classification result to output the final score table. The text attender and the image prompt cross-attentive each contain 12 blocks, the text attender contains a self-attention module, a normalization module, a feedforward neural network module and a residual connection module; the image prompt cross-attentive contains a self-attention module, a cross-attention module, a feedforward neural network module and a residual connection module, wherein the self-attention module shares weights with the self-attention module in the text attender, and the cross-attention module in each block also needs input from the external prompt word.

[0122] Through the above-mentioned innovative multi-modal interaction architecture design, the text, image and classification result are deeply fused, the self-attention module with shared weights is used to realize cross-modal semantic alignment, and the consistency of feature expression is effectively improved. At the same time, the learnable dynamic prompt word mechanism is introduced in the image sub-module, the key visual information is focused by combining the cross-attention, the semantic analysis ability for complex practical training scenes is enhanced, and the calculation efficiency is greatly optimized. The hierarchical residual design ensures the training stability, and reliable output is still maintained when input noise or part of data is missing, and the modular stacking structure supports flexible adjustment of model depth and multi-task expansion, and the edge device deployment and high-performance computing demand are considered.

[0123] As shown in Figure 2 The application also provides a body intelligent-based mechanical engineering practical training scoring method, and the steps include:

[0124] S1: The mechanical engineering practical trainer performs mechanical engineering practical training in the image data acquisition and preprocessing module, and the image data acquisition and preprocessing module performs image acquisition;

[0125] S2: The image acquisition data is scored in the body intelligent-based mechanical engineering practical training scoring system to generate an experimental score table.

[0126] Figure 6 An example of a schematic diagram of a physical structure of an electronic device is shown in Figure 6 The electronic device can include a processor 810, a communication interface 820, a memory 830, and a communication bus 840, wherein the processor 810, the communication interface 820, and the memory 830 can communicate with each other through the communication bus 840. The processor 810 can invoke the logical instructions in the memory 830 to execute a mechanical engineering practical training scoring method based on embodied intelligence, which includes:

[0127] S1: The mechanical engineering practical trainer performs mechanical engineering practical training in the image data acquisition and preprocessing module, and the image data acquisition and preprocessing module performs image acquisition.

[0128] S2: The image acquisition data is scored in the mechanical engineering practical training scoring system based on embodied intelligence to generate an experimental score table.

[0129] In addition, the logical instructions in the memory 830 described above can be implemented in the form of a software function unit and sold or used as an independent product, which can be stored in a computer readable storage medium. Based on this understanding, the technical solutions of the present application essentially or the part that contributes to the prior art or part of the technical solutions can be embodied in the form of a software product, which is stored in a storage medium, including a plurality of instructions to make a computer device (which can be a personal computer, a server, or a network device, etc.) execute all or part of the steps of the method described in various embodiments of the present application. The aforementioned storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, Random Access Memory), a magnetic disk or an optical disk, and various program code storage media.

[0130] The device embodiments described above are only schematic, wherein the units shown as separate components can or can not be physically separated, and the components shown as units can or can not be physical units, i.e. they can be located in one place, or distributed on multiple network units. Part or all of the modules can be selected according to actual needs to achieve the purpose of the present embodiment. Those skilled in the art can understand and implement it without creative labor.

[0131] Those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus a necessary universal hardware platform, and of course can also be implemented by hardware, through the description of the above embodiments. Based on such understanding, the above technical solutions can be embodied in the form of a software product in essence or in the form of a part of the prior art. The computer software product can be stored in a computer readable storage medium, such as a ROM / RAM, a magnetic disk, an optical disk, etc., and includes a plurality of instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the method described in each embodiment or some parts of the embodiment.

[0132] Finally, it should be noted that: the above embodiments are only used to illustrate the technical solutions of the present application, but not to limit them; although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that: it can still modify the technical solutions recorded in the foregoing embodiments, or make equivalent replacement for some technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application.

[0133] It should be noted that embodiments of the present disclosure can be realized by hardware, software or a combination of software and hardware. The hardware part can be realized by special logic: the software part can be stored in a memory and executed by a suitable instruction execution system, such as a microprocessor or a special designed hardware. Those skilled in the art can understand that the above devices and methods can be realized by computer executable instructions and / or included in processor control code, such as providing such code on a programmable memory or a data carrier such as optical or electronic signal carrier.

[0134] In addition, although the operations of the method of the present disclosure are described in a specific order in the accompanying drawings, this does not require or imply that the operations must be performed in this specific order, or that all of the shown operations must be performed to achieve the desired result. On the contrary, the steps depicted in the flowchart can change the order of execution. Additionally or alternatively, some steps can be omitted, combined into one step, and / or divided into multiple steps. It should also be noted that the features and functions of two or more devices according to the present disclosure can be embodied in one device. Conversely, the features and functions of one device described above can be further divided into multiple devices.

[0135] Although the present disclosure has been described with reference to several specific embodiments, it should be understood that the present disclosure is not limited to the disclosed specific embodiments. The present disclosure is intended to cover various modifications and equivalent arrangements included within the spirit and scope of the appended claims.

Claims

1. A mechanical engineering training scoring system based on embodied intelligence, characterized in that, It includes the following modules: image data acquisition and preprocessing module, human target position detection module, human joint recognition and pose estimation module, human motion recognition module, and report generation module; The image data acquisition and preprocessing module is used to standardize the raw image and video data, outputting action sequence data with a uniform format; the image data acquisition and preprocessing module includes: an image data acquisition module and an image preprocessing module. The image data acquisition module is used to acquire images and videos from mechanical engineering training. The image preprocessing module is used to synchronize the images and videos of the mechanical engineering training using the NTP protocol, extract key frames, and obtain action sequence data; The human target location detection module is used to mark the target boxes of the human body and components according to the action sequence data to obtain a human body bounding map; the human target location detection module uses the CGYOLO human target detection network to detect the action sequence data to obtain the human body bounding map; The CGYOLO human target detection network includes a backbone network, a neck network, and a detection head: The backbone network includes a C3k2 module and a C2PSA module. The input features are split into two branches. One branch is processed by grouped convolution and C2PSA attention mechanism, while the other branch retains the original features. Finally, cross-stage information reuse is achieved by concatenation. The neck network uses the CARAFE lightweight upsampling operator to replace the traditional bilinear interpolation, and uses the WIoU loss function to replace the original EIoU loss function. The detection head integrates a CSPRGB module for feature fusion. By combining a cross-stage partial network structure with fine-grained optimization of RGB channels, the input features are first divided into two paths. The main path convolution extracts local features, while the side path retains global information. Then, the two are fused and complemented. At the same time, channel attention, channel shuffling, or group convolution are introduced to enhance RGB color interaction, and color space conversion is supported to decouple brightness and chromaticity information. The human joint recognition and pose estimation module is used to identify key points of the torso and hands in the human body bounding map to obtain a human body key point map; the workflow of the human joint recognition and pose estimation module is as follows: S31: Input the human body diagram into the SRTMPose model, and classify the video actions according to the spatiotemporal variation characteristics of the action posture to obtain classified video data; S32: Calculate the classified video data, generate an attention mask for the classified video data according to the SimAM attention mechanism, and suppress the attention mask region to obtain the human key point map; the SRTMPose model adds a multi-scale feature fusion Inception module and a parameterless attention optimization SimAM module to the backbone network; The human motion recognition module is used to classify the actions of human keypoint maps to obtain motion classification data; the workflow of the human motion recognition module is as follows: S41: Input the human body key point map into the STAR-Transformer model, and classify the video actions according to the spatiotemporal change characteristics of the action posture to obtain classified action video data; S42: Input the classified motion video data into the encoder to obtain encoded data. The encoder includes a full attention FAttn module and a zigzag attention ZAttn module. The formula for full attention FAttn is: in, Let be the attention function. For activation function, For querying the matrix, The key matrix, For value matrices, For the transpose of the key matrix, The dimension of the key vector. For video spatiotemporal feature tensors, To query the matrix weight matrix, Indicates matrix concatenation. For the human skeleton sequence feature tensor The key matrix and the weight matrix are, Value matrix and weight matrix; The formula for calculating zigzag attention ZAttn is as follows: in, For sparse masking, For position The time step in which it is located For position The time step in which it is located For position Two-dimensional spatial coordinates, For position Two-dimensional spatial coordinates, The radius of the domain; S43: Input the encoded data into the decoder to obtain action classification data. The decoder includes a full attention FATtn module and a binary attention BATtn module. The BAttn module has the following formula: in, The value is a binary representation. It is a binary function. To set a threshold; The report generation module is used to generate a scoring table for mechanical engineering training based on action classification data and action sequence data. The report generation module includes a text affine generator submodule, an image cue word cross-affine generator submodule, and a fusion module. The workflow of the report generation module is as follows: S51: Input the action classification data and action sequence data into the image cue word cross-affine submodule to generate image fusion data. The image cue word cross-affine submodule includes a self-attention module, a cross-attention module, a feedforward neural network module, and a residual connection module. S52: Input the text prompt words into the text affine submodule to obtain text fusion data. The text affine submodule includes a self-attention module, a normalization module, a feedforward neural network module, and a residual connection module. S53: Input the action classification data, image fusion data and text fusion data into the fusion module to generate an experimental scoring table.

2. A mechanical engineering training scoring method based on embodied intelligence, used to execute the mechanical engineering training scoring system based on embodied intelligence as described in any one of claims 1, characterized in that, include: S1: Mechanical engineering trainees conduct mechanical engineering training in the image data acquisition and preprocessing module, which performs image acquisition; S2: The image data acquisition and preprocessing module standardizes the original image and video data, outputting action sequence data in a unified format; the human target position detection module marks the target boxes of the human body and components according to the action sequence data to obtain a human body bounding box; the human joint recognition and posture estimation module performs key point recognition on the torso and hands of the human body bounding box to obtain a human body key point map; the human action recognition module classifies the actions in the human body key point map to obtain action classification data; the report generation module is used to generate a scoring table for mechanical engineering training based on the action classification data and the action sequence data.

3. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the mechanical engineering training scoring method based on embodied intelligence as described in claim 2.

Images