Employee ability multi-mode virtual training system and method based on digital twinning

Abstract

The invention provides an employee ability multi-mode virtual training method and system based on digital twinning, and relates to the technical field of digital twinning. The method comprises the following steps: dividing different lithium battery storage areas and an infrared segmented video of each lithium battery storage area by using a division model based on a panoramic image of lithium battery storage and an infrared monitoring video of lithium battery storage; clustering by using a clustering algorithm based on the change sequence of the plurality of lithium battery data of each lithium battery storage area to obtain a plurality of clusters; determining the latest storage period of each cluster based on the segmented video of the infrared rays of each lithium battery storage area in each cluster; and determining whether the working capability of the employees reaches the standard or not based on different lithium battery storage areas divided by the employees, the employee pre-estimated latest storage deadline of each lithium battery storage area divided by the employees and the latest storage deadline of each cluster. The method can accurately evaluate the working capability of lithium battery storage management of the employees.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of digital twinning, in particular to a staff capacity multi-modal virtual training system and method based on digital twinning. BACKGROUND

[0002] In lithium battery storage management, accurate control of the storage period of lithium batteries requires the professional ability of employees. When judging whether the work ability of employees to accurately store and manage lithium batteries meets the standard, the traditional method has two core pain points. First, it is difficult to objectively and quantitatively evaluate the ability of employees, and there is a lack of unified and clear judgment criteria. Second, it is difficult for the traditional method to form a scientific evaluation reference. If the state of a variety of lithium batteries in the warehouse is to be analyzed comprehensively and combined with storage environment parameters (such as temperature, humidity, voltage, internal resistance, etc.) to accurately determine the storage period of all lithium batteries, it is necessary to rely on manual detection of battery appearance (such as bulge, leakage, contact oxidation degree, etc.), reading of electrochemical parameters and recording of storage logs. This manual operation mode not only consumes time and effort, but also highly depends on individual experience, and is prone to judgment errors due to subjective differences (for example, different employees have different judgment standards for "capacity attenuation critical value"). As the scale of lithium battery storage expands and the types and specifications of lithium batteries diversify, manual inspection is more difficult to cover all inventory in real time, further exacerbating the ambiguity of employee ability evaluation and the error of the results.

[0003] Therefore, how to accurately evaluate the work ability of employees in lithium battery storage management is a problem to be solved at present. SUMMARY

[0004] The technical problem solved by the present application is to accurately evaluate the work ability of employees in lithium battery storage management.

[0005] According to a first aspect, the present application provides a staff ability multi-modal virtual training method based on digital twinning, comprising: acquiring panoramic images of lithium battery storage and infrared monitoring videos of lithium battery storage; dividing different lithium battery storage areas and segmented videos of infrared lines of each lithium battery storage area based on the panoramic images of lithium battery storage and the infrared monitoring videos of lithium battery storage using a division model; determining a change sequence of a plurality of lithium battery data of each lithium battery storage area based on the segmented videos of infrared lines of each lithium battery storage area; obtaining a plurality of clusters by clustering based on the change sequence of the plurality of lithium battery data of each lithium battery storage area; determining the latest storage deadline of each cluster based on the segmented videos of infrared lines of each lithium battery storage area in each cluster; playing the panoramic images of lithium battery storage and the infrared monitoring videos of lithium battery storage on a VR device worn by a staff; acquiring different lithium battery storage areas divided by the staff and staff-estimated latest storage deadlines of each lithium battery storage area divided by the staff; and determining whether the staff's working ability meets the standard based on the different lithium battery storage areas divided by the staff, the staff-estimated latest storage deadlines of each lithium battery storage area divided by the staff, and the latest storage deadline of each cluster.

[0006] In a possible implementation, the determination of the latest storage deadline of each cluster based on the segmented videos of infrared lines of each lithium battery storage area in each cluster comprises: generating future prediction storage videos of infrared lines of each lithium battery storage area in each cluster based on the segmented videos of infrared lines of each lithium battery storage area in each cluster; constructing a knowledge graph, each knowledge graph comprising a plurality of nodes and a plurality of edges between the nodes, each node comprising a cluster, and the node features of each node comprising the future prediction storage videos of infrared lines of each lithium battery storage area in each cluster, and the edges between the nodes representing the distances of each cluster; and determining the latest storage deadline of each cluster based on processing of the knowledge graph by a graph neural network.

[0007] In a possible implementation, the division model is a recurrent neural network model.

[0008] In a possible implementation, the clustering algorithm is a K-means clustering algorithm.

[0009] According to a second aspect, the present application provides a staff ability multi-modal virtual training system based on digital twinning, comprising:

[0010] An acquisition module is configured to acquire panoramic images of lithium battery storage and infrared monitoring videos of lithium battery storage;

[0011] The region division module is configured to divide different lithium battery storage regions and segmented videos of infrared rays of each lithium battery storage region based on the panoramic image of the lithium battery storage and the infrared monitoring video of the lithium battery storage using a division model.

[0012] The change sequence calculation module is configured to determine a change sequence of a plurality of lithium battery data of each lithium battery storage region based on the segmented videos of infrared rays of each lithium battery storage region.

[0013] The clustering processing module is configured to use a clustering algorithm to cluster the change sequence of the plurality of lithium battery data of each lithium battery storage region to obtain a plurality of clusters.

[0014] The time analysis module is configured to determine a latest storage deadline of each cluster based on the segmented videos of infrared rays of each lithium battery storage region in each cluster.

[0015] The data visualization module is configured to play the panoramic image of the lithium battery storage and the infrared monitoring video of the lithium battery storage on a VR device worn by an employee.

[0016] The second acquisition module is configured to acquire different lithium battery storage regions divided by the employee and an employee-estimated latest storage deadline of each lithium battery storage region divided by the employee.

[0017] The evaluation module is configured to determine whether the employee's work capacity meets the standard based on the different lithium battery storage regions divided by the employee, the employee-estimated latest storage deadline of each lithium battery storage region divided by the employee, and the latest storage deadline of each cluster.

[0018] In a possible implementation, the time analysis module is further configured to: generate a future prediction storage video of infrared rays of each lithium battery storage region in each cluster based on the segmented videos of infrared rays of each lithium battery storage region in each cluster; construct a knowledge graph, each knowledge graph including a plurality of nodes and a plurality of edges between the nodes, each node including a cluster, a node feature of each node including the future prediction storage video of infrared rays of each lithium battery storage region in each cluster, and an edge between nodes representing a distance of each cluster; and determine the latest storage deadline of each cluster by processing the knowledge graph based on a graph neural network.

[0019] In a possible implementation, the division model is a recurrent neural network model.

[0020] In a possible implementation, the clustering algorithm is a K-means clustering algorithm.

[0021] According to a third aspect, embodiments of the present application provide an electronic device, comprising: a processor; a memory; and a computer program; wherein the computer program is stored in the memory and configured to be executed by the processor to implement the method as described above, the method comprising: obtaining a panoramic image of a lithium battery storage and an infrared monitoring video of the lithium battery storage; dividing different lithium battery storage areas and a segmented video of infrared of each lithium battery storage area based on the panoramic image of the lithium battery storage, the infrared monitoring video of the lithium battery storage using a division model; determining a change sequence of a plurality of lithium battery data of each lithium battery storage area based on the segmented video of infrared of each lithium battery storage area; obtaining a plurality of clusters based on the change sequence of the plurality of lithium battery data of each lithium battery storage area using a clustering algorithm; determining a latest storage deadline of each cluster based on the segmented video of infrared of each lithium battery storage area in each cluster; playing the panoramic image of the lithium battery storage and the infrared monitoring video of the lithium battery storage on a VR device worn by an employee; obtaining different lithium battery storage areas divided by the employee and an employee-estimated latest storage deadline of each lithium battery storage area divided by the employee; and determining whether the employee's work ability meets the standard based on the different lithium battery storage areas divided by the employee, the employee-estimated latest storage deadline of each lithium battery storage area divided by the employee, and the latest storage deadline of each cluster.

[0022] According to a fourth aspect, embodiments of the present application provide a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the employee ability multi-modal virtual training method based on digital twinning as described above, the method comprising: obtaining a panoramic image of a lithium battery storage and an infrared monitoring video of the lithium battery storage; dividing different lithium battery storage areas and a segmented video of infrared of each lithium battery storage area based on the panoramic image of the lithium battery storage, the infrared monitoring video of the lithium battery storage using a division model; determining a change sequence of a plurality of lithium battery data of each lithium battery storage area based on the segmented video of infrared of each lithium battery storage area; obtaining a plurality of clusters based on the change sequence of the plurality of lithium battery data of each lithium battery storage area using a clustering algorithm; determining a latest storage deadline of each cluster based on the segmented video of infrared of each lithium battery storage area in each cluster; playing the panoramic image of the lithium battery storage and the infrared monitoring video of the lithium battery storage on a VR device worn by an employee; obtaining different lithium battery storage areas divided by the employee and an employee-estimated latest storage deadline of each lithium battery storage area divided by the employee; and determining whether the employee's work ability meets the standard based on the different lithium battery storage areas divided by the employee, the employee-estimated latest storage deadline of each lithium battery storage area divided by the employee, and the latest storage deadline of each cluster.

[0023] The application provides a kind of employee ability multimodal virtual training method and system based on digital twinning, which comprises obtaining panoramic image of lithium battery storage and infrared monitoring video of lithium battery storage;Different lithium battery storage areas and infrared segmented video of each lithium battery storage area are divided based on the panoramic image of lithium battery storage, infrared monitoring video of lithium battery storage using division model;The change sequence of multiple lithium battery data of each lithium battery storage area is determined based on the infrared segmented video of each lithium battery storage area;Multiple clusters are obtained by clustering based on the change sequence of multiple lithium battery data of each lithium battery storage area using clustering algorithm;The latest storage deadline of each cluster is determined based on the infrared segmented video of each lithium battery storage area in each cluster;The panoramic image of lithium battery storage and the infrared monitoring video of lithium battery storage are played on the VR device worn by the employee;Different lithium battery storage areas divided by the employee and employee estimated latest storage deadline of each lithium battery storage area divided by the employee are obtained;Whether the employee's work ability meets the standard is determined based on the different lithium battery storage areas divided by the employee, the employee estimated latest storage deadline of each lithium battery storage area divided by the employee and the latest storage deadline of each cluster, which can accurately evaluate the employee's work ability of lithium battery storage management. BRIEF DESCRIPTION OF DRAWINGS

[0024] Figure 1 A flowchart of a kind of employee ability multimodal virtual training method based on digital twinning provided for the embodiment of the application;

[0025] Figure 2 A flowchart of determining the latest storage deadline of each cluster provided for the embodiment of the application;

[0026] Figure 3 A schematic diagram of a kind of employee ability multimodal virtual training system based on digital twinning provided for the embodiment of the application; DETAILED DESCRIPTION

[0027] The application will be further described in detail by specific implementation manners combined with drawings. Similar elements in different embodiments use similar element reference numbers. In the following embodiments, many details are described in order to make the application better understood. However, those skilled in the art can easily recognize that some features can be omitted or replaced by other elements, materials or methods in different cases. In some cases, some operations related to the application are not shown or described in the specification in order to avoid the core part of the application being overwhelmed by too much description, and it is not necessary to describe these related operations in detail for those skilled in the art according to the description in the specification and general technical knowledge in the art.

[0028] In the embodiments of the present application, a method for employee ability multi-modal virtual training based on digital twinning is provided, as shown in the method for employee ability multi-modal virtual training based on digital twinning comprises steps S1-S8: Figure 1

[0029] In step S1, panoramic images of lithium battery storage and infrared monitoring videos of lithium battery storage are acquired.

[0030] The panoramic images of lithium battery storage are complete pictures of the lithium battery storage area scanned and photographed by a high-pixel wide-angle camera. The panoramic images can clearly show the appearance feature information of the lithium batteries in the warehouse and the distribution information of the lithium batteries stored at different positions.

[0031] The infrared monitoring videos of lithium battery storage are dynamic video images collected by an infrared thermal imager device. The infrared monitoring videos contain temperature distribution data of the lithium battery storage area, and can reflect the thermal stability, aging degree and other characteristics of the lithium batteries through temperature field changes. The surface temperature of the lithium batteries in each area is directly displayed according to different color blocks in the infrared monitoring videos, and key information such as temperature uniformity and local hot spots between battery monomers can be captured through specific infrared bands. For example, a high-temperature area showing red color may indicate that the chemical reaction inside the lithium battery is intensified and the capacity attenuation is accelerated, and an abnormally low-temperature area showing blue color may indicate that the activity of the electrolyte is reduced or self-discharge abnormality caused by long-term static state.

[0032] In step S2, a division model is used to divide different lithium battery storage areas and infrared segmented videos of each lithium battery storage area based on the panoramic images of lithium battery storage and the infrared monitoring videos of lithium battery storage.

[0033] The division model is a recurrent neural network model, the input of the division model is the panoramic images of lithium battery storage and the infrared monitoring videos of lithium battery storage, and the output of the division model is different lithium battery storage areas and infrared segmented videos of each lithium battery storage area.

[0034] The recurrent neural network model includes a recurrent neural network (RNN), and the internal memory unit of the recurrent neural network can save historical frame information and understand dynamic change processes by analyzing the correlation between front and back frames. This feature enables the recurrent neural network to process sequence data and capture subtle changes in the state of lithium batteries over time.

[0035] The different lithium battery storage areas are storage areas divided by the division model according to lithium battery models, specifications, and temperature sensitivity levels.

[0036] ​The temperature sensitivity level includes high-temperature storage prohibition, low-temperature application, etc.

[0037] The infrared segmented video of each lithium battery storage area is independent video data obtained by spatially cutting the infrared monitoring video picture of the lithium battery storage according to the divided area. Each independent video data corresponds to a lithium battery storage area of a type, and records the temperature distribution change, thermal stability fluctuation and shell integrity evolution of the lithium battery in the storage process.

[0038] The recurrent neural network model can extract spatial features in the panoramic image of the lithium battery storage, including the model identification clarity of lithium batteries in different areas, the bulging / leakage signs of the shell, the stacking density, and other visual information. When processing the infrared monitoring video, the recurrent neural network model tracks the time sequence law of the temperature field distribution change through its recurrent connection structure, such as the migration of hot spots and the trend of the temperature difference of single batteries. By fusing the static features of the panoramic image and the dynamic features of the infrared video, the recurrent neural network model can accurately identify the lithium battery cluster with the same storage characteristics.

[0039] In some embodiments, the division model includes an image processing layer, an infrared analysis layer, and a region determination layer. The input of the image processing layer is the panoramic image of the lithium battery storage, and the output of the image processing layer is the lithium battery shape feature change degree distribution map, the contact oxidation degree distribution map, and the preliminary outline of the storage area. The input of the infrared analysis layer is the infrared monitoring video of the lithium battery storage and the preliminary outline of the storage area, and the output of the infrared video analysis layer is the final outline of each lithium battery storage area and the lithium battery temperature distribution sequence. The input of the region determination layer is the final outline of each lithium battery storage area, the lithium battery temperature distribution sequence, the lithium battery shape feature change degree distribution map, and the contact oxidation degree distribution map, and the output of the region division layer is the different lithium battery storage areas and the infrared segmented video of each lithium battery storage area.

[0040] The lithium battery shape feature change degree distribution map records the shape feature changes of the lithium battery shape at all positions in the warehouse, such as the shell bulging deformation amount and the size deviation rate.

[0041] Each frame of the contact oxidation degree distribution map shows the oxidation state of the positive and negative contacts of the lithium battery in all areas of the warehouse, and the high oxidation risk area (oxidation area ratio > 10% or color saturation > threshold) and the low risk area are distinguished by different color scales, so that the contact performance degradation process of the lithium battery caused by environmental humidity or electrochemical corrosion during storage can be clearly presented.

[0042] The preliminary outline of the storage area is the boundary line of the lithium battery storage area preliminarily divided from the panoramic image of the lithium battery storage.

[0043] The final outline of each lithium battery storage area is an optimized accurate area boundary, which eliminates the errors in the initial outline and is highly consistent with the actual storage range.

[0044] The sequence of lithium battery temperature distribution maps completely records the temperature distribution changes of all areas in the warehouse. Each image in the sequence of lithium battery temperature distribution maps presents the temperature distribution of the lithium battery surface through infrared thermal imaging technology, and different color areas correspond to different temperature intervals.

[0045] The image processing layer is responsible for extracting key visual features from the panoramic image of lithium battery storage, including the preliminary outline of the storage area and the degree of change in the shape of the lithium battery, and the distribution map of the contact oxidation degree. The infrared analysis layer focuses on analyzing the infrared monitoring video data and combining the preliminary outline information, which not only optimizes the more accurate final outline, but also extracts the key thermodynamic features of the temperature distribution sequence of the lithium battery. The area determination layer can complete the accurate division of the storage area and the segmentation of the infrared video. This hierarchical processing architecture allows each module to focus on specific types of data processing. This design not only improves the processing accuracy of the system, but also facilitates independent optimization for different levels of task characteristics, ensuring that the processing efficiency is improved while significantly improving the accuracy and reliability of the system.

[0046] Step S3, based on the segmented video of the infrared of each lithium battery storage area, determine the change sequence of multiple lithium battery data of each lithium battery storage area.

[0047] In some embodiments, the change sequence of multiple lithium battery data of each lithium battery storage area can be determined based on the segmented video of the infrared of each lithium battery storage area using a lithium battery video processing model, the lithium battery video processing model is a recurrent neural network, the input of the lithium battery video processing model is the segmented video of the infrared of each lithium battery storage area, and the output of the lithium battery video processing model is the change sequence of multiple lithium battery data of each lithium battery storage area.

[0048] The change sequence of multiple lithium battery data of the lithium battery storage area is analyzed by the lithium battery video processing model on the infrared segmented video of the area, and then a set of multiple feature data reflecting the performance changes of the same type of lithium battery is extracted.

[0049] The change sequence of multiple lithium battery data specifically includes surface temperature change sequence, temperature difference fluctuation change sequence, thermal response coefficient change sequence, etc.

[0050] The recurrent neural network model is good at processing time series data analysis tasks because it has a unique memory cell structure that can effectively capture the time series features in the infrared video. The recurrent neural network can automatically learn the time dependence between infrared video frames through a gating mechanism, and convert discrete video frames into continuous parameter change curves. When processing the infrared segmentation video of each lithium battery storage area, the recurrent neural network can identify and associate the thermal radiation features at different time points, and quantify these visual information into specific parameter change sequences.

[0051] Step S4, using a clustering algorithm based on the change sequence of the plurality of lithium battery data of each lithium battery storage area to obtain a plurality of clusters.

[0052] The clustering algorithm is a K-means clustering algorithm, and the K value is determined in an artificial preset manner. In some embodiments, the specific implementation of the K-means clustering algorithm includes the following processes: first, randomly select K samples from all lithium battery data change sequences as initial cluster centers; then calculate the dynamic time warping distance (DTW) of each data point and each cluster center, and divide each data point to the nearest cluster; then recalculate the mean of the data points in each cluster as the new cluster center; repeat the above assignment and update steps until the cluster center position converges or the maximum iteration number is reached.

[0053] Each cluster represents a group of lithium battery storage data with highly similar aging characteristics, and the lithium batteries in the same cluster are consistent in remaining storage life, performance decay rate and other indicators.

[0054] The difference in distance between clusters reflects the difference in lithium battery aging degree. Clusters with closer distances indicate that the lithium batteries have similar aging states (such as being in the low decay stage or entering the accelerated aging period); clusters with farther distances represent significant differences in aging degree (such as new batteries that have not aged and aged batteries with more than 30% capacity decay).

[0055] Step S5, determining the latest storage deadline of each cluster based on the segmented video of the infrared rays of each lithium battery storage area in each cluster.

[0056] In some embodiments, Figure 2 A flowchart for determining the latest storage deadline of each cluster is provided for the embodiments of the present application, and the determination of the latest storage deadline of each cluster includes steps S21-S23:

[0057] Step S21, generating a future prediction storage video of infrared rays of each lithium battery storage area in each cluster based on the segmented video of infrared rays of each lithium battery storage area in each cluster.

[0058] In some embodiments, a future prediction storage video of infrared rays of each lithium battery storage area in each cluster can be generated based on the segmented video of infrared rays of each lithium battery storage area in each cluster using a variational autoencoder. The input of the variational autoencoder is the segmented video of infrared rays of each lithium battery storage area in each cluster, and the output of the variational autoencoder is the future prediction storage video of infrared rays of each lithium battery storage area in each cluster.

[0059] A variational autoencoder (VAE) is an advanced generative model that learns a latent space representation of input data through probabilistic modeling and generates new samples with similar characteristics based on the space representation. Through an encoder-decoder architecture, a variational autoencoder can effectively capture the internal distribution of data, thereby achieving high-quality generation from latent variables to new samples.

[0060] The future prediction storage video of infrared rays refers to an infrared video sequence generated by a variational autoencoder model to predict the future state of a lithium battery storage area. The future prediction storage video can predict the possible state evolution of the lithium battery under the current storage condition in the future.

[0061] Through learning a large amount of historical infrared video data, a variational autoencoder can establish a deep correlation between lithium battery aging characteristics and infrared thermal distribution characteristics. The latent variables extracted by the encoder not only contain current state information, but also encode typical aging trend patterns. Secondly, the time sequence generation module of the decoder can be optimized specifically for the characteristics of lithium battery storage, and can accurately predict the infrared representation of physical processes such as temperature distribution evolution and shell deformation expansion. Finally, the input video has been clustered according to sequence change characteristics, and the videos within the same cluster have highly similar change rules, thereby significantly improving the accuracy of prediction. By combining physical prior knowledge and data-driven learning, a variational autoencoder can achieve reliable prediction of the aging process of lithium batteries under complex storage environments.

[0062] Step S22, constructing a knowledge graph, each knowledge graph comprising a plurality of nodes and a plurality of edges between the nodes, each node comprising a cluster, the node features of each node comprising a future prediction storage video of infrared rays of each lithium battery storage area in each cluster, and the edges between the nodes representing the distance of each cluster.

[0063] Each node in the knowledge graph corresponds to a lithium battery storage cluster obtained through clustering analysis, and the node features contain future infrared prediction video data of all storage areas in the cluster. The edges between nodes represent the distance relationship between different clusters, and the weight value of the edge reflects the similarity of the aging characteristics between clusters. The greater the weight, the closer the remaining storage life of the corresponding cluster. This knowledge graph structure can intuitively show the spatio-temporal correlation between different lithium battery aging states, and integrate the clustering analysis results with the prediction data, thereby forming a structured knowledge representation.

[0064] In step S23, the knowledge graph is processed based on a graph neural network to determine the latest storage period of each cluster.

[0065] A graph neural network (GNN) is a deep learning model specifically designed for processing knowledge graph data. The graph neural network propagates and aggregates information between nodes in the knowledge graph through a message passing mechanism, and can learn the feature representation of each node. In the knowledge graph, nodes represent entities and edges represent the relationship between entities. GNN can effectively capture the complex dependency relationship in this topological structure. The input of the graph neural network is the knowledge graph, and the output of the graph neural network is the latest storage period of each cluster.

[0066] The knowledge graph systematically represents the correlation between lithium battery storage in different clusters. Each cluster represents a group of lithium battery samples with the same sequence of aging characteristic changes, and the edges quantify the difference in aging rate between clusters. This knowledge graph can intuitively reflect the distribution of lithium battery aging states, such as which clusters have similar aging processes and which clusters may have large differences in remaining life. Through the knowledge graph, the overall state of the lithium battery storage environment can be more comprehensively understood.

[0067] Node features can provide rich contextual information for the graph neural network. The future prediction storage video data of infrared rays fully presents the temperature evolution sequence, shell deformation rate and other aging characteristics of lithium batteries during future storage. These time series data can reflect the capacity degradation trend of lithium batteries, such as increased temperature fluctuations corresponding to increased internal resistance, and accelerated shell deformation rate indicating internal gas production. Therefore, node features can provide rich contextual information for the graph neural network, enabling the graph neural network to more accurately assess the aging degree of each cluster and predict the latest storage period.

[0068] The latest storage period of each cluster is determined by the graph neural network as the storage time when the capacity of the cluster area decays to the quality critical point.

[0069] The graph neural network can utilize the node features and edge relationships in the knowledge graph, and learn the evolution law of the lithium battery aging process. By analyzing the storage time difference of similar clusters, the graph neural network can infer the influence of storage environment parameters (such as temperature fluctuation amplitude, contact oxidation rate) on shelf life. At the same time, the infrared prediction video in the node feature provides time sequence change information, so that the graph neural network can capture the precursor features of thermal runaway (such as local temperature sudden rise, temperature difference fluctuation intensification). Through the data-driven analysis method, the graph neural network can also learn the optimal decision strategy from the historical storage records, and significantly improve the accuracy and reliability of the prediction.

[0070] Step S6, playing the panoramic image of the lithium battery storage and the infrared monitoring video of the lithium battery storage on the VR device worn by the employee.

[0071] The VR device is a virtual reality device worn by the employee, which includes a head-mounted display, a positioning sensor and other components, and can convert digital signals into virtual three-dimensional visual and auditory scenes. After wearing the VR device, the employee can watch the panoramic image and infrared monitoring video through the display screen built-in the device, and can also operate in the virtual warehouse environment with the help of interactive devices such as hand controllers. Through the VR device, the employee can perform operation training such as storage area division and estimated storage time in a simulated real scene.

[0072] Step S7, obtaining different lithium battery storage areas divided by the employee and the employee's estimated longest storage period of each lithium battery storage area divided by the employee;

[0073] The different lithium battery storage areas divided by the employee are the results of the employee dividing the lithium battery storage space in the virtual lithium battery warehouse environment according to his own knowledge, experience and understanding of lithium battery warehouse management, that is, the lithium battery storage panoramic image and infrared monitoring video presented by the VR device. These areas are the space range suitable for storing different types or similar storage characteristic lithium batteries according to the employee.

[0074] The employee's estimated longest storage period is the longest time that the lithium battery in each area can maintain the quality critical point under the current storage condition after the employee completes the area division, referring to the lithium battery storage life industry standard and his own experience. By determining the employee's estimated longest storage period of each lithium battery storage area, the employee's analysis and judgment ability of the lithium battery storage state and the aging process prediction ability can be intuitively presented.

[0075] Step S8, determining whether the employee's work ability meets the standard based on the different lithium battery storage areas divided by the employee, the employee's estimated longest storage period of each lithium battery storage area divided by the employee and the longest storage period of each cluster.

[0076] In some embodiments, the employee work ability can be determined based on the different employee-divided lithium battery storage areas and the employee-estimated maximum storage deadline of each employee-divided lithium battery storage area and the maximum storage deadline of each cluster by a deep neural network model, an input of the deep neural network model being the different employee-divided lithium battery storage areas and the employee-estimated maximum storage deadline of each employee-divided lithium battery storage area and the maximum storage deadline of each cluster, and an output of the deep neural network model being whether the employee work ability is up to standard.

[0077] The deep neural network model includes a deep neural network (DNN). The deep neural network is a machine learning model based on a multi-layer nonlinear neural network structure, which can automatically extract multi-level abstract features from data by simulating the information transmission mechanism of the biological nervous system. The deep neural network contains multiple hidden layers and can perform complex transformation and feature learning on input data layer by layer, and can be used to process high-dimensional and nonlinear data relationships.

[0078] The evaluation of whether the employee work ability is up to standard is to quantitatively analyze the operation results of the employee in the virtual warehouse environment by the deep neural network model to determine whether the employee has qualified professional skills. The deep neural network model can simultaneously perform spatial matching degree analysis, can perform geometric overlap degree calculation and classification logic verification on the lithium battery storage areas manually divided by the employee and the clusters generated by the clustering algorithm, and then compare the employee-estimated maximum storage deadline with the system-predicted cluster maximum storage deadline to analyze absolute error values, trend consistency and other dimensions. Through automatic feature extraction and correlation learning of the above multi-dimensional data, the deep neural network model can finally output a binary classification result of up to standard or not.

[0079] Based on the same inventive concept, Figure 3 A digital-twin-based employee ability multi-modal virtual training system schematic diagram is provided for the embodiments of the present application, and the digital-twin-based employee ability multi-modal virtual training system comprises:

[0080] The acquisition module 31 is configured to acquire panoramic images of lithium battery storage and infrared monitoring videos of lithium battery storage.

[0081] The area division module 32 is configured to divide different lithium battery storage areas and infrared segmented videos of each lithium battery storage area based on the panoramic images of lithium battery storage and the infrared monitoring videos of lithium battery storage using a division model.

[0082] a change sequence calculation module 33 configured to determine a change sequence of the plurality of lithium battery data of each lithium battery storage area based on the segmented video of the infrared rays of each lithium battery storage area;

[0083] a clustering processing module 34 configured to cluster the change sequence of the plurality of lithium battery data of each lithium battery storage area based on a clustering algorithm to obtain a plurality of clusters;

[0084] a time analysis module 35 configured to determine a latest storage deadline of each cluster based on the segmented video of the infrared rays of each lithium battery storage area in each cluster;

[0085] a data visualization module 36 configured to play the panoramic image of the lithium battery storage and the infrared monitoring video of the lithium battery storage on a VR device worn by an employee;

[0086] a second acquisition module 37 configured to acquire different lithium battery storage areas divided by the employee and an employee-estimated latest storage deadline of each lithium battery storage area divided by the employee;

[0087] an evaluation module 38 configured to determine whether the employee's work ability meets the standard based on the different lithium battery storage areas divided by the employee, the employee-estimated latest storage deadline of each lithium battery storage area divided by the employee, and the latest storage deadline of each cluster.

[0088] Furthermore, the order of the processing elements and sequences, the use of numerical

[0089] Similarly, it is to be noticed that the term "comprising", used in the description, should not be interpreted as being restricted only to the means listed thereafter; it does not exclude other elements or steps. It is thus to be interpreted as specifying the presence of the stated features, integers, steps or components as referred to, but does not preclude the presence or addition of one or more other features, integers, steps or components, or groups thereof. Thus, the scope of the expression should be commensurate to the disclosure and defined by the claims.

[0090] Finally, it should be understood that the embodiments described herein are only given by way of example and that other modifications can occur to persons skilled in the art. Therefore, the scope of the present description is not intended to be limited to the embodiments described herein but is only limited by the claims that follow.

Claims

1. A multimodal virtual training method for employee capabilities based on digital twins, characterized in that, include: Acquire panoramic images and infrared monitoring videos of the lithium battery storage area; Based on the panoramic image of the lithium battery storage and the infrared monitoring video of the lithium battery storage, a segmentation model is used to divide different lithium battery storage areas and segment the infrared video of each lithium battery storage area. Based on the segmented infrared video of each lithium battery storage area, the change sequence of multiple lithium battery data in each lithium battery storage area is determined. Multiple clusters are obtained by using a clustering algorithm based on the change sequences of multiple lithium battery data in each lithium battery storage area; The latest storage period for each cluster is determined based on the segmented video of the infrared data of each lithium battery storage area in each cluster. The panoramic images and infrared monitoring videos stored in the lithium battery are played on VR devices worn by employees. Obtain the different lithium battery storage areas assigned by employees and the estimated latest storage period for each lithium battery storage area assigned by employees. The employee's work ability is determined based on the different lithium battery storage areas assigned to the employees, the estimated latest storage period for each lithium battery storage area assigned to the employees, and the latest storage period for each cluster.

2. The multimodal virtual training method for employee capabilities based on digital twins as described in claim 1, characterized in that, The determination of the latest storage period for each cluster based on the segmented video of infrared light from each lithium battery storage area in each cluster includes: Based on the segmented video of the infrared light of each lithium battery storage area in each cluster, a future predicted storage video of the infrared light of each lithium battery storage area in each cluster is generated. A knowledge graph is constructed, wherein each knowledge graph includes multiple nodes and multiple edges between the nodes, each node includes a cluster, and the node features of each node include the future prediction of the infrared storage video of each lithium battery storage area in each cluster, and the edges between nodes represent the distance between each cluster. The knowledge graph is processed using a graph neural network to determine the latest storage expiration date for each cluster.

3. The multimodal virtual training method for employee capabilities based on digital twins as described in claim 1, characterized in that, The partitioning model is a recurrent neural network model.

4. The multimodal virtual training method for employee capabilities based on digital twins as described in claim 1, characterized in that, The clustering algorithm is the K-means clustering algorithm.

5. A multimodal virtual training system for employee capabilities based on digital twins, characterized in that, include: The acquisition module is used to acquire panoramic images and infrared monitoring videos of the lithium battery storage area. The region segmentation module is used to divide different lithium battery storage areas and segmented infrared video of each lithium battery storage area based on the panoramic image of the lithium battery storage and the infrared monitoring video of the lithium battery storage using a segmentation model. The change sequence calculation module is used to determine the change sequence of multiple lithium battery data in each lithium battery storage area based on the segmented video of the infrared light of each lithium battery storage area. The clustering module is used to cluster multiple clusters based on the change sequences of multiple lithium battery data in each lithium battery storage area using a clustering algorithm. The time analysis module is used to determine the latest storage period for each cluster based on the segmented video of the infrared light from each lithium battery storage area in each cluster; The data visualization module is used to play the panoramic image of the lithium battery and the infrared monitoring video of the lithium battery on the VR device worn by the employee. The second acquisition module is used to acquire the different lithium battery storage areas divided by the employees and the estimated latest storage period for each lithium battery storage area divided by the employees. The evaluation module is used to determine whether an employee's work ability meets the standards based on the different lithium battery storage areas assigned by the employee, the estimated latest storage period for each lithium battery storage area assigned by the employee, and the latest storage period for each cluster.

6. The multimodal virtual training system for employee capabilities based on digital twins as described in claim 5, characterized in that, The time analysis module is also used for: Based on the segmented video of the infrared light of each lithium battery storage area in each cluster, a future predicted storage video of the infrared light of each lithium battery storage area in each cluster is generated. A knowledge graph is constructed, wherein each knowledge graph includes multiple nodes and multiple edges between the nodes, each node includes a cluster, and the node features of each node include the future prediction of the infrared storage video of each lithium battery storage area in each cluster, and the edges between nodes represent the distance between each cluster. The knowledge graph is processed using a graph neural network to determine the latest storage expiration date for each cluster.

7. The multimodal virtual training system for employee capabilities based on digital twins as described in claim 5, characterized in that, The partitioning model is a recurrent neural network model.

8. The multimodal virtual training system for employee capabilities based on digital twins as described in claim 5, characterized in that, The clustering algorithm is the K-means clustering algorithm.

9. An electronic device, characterized in that, include: processor; Memory; And a computer program; wherein the computer program is stored in the memory and configured to be executed by the processor to implement the digital twin-based multimodal virtual training method for employee capabilities as described in any one of claims 1 to 4.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the digital twin-based multimodal virtual training method for employee capabilities as described in any one of claims 1 to 4.

Images