A data processing method and device for off-road container fork truck VR training

Abstract

The application discloses a data processing method and device for off-road container forklift VR training, and the method comprises the following steps: acquiring gesture image information of a user; the gesture image information comprises N images at different angles; pre-processing the gesture image information of the user to obtain pre-processed image information; and performing action recognition on the pre-processed image information to obtain a gesture recognition result. The method can more comprehensively capture gesture details and obtain more accurate user gesture recognition results by performing denoising, enhancement, scene correction and data fusion processing on gesture images in a VR training scene.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, and in particular to a data processing method and apparatus for VR training of off-road container forklifts. Background Technology

[0002] With the development of information and digital technologies, traditional forklift maintenance training methods can no longer meet people's needs, leading to the emergence of VR-based forklift maintenance training methods. Through VR training, trainees enter a virtual environment using VR devices, learning forklift maintenance in an immersive way. Trainees can directly interact with the virtual forklift, performing disassembly, assembly, and fault diagnosis, without touching the real equipment, thus avoiding safety hazards caused by operational errors. Trainees can practice repeatedly until they master the skills, without being restricted by time and space. In VR-based forklift maintenance training systems, user gesture recognition and processing are crucial. Existing recognition methods suffer from low accuracy, slow processing speed, and latency issues. Moreover, most current gesture recognition methods are based on computer vision, which is easily affected by environmental factors and is not suitable for practical applications such as communication and human-computer interaction. Therefore, researching image processing methods for VR training is of great significance. Summary of the Invention

[0003] The technical problem to be solved by the present invention is to provide a data processing method and device for VR training of off-road container forklifts, which solves the problems of low accuracy, slow processing speed and delay in existing recognition methods.

[0004] To address the aforementioned technical problems, a first aspect of the present invention discloses a data processing method for VR training of off-road container forklifts, the method comprising:

[0005] S1, acquire the user's gesture image information; the gesture image information includes N images from different angles;

[0006] S2, preprocess the user's gesture image information to obtain preprocessed image information;

[0007] S3, perform action recognition on the preprocessed image information to obtain gesture recognition results.

[0008] As an optional implementation, in the first aspect of the present invention, the preprocessing of the user's gesture image information to obtain preprocessed image information includes:

[0009] S21, Denoise the user's gesture image information to obtain denoised gesture image information;

[0010] S22, the denoised gesture image information is enhanced to obtain enhanced gesture image information;

[0011] S23, Perform scene correction on the enhanced gesture image information to obtain corrected gesture image information;

[0012] S24, perform data fusion on the corrected gesture image information to obtain preprocessed image information.

[0013] As an optional implementation, in the first aspect of the present invention, the step of denoising the user's gesture image information to obtain denoised gesture image information includes:

[0014] S211, Process the user's gesture image information to obtain spectrogram information;

[0015] S212, Obtain reference gesture image information;

[0016] S213, Process the reference gesture image information to obtain reference spectrogram information;

[0017] S214, using a preset denoising network model, the spectral information and the reference spectral information are mapped to obtain denoised gesture image information.

[0018] As an optional implementation, in the first aspect of the present invention, the enhancement processing of the denoised gesture image information to obtain enhanced gesture image information includes:

[0019] S221, Obtain a normal denoised gesture image H, and use a preset image simulation method to convert the normal denoised gesture image H into a low-light gesture image L, to obtain an image pair (L,H);

[0020] S222, Construct an image enhancement model; the image enhancement model includes two Unet generators Gl and Gh with the same structure but different parameters, and two PatchGAN discriminators Dl and Dh with the same structure but different parameters.

[0021] S223, using preset generator and discriminator loss functions, the image enhancement model is subjected to adversarial training to obtain an optimized image enhancement model;

[0022] S224, The optimized image enhancement model is used to process the denoised gesture image information to be processed to obtain enhanced gesture image information.

[0023] As an optional implementation, in the first aspect of the present invention, the step of performing scene correction on the enhanced gesture image information to obtain corrected gesture image information includes:

[0024] S231, The enhanced gesture image information is divided into scenes to obtain a first scene image and a second scene image;

[0025] The field of view of the first scene image is smaller than a preset threshold; the field of view of the second scene image is larger than a preset threshold.

[0026] S232, process the first scene image to obtain the first gesture deflection angle;

[0027] The expression for the first gesture deflection angle is:

[0028]

[0029] in, For the focal length of a short-focal-length camera, For the focal length of a telephoto camera, for The angle at which the directional gesture deviates from the center of the image. for The angle at which the directional gesture deviates from the center of the image. for The number of pixels by which directional gestures deviate from the center point. for The number of pixels by which directional gestures deviate from the center point. for The pixel size of the imaging plane of the directional camera is an inherent parameter of the camera. It is assumed that the pixel size of the telephoto and short-focus cameras is the same. for The pixel size of the imaging plane of the directional camera;

[0030] S233, process the second scene image to obtain the second gesture deflection angle;

[0031] S234, the second gesture deflection angle is corrected using the first gesture deflection angle to obtain corrected gesture image information; the corrected gesture image information includes N corrected images.

[0032] As an optional implementation, in the first aspect of the present invention, the step of data fusion of the compensated gesture image information to obtain preprocessed image information includes:

[0033] S241, Using a semantic segmentation model, the foreground objects in the overlapping region of the compensated gesture image information are segmented to obtain the label information of the image pixels;

[0034] S242, Process the label information of the image pixels to obtain the energy matrix;

[0035] S243, Perform dynamic programming on the energy matrix to obtain fusion reference information;

[0036] S244, Based on the fusion reference information, perform data fusion on the compensated gesture image information to obtain preprocessed image information.

[0037] As an optional implementation, in the first aspect of the present invention, the step of performing action recognition on the preprocessed image information to obtain a gesture recognition result includes:

[0038] S31, perform data enhancement on the preprocessed image information to obtain enhanced image information;

[0039] S32, using the enhanced image information, the preset gesture recognition model is trained to obtain an optimized gesture recognition model;

[0040] S33, using the optimized gesture recognition model, the enhanced image information to be processed is processed to obtain the gesture recognition result.

[0041] A second aspect of this invention discloses a data processing device for VR training of off-road container forklifts, the device comprising:

[0042] The data acquisition module is used to acquire the user's gesture image information; the gesture image information includes N images from different angles;

[0043] The preprocessing module is used to preprocess the user's gesture image information to obtain preprocessed image information;

[0044] The gesture recognition module is used to perform action recognition on the preprocessed image information to obtain gesture recognition results.

[0045] As an optional implementation, in a second aspect of the present invention, the preprocessing of the user's gesture image information to obtain preprocessed image information includes:

[0046] S21, Denoise the user's gesture image information to obtain denoised gesture image information;

[0047] S22, the denoised gesture image information is enhanced to obtain enhanced gesture image information;

[0048] S23, Perform scene correction on the enhanced gesture image information to obtain corrected gesture image information;

[0049] S24, perform data fusion on the corrected gesture image information to obtain preprocessed image information.

[0050] As an optional implementation, in the second aspect of the present invention, the step of denoising the user's gesture image information to obtain denoised gesture image information includes:

[0051] S211, Process the user's gesture image information to obtain spectrogram information;

[0052] S212, Obtain reference gesture image information;

[0053] S213, Process the reference gesture image information to obtain reference spectrogram information;

[0054] S214, using a preset denoising network model, the spectral information and the reference spectral information are mapped to obtain denoised gesture image information.

[0055] As an optional implementation, in a second aspect of the present invention, the enhancement processing of the denoised gesture image information to obtain enhanced gesture image information includes:

[0056] S221, Obtain a normal denoised gesture image H, and use a preset image simulation method to convert the normal denoised gesture image H into a low-light gesture image L, to obtain an image pair (L,H);

[0057] S222, Construct an image enhancement model; the image enhancement model includes two Unet generators Gl and Gh with the same structure but different parameters, and two PatchGAN discriminators Dl and Dh with the same structure but different parameters.

[0058] S223, using preset generator and discriminator loss functions, the image enhancement model is subjected to adversarial training to obtain an optimized image enhancement model;

[0059] S224, The optimized image enhancement model is used to process the denoised gesture image information to be processed to obtain enhanced gesture image information.

[0060] As an optional implementation, in a second aspect of the present invention, the step of performing scene correction on the enhanced gesture image information to obtain corrected gesture image information includes:

[0061] S231, The enhanced gesture image information is divided into scenes to obtain a first scene image and a second scene image;

[0062] The field of view of the first scene image is smaller than a preset threshold; the field of view of the second scene image is larger than a preset threshold.

[0063] S232, process the first scene image to obtain the first gesture deflection angle;

[0064] The expression for the first gesture deflection angle is:

[0065]

[0066] in, For the focal length of a short-focal-length camera, for The angle at which the directional gesture deviates from the center of the image. for The angle at which the directional gesture deviates from the center of the image. for The number of pixels by which directional gestures deviate from the center point. for The number of pixels by which directional gestures deviate from the center point. for The pixel size of the imaging plane of the directional camera is an inherent parameter of the camera. It is assumed that the pixel size of the telephoto and short-focus cameras is the same. for The pixel size of the imaging plane of the directional camera;

[0067] S233, process the second scene image to obtain the second gesture deflection angle;

[0068] S234, the second gesture deflection angle is corrected using the first gesture deflection angle to obtain corrected gesture image information; the corrected gesture image information includes N corrected images.

[0069] As an optional implementation, in a second aspect of the present invention, the step of data fusion of the compensated gesture image information to obtain preprocessed image information includes:

[0070] S241, Using a semantic segmentation model, the foreground objects in the overlapping region of the compensated gesture image information are segmented to obtain the label information of the image pixels;

[0071] S242, Process the label information of the image pixels to obtain the energy matrix;

[0072] S243, Perform dynamic programming on the energy matrix to obtain fusion reference information;

[0073] S244, Based on the fusion reference information, perform data fusion on the compensated gesture image information to obtain preprocessed image information.

[0074] As an optional implementation, in the second aspect of the present invention, the step of performing action recognition on the preprocessed image information to obtain a gesture recognition result includes:

[0075] S31, perform data enhancement on the preprocessed image information to obtain enhanced image information;

[0076] S32, using the enhanced image information, the preset gesture recognition model is trained to obtain an optimized gesture recognition model;

[0077] S33, using the optimized gesture recognition model, the enhanced image information to be processed is processed to obtain the gesture recognition result.

[0078] A third aspect of the present invention discloses another data processing apparatus for VR training of off-road container forklifts, the apparatus comprising:

[0079] Memory containing executable program code;

[0080] A processor coupled to the memory;

[0081] The processor calls the executable program code stored in the memory to execute some or all of the steps in the data processing method for VR training of off-road container forklifts disclosed in the first aspect of the present invention.

[0082] The fourth aspect of the present invention discloses a computer-storable medium storing computer instructions, which, when invoked, are used to execute some or all of the steps in the data processing method for VR training of off-road container forklifts disclosed in the first aspect of the present invention.

[0083] Compared with the prior art, the embodiments of the present invention have the following beneficial effects:

[0084] This invention discloses a data processing method and apparatus for VR training of off-road container forklifts. The method acquires user gesture image information, preprocesses it to obtain preprocessed image information, and then performs motion recognition on the preprocessed image information to obtain gesture recognition results. By performing denoising, enhancement, scene correction, and data fusion processing on gesture images in VR training scenarios, this invention can more comprehensively capture gesture details and obtain more accurate user gesture recognition results. Attached Figure Description

[0085] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0086] Figure 1This is a flowchart illustrating a data processing method for VR training of off-road container forklifts disclosed in an embodiment of the present invention.

[0087] Figure 2 This is a schematic diagram of the gesture recognition model disclosed in an embodiment of the present invention;

[0088] Figure 3 This is a schematic diagram of the structure of a data processing device for VR training of off-road container forklifts disclosed in an embodiment of the present invention;

[0089] Figure 4 This is a schematic diagram of another data processing device for VR training of off-road container forklifts disclosed in an embodiment of the present invention. Detailed Implementation

[0090] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0091] The terms "first," "second," etc., used in the specification, claims, and accompanying drawings of this invention are used to distinguish different objects, not to describe a specific order. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion. For example, a process, method, apparatus, product, or device that includes a series of steps or units is not limited to the listed steps or units, but may optionally include steps or units not listed, or may optionally include other steps or units inherent to these processes, methods, products, or devices.

[0092] In this document, the term "embodiment" means that a particular feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of the invention. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a separate or alternative embodiment mutually exclusive with other embodiments. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described herein can be combined with other embodiments.

[0093] This invention discloses a data processing method and apparatus for VR training of off-road container forklifts. The method includes: acquiring user gesture image information; the gesture image information including N images from different angles; preprocessing the user's gesture image information to obtain preprocessed image information; and performing motion recognition on the preprocessed image information to obtain gesture recognition results. This invention, by performing denoising, enhancement, scene correction, and data fusion processing on gesture images in VR training scenarios, can more comprehensively capture gesture details and obtain more accurate user gesture recognition results. These are described in detail below.

[0094] Example 1

[0095] Please see Figure 1 , Figure 1 This is a flowchart illustrating a data processing method for VR training of off-road container forklifts disclosed in an embodiment of the present invention. Figure 1 The data processing method described for VR training of off-road container forklifts is applied in the field of image processing technology, and the embodiments of the present invention are not limited thereto. Figure 1 As shown, the data processing method for VR training of off-road container forklifts may include the following operations:

[0096] S1, acquire the user's gesture image information; the gesture image information includes N images from different angles;

[0097] Obtain N gesture images from different angles during the user's demonstration of operating gestures on an off-road container forklift, where N≥3, covering key operating perspectives such as the front, side, and fingertip orientation of the hand, and each image carries a synchronized timestamp.

[0098] S2, preprocess the user's gesture image information to obtain preprocessed image information;

[0099] S3, perform action recognition on the preprocessed image information to obtain gesture recognition results.

[0100] Output gesture recognition results corresponding to the operation commands of the off-road container forklift (including fork lifting, mast tilting, steering control, etc.).

[0101] Optionally, the preprocessing of the user's gesture image information to obtain preprocessed image information includes:

[0102] S21, Denoise the user's gesture image information to obtain denoised gesture image information;

[0103] Eliminate noise interference caused by changes in lighting and device vibration in VR scenes;

[0104] S22, the denoised gesture image information is enhanced to obtain enhanced gesture image information;

[0105] S23, Perform scene correction on the enhanced gesture image information to obtain corrected gesture image information;

[0106] S24, perform data fusion on the corrected gesture image information to obtain preprocessed image information.

[0107] Optionally, the step of denoising the user's gesture image information to obtain denoised gesture image information includes:

[0108] S211, Process the user's gesture image information to obtain spectrogram information;

[0109] The user's gesture image information is processed using short-time Fourier transform to obtain spectral information;

[0110] S212, Obtain reference gesture image information;

[0111] Select images from the gesture database that match the current user's gesture category and have high clarity as reference gestures to obtain reference gesture image information;

[0112] S213, Process the reference gesture image information to obtain reference spectrogram information;

[0113] S214, using a preset denoising network model, the spectral information and the reference spectral information are mapped to obtain denoised gesture image information.

[0114] The preset denoising network model is a fully convolutional network consisting of 16 convolutional layers. The first layer has a kernel size of 9×8 and 18 kernels. The second to 13th convolutional layers are groups of 3 layers repeated 4 times, with kernel widths of 5, 9 and 9 respectively, and a total of 64 kernels. The last convolutional layer has a kernel width of 129 and only 1 kernel. In this network, convolution is performed only in the frequency dimension, and all layers except the first layer have a kernel width of 1 along the time dimension. After each convolutional layer, the kernels pass through a batch normalization layer and a ReLU activation function layer, and finally output through a regression layer.

[0115] Establish a denoising network model to map spectral information to reference spectral information. F This achieves the purpose of noise reduction.

[0116] Denoising gesture image information is ,in, For spectral information, For reference spectral information, and These represent the total number of time frames and frequency points, respectively.

[0117] The loss function of the denoising network model can be expressed as:

[0118]

[0119] and The weights are determined experimentally. + =1.

[0120] Optionally, the enhancement processing of the denoised gesture image information to obtain enhanced gesture image information includes:

[0121] S221, Obtain a normal denoised gesture image H, and use a preset image simulation method to convert the normal denoised gesture image H into a low-light gesture image L, to obtain an image pair (L,H);

[0122] The method is as follows:

[0123] The first gesture image is obtained by processing the normal denoised gesture image H:

[0124]

[0125] in, For the first gesture image, image H is divided into: A small block, Indicates the row and column indices of the sub-block. for The sub-block with the highest information entropy among all sub-blocks;

[0126]

[0127]

[0128] It is the mean gray level of the image H. For the first The average grayscale value of the block. For spatial dimension bandwidth, Bandwidth is the feature dimension. For the first The center coordinates of the block. x and y are the coordinates of the target pixel currently being processed.

[0129] The image brightness value of the first gesture image is inversely mapped using a preset camera response function to obtain the exposure of the first gesture image;

[0130] The exposure of the first gesture image is weighted using a random decimal lambda to obtain the weighted exposure.

[0131] Shot noise and random noise are added to the weighted exposure to obtain the noisy exposure.

[0132] Using the preset camera response function, the noise-added exposure is mapped to a new image brightness value to obtain a low-light gesture image L.

[0133] S222, Construct an image enhancement model; the image enhancement model includes two CycleGAN generators G1 and G2 with the same structure but different parameters, and two SqueezeNet discriminators D1 and D2 with the same structure but different parameters;

[0134] S223, using preset generator and discriminator loss functions, the image enhancement model is subjected to adversarial training to obtain an optimized image enhancement model;

[0135] The model's total loss = generator total loss + discriminator total loss

[0136]

[0137] and The pair consists of the normal denoised gesture image (reference gesture image information) and the low-light gesture image (user's gesture image information).

[0138] S224, The optimized image enhancement model is used to process the denoised gesture image information to be processed to obtain enhanced gesture image information.

[0139] Optionally, the step of performing scene correction on the enhanced gesture image information to obtain corrected gesture image information includes:

[0140] S231, The enhanced gesture image information is divided into scenes to obtain a first scene image and a second scene image;

[0141] The field of view of the first scene image is smaller than a preset threshold; the field of view of the second scene image is larger than a preset threshold.

[0142] S232, process the first scene image to obtain the first gesture deflection angle;

[0143] The expression for the first gesture deflection angle is:

[0144]

[0145] in, For the focal length of a short-focal-length camera, For the focal length of a telephoto camera, for The angle at which the directional gesture deviates from the center of the image. for The angle at which the directional gesture deviates from the center of the image. for The number of pixels by which directional gestures deviate from the center point. for The number of pixels by which directional gestures deviate from the center point. for The pixel size of the imaging plane of the directional camera is an inherent parameter of the camera. It is assumed that the pixel size of the telephoto and short-focus cameras is the same. for The pixel size of the imaging plane of the directional camera;

[0146] S233, process the second scene image to obtain the second gesture deflection angle;

[0147]

[0148] in, for The angle at which the directional gesture deviates from the center of the image. for The angle at which the directional gesture deviates from the center of the image. for The number of pixels by which directional gestures deviate from the center point. for The number of pixels by which directional gestures deviate from the center point. for The pixel size of the imaging plane of the directional camera is an inherent parameter of the camera. It is assumed that the pixel size of the telephoto and short-focus cameras is the same. for The pixel size of the imaging plane of the directional camera;

[0149] S234, the second gesture deflection angle is corrected using the first gesture deflection angle to obtain corrected gesture image information; the corrected gesture image information includes N corrected images.

[0150] Specifically: For images where the second gesture deflection angle is greater than a preset threshold, let:

[0151]

[0152] Optionally, the step of data fusion of the compensated gesture image information to obtain preprocessed image information includes:

[0153] S241, Using a semantic segmentation model, the foreground objects in the overlapping region of the compensated gesture image information are segmented to obtain the label information of the image pixels;

[0154] The semantic segmentation model consists of two stages. In the first stage, a fully convolutional neural network is used to perform coarse pixel-level segmentation of the image. In the second stage, a dense conditional random field is used to model the constraint relationship between pixels to correct the errors in the segmentation results of the first stage. Then, it is constructed into a recurrent convolutional neural network structure and combined with FCN to form a semantic segmentation model.

[0155] S242, Process the label information of the image pixels to obtain the energy matrix;

[0156] The energy matrix is ​​defined as follows:

[0157]

[0158]

[0159] in , , and , , These represent the images to be fused. and The components of the three channels in the YUV color space. , , For the weight of each channel, satisfy the following: , The values ​​are between 0 and 1, representing the balance weight between color energy and mask energy, with x and y being the horizontal and vertical coordinates of the pixel.

[0160]

[0161] in It is a constant. The preset mask image, Let (x,y) be the edge strength. The weights contributed to the edge.

[0162] S243, Perform dynamic programming on the energy matrix to obtain fusion reference information;

[0163] Using the DTW method of dynamic programming, a path with minimum energy is searched on the energy matrix as the fusion reference information;

[0164] S244, Based on the fusion reference information, perform data fusion on the compensated gesture image information to obtain preprocessed image information.

[0165] Specifically, the images to be fused are stitched together along the path of least energy.

[0166] Optionally, the step of performing action recognition on the preprocessed image information to obtain gesture recognition results includes:

[0167] S31, perform data enhancement on the preprocessed image information to obtain enhanced image information;

[0168] Data augmentation is performed on preprocessed image information through rotation and translation transformations;

[0169] S32, using the enhanced image information, the preset gesture recognition model is trained to obtain an optimized gesture recognition model;

[0170] Gesture recognition models such as Figure 2 As shown, the system consists of 10 stacked TMSF blocks, each composed of GCN, MSTCM, and GMF. GCN comprises a hybrid module of graph convolution and attention. MSTCM consists of two modules: multi-scale feature extraction and multi-scale feature fusion based on a pyramid structure. The multi-scale feature extraction module uses a pyramid structure to extract multi-scale features, generating multiple receptive field combinations hierarchically. By expanding the receptive field, contextual semantic temporal information with different scales is obtained. The multi-scale feature fusion module calculates the weights of features at different temporal scales based on the temporal features of the samples, and uses the Softmax function to readjust the proportion of corresponding scale features in the global information, achieving efficient and effective multi-scale feature fusion. GFM is used to perceive the global contextual information of action sequences from the frequency domain, adapting to different gesture action scenarios. The Global Filter (GFM) module transforms the input spatial features to a two-dimensional discrete Fourier transform in the frequency domain, then performs element-wise multiplication between the learnable global filter and the frequency domain features, and finally maps the features back to a two-dimensional inverse Fourier transform in the spatial domain.

[0171] S33, using the optimized gesture recognition model, the enhanced image information to be processed is processed to obtain the gesture recognition result.

[0172] As can be seen, this invention discloses an image processing method for VR training. It acquires user gesture image information, preprocesses the user's gesture image information to obtain preprocessed image information, and then performs action recognition on the preprocessed image information to obtain gesture recognition results. This invention's method, by performing denoising, enhancement, scene correction, and data fusion processing on gesture images in VR training scenarios, can more comprehensively capture gesture details and obtain more accurate user gesture recognition results.

[0173] Example 2

[0174] Please see Figure 3 , Figure 3 This is a schematic diagram of the structure of a data processing device for VR training of off-road container forklifts disclosed in an embodiment of the present invention. Figure 3 The data processing device described for VR training of off-road container forklifts is applied in the field of image processing technology, and the embodiments of the present invention are not limited thereto. Figure 3 As shown, the data processing device for VR training of off-road container forklifts may include the following operations:

[0175] S301, Data acquisition module, used to acquire user gesture image information; the gesture image information includes N images from different angles;

[0176] S302, Preprocessing module, used to preprocess the user's gesture image information to obtain preprocessed image information;

[0177] S303, gesture recognition module, used to perform action recognition on the preprocessed image information to obtain gesture recognition results.

[0178] Example 3

[0179] Please see Figure 4 , Figure 4 This is a schematic diagram of another data processing device for VR training of off-road container forklifts disclosed in an embodiment of the present invention. Figure 4 The image processing apparatus for VR training described herein is applied in the field of image processing technology, and the embodiments of this invention are not limited thereto. Figure 4 As shown, the image processing device for VR training may include the following operations:

[0180] Memory 401 storing executable program code;

[0181] Processor 402 coupled to memory 401;

[0182] The processor 402 calls the executable program code stored in the memory 401 to perform the steps in the data processing method for VR training of off-road container forklifts described in Embodiment 1.

[0183] Example 4

[0184] This invention discloses a computer-readable storage medium storing a computer program for electronic data interchange, wherein the computer program enables a computer to perform the steps in the data processing method for VR training of off-road container forklifts described in Embodiment 1.

[0185] The device embodiments described above are merely illustrative. The modules described as separate components may or may not be physically separate, and the components shown as modules may or may not be physical modules; that is, they may be located in one place or distributed across multiple network modules. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0186] Through the detailed description of the above embodiments, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, including read-only memory (ROM), random access memory (RAM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), one-time programmable read-only memory (OTPROM), electrically-Erasable Programmable Read-Only Memory (EEPROM), compact disc read-only memory (CD-ROM) or other optical disc storage, disk storage, magnetic tape storage, or any other computer-readable medium that can be used to carry or store data.

[0187] Finally, it should be noted that the data processing method and apparatus for VR training of off-road container forklifts disclosed in the embodiments of the present invention are merely preferred embodiments of the present invention and are only used to illustrate the technical solutions of the present invention, not to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A data processing method for VR training of off-road container forklifts, characterized in that, The method includes: S1, acquire the user's gesture image information; the gesture image information includes N images from different angles; S2, preprocess the user's gesture image information to obtain preprocessed image information, including: S21, Denoise the user's gesture image information to obtain denoised gesture image information; S22, the denoised gesture image information is enhanced to obtain enhanced gesture image information, including: S221, Obtain a normal denoised gesture image H, and use a preset image simulation method to convert the normal denoised gesture image H into a low-light gesture image L, to obtain an image pair (L,H); S222, Construct an image enhancement model; the image enhancement model includes two Unet generators Gl and Gh with the same structure but different parameters, and two PatchGAN discriminators Dl and Dh with the same structure but different parameters. S223, using preset generator and discriminator loss functions, the image enhancement model is subjected to adversarial training to obtain an optimized image enhancement model; S224, The optimized image enhancement model is used to process the denoised gesture image information to be processed to obtain enhanced gesture image information; S23, perform scene correction on the enhanced gesture image information to obtain corrected gesture image information, including: S231, The enhanced gesture image information is divided into scenes to obtain a first scene image and a second scene image; The field of view of the first scene image is smaller than a preset threshold; the field of view of the second scene image is larger than a preset threshold. S232, process the first scene image to obtain the first gesture deflection angle; The expression for the first gesture deflection angle is: in, For the focal length of a short-focal-length camera, For the focal length of a telephoto camera, for The angle at which the directional gesture deviates from the center of the image. for The angle at which the directional gesture deviates from the center of the image. for The number of pixels by which directional gestures deviate from the center point. for The number of pixels by which directional gestures deviate from the center point. for The pixel size of the imaging plane of the directional camera is an inherent parameter of the camera. It is assumed that the pixel size of the telephoto and short-focus cameras is the same. for The pixel size of the imaging plane of the directional camera; S233, process the second scene image to obtain the second gesture deflection angle; S234, the second gesture deflection angle is corrected using the first gesture deflection angle to obtain corrected gesture image information; the corrected gesture image information includes N corrected images; S24, perform data fusion on the corrected gesture image information to obtain preprocessed image information; S3, perform action recognition on the preprocessed image information to obtain gesture recognition results.

2. The data processing method for VR training of off-road container forklifts according to claim 1, characterized in that, The step of denoising the user's gesture image information to obtain denoised gesture image information includes: S211, Process the user's gesture image information to obtain spectrogram information; S212, Obtain reference gesture image information; S213, Process the reference gesture image information to obtain reference spectrogram information; S214, using a preset denoising network model, the spectral information and the reference spectral information are mapped to obtain denoised gesture image information.

3. The data processing method for VR training of off-road container forklifts according to claim 1, characterized in that, The step of fusing the corrected gesture image information to obtain preprocessed image information includes: S241, Using a semantic segmentation model, the foreground objects in the overlapping region of the enhanced gesture image information are segmented to obtain the label information of the image pixels; S242, Process the label information of the image pixels to obtain the energy matrix; S243, Perform dynamic programming on the energy matrix to obtain fusion reference information; S244, Based on the fusion reference information, perform data fusion on the enhanced gesture image information to obtain preprocessed image information.

4. The data processing method for VR training of off-road container forklifts according to claim 1, characterized in that, The step of performing action recognition on the preprocessed image information to obtain gesture recognition results includes: S31, perform data enhancement on the preprocessed image information to obtain enhanced image information; S32, using the enhanced image information, the preset gesture recognition model is trained to obtain an optimized gesture recognition model; S33, using the optimized gesture recognition model, the enhanced image information to be processed is processed to obtain the gesture recognition result.

5. A data processing device for VR training of off-road container forklifts, characterized in that, The device includes: The data acquisition module is used to acquire the user's gesture image information; the gesture image information includes N images from different angles; The preprocessing module is used to preprocess the user's gesture image information to obtain preprocessed image information, including: S21, Denoise the user's gesture image information to obtain denoised gesture image information; S22, the denoised gesture image information is enhanced to obtain enhanced gesture image information, including: S221, Obtain a normal denoised gesture image H, and use a preset image simulation method to convert the normal denoised gesture image H into a low-light gesture image L, to obtain an image pair (L,H); S222, Construct an image enhancement model; the image enhancement model includes two Unet generators Gl and Gh with the same structure but different parameters, and two PatchGAN discriminators Dl and Dh with the same structure but different parameters. S223, using preset generator and discriminator loss functions, the image enhancement model is subjected to adversarial training to obtain an optimized image enhancement model; S224, The optimized image enhancement model is used to process the denoised gesture image information to be processed to obtain enhanced gesture image information; S23, perform scene correction on the enhanced gesture image information to obtain corrected gesture image information, including: S231, The enhanced gesture image information is divided into scenes to obtain a first scene image and a second scene image; The field of view of the first scene image is smaller than a preset threshold; the field of view of the second scene image is larger than a preset threshold. S232, process the first scene image to obtain the first gesture deflection angle; The expression for the first gesture deflection angle is: in, For the focal length of a short-focal-length camera, For the focal length of a telephoto camera, for The angle at which the directional gesture deviates from the center of the image. for The angle at which the directional gesture deviates from the center of the image. for The number of pixels by which directional gestures deviate from the center point. for The number of pixels by which directional gestures deviate from the center point. for The pixel size of the imaging plane of the directional camera is an inherent parameter of the camera. It is assumed that the pixel size of the telephoto and short-focus cameras is the same. for The pixel size of the imaging plane of the directional camera; S233, process the second scene image to obtain the second gesture deflection angle; S234, the second gesture deflection angle is corrected using the first gesture deflection angle to obtain corrected gesture image information; the corrected gesture image information includes N corrected images; S24, perform data fusion on the corrected gesture image information to obtain preprocessed image information; The gesture recognition module is used to perform action recognition on the preprocessed image information to obtain gesture recognition results.

6. A data processing device for VR training of off-road container forklifts, characterized in that, The device includes: Memory containing executable program code; A processor coupled to the memory; The processor calls the executable program code stored in the memory to execute the data processing method for VR training of off-road container forklifts as described in any one of claims 1-4.

7. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions, which, when invoked, are used to execute the data processing method for VR training of off-road container forklifts as described in any one of claims 1-4.

Images