Mobile truck identification system and method of mobile robot

Abstract

The invention relates to the field of mobile truck identification, and particularly discloses a mobile truck identification system and method of a mobile robot. The method comprises the following steps: firstly, acquiring multi-region images of the surrounding environment of the mobile robot acquired by a camera, ultrasonic distance data between the mobile robot and the surrounding environment acquired by an ultrasonic sensor, and environment sound data at a plurality of preset time points acquired by a sound sensor; according to the method, feature extraction and correlation analysis are carried out on the three parts, and finally whether the mobile robot needs to carry out avoidance operation is judged through a classifier, so that the understanding ability of the mobile robot to the surrounding environment is improved in the complex and dynamic industrial environment, and efficient and safe operation on the mobile truck in the industrial environment is ensured.

Description

Technical Field

[0001] The present application relates to the field of mobile truck identification, and more specifically, to a mobile truck identification system and method for a mobile robot. Background Art

[0002] Mobile robots are intelligent devices that can move and operate autonomously in various environments and are widely used in industries such as industry, logistics, and medical care. As mobile robots are increasingly used in the industrial field, a key challenge is how to enable these robots to effectively identify and avoid moving trucks in dynamic industrial environments.

[0003] In the industrial field, the driving route, speed and occasional parking behavior of trucks may affect the normal operation of robots. At present, most mobile robots rely on single camera detection to identify moving trucks. Single camera detection has a single detection angle and is prone to visual blind spots, resulting in limited viewing angles and blind spot problems.

[0004] Therefore, a mobile truck identification system and method for a mobile robot is desired. Summary of the invention

[0005] In order to solve the above technical problems, the present application is proposed. The embodiment of the present application provides a mobile robot mobile truck recognition system and method, which first obtains multi-region images of the mobile robot's surrounding environment collected by a camera, ultrasonic distance data between the mobile robot and the surrounding environment collected by an ultrasonic sensor, and environmental sound data at multiple predetermined time points collected by a sound sensor, and then uses deep learning technology to perform feature extraction and correlation analysis on the three, and finally uses a classifier to determine whether the mobile robot needs to perform an avoidance operation, thereby improving the mobile robot's ability to understand the surrounding environment in a complex and dynamic industrial environment, and ensuring efficient and safe operation of mobile trucks in industrial environments.

[0006] According to one aspect of the present application, a mobile truck identification system of a mobile robot is provided, comprising:

[0007] A mobile truck identification data acquisition module is used to acquire multi-region images of the mobile robot's surrounding environment collected by the camera, ultrasonic distance data between the mobile robot and the surrounding environment collected by the ultrasonic sensor, and environmental sound data at multiple predetermined time points collected by the sound sensor;

[0008] A mobile truck identification data extraction module is used to extract a surrounding truck tracking feature vector and an environment multimodal association feature vector from the multi-region images of the mobile robot's surrounding environment collected by the camera, the ultrasonic distance data between the mobile robot and the surrounding environment collected by the ultrasonic sensor, and the environmental sound data at multiple predetermined time points collected by the sound sensor;

[0009] The mobile robot avoidance operation judgment module is used to judge whether the mobile robot needs to perform an avoidance operation based on the surrounding environment truck tracking feature vector and the environment multimodal association feature vector.

[0010] According to another aspect of the present application, a mobile truck identification method of a mobile robot is provided, comprising:

[0011] Acquire multi-region images of the mobile robot's surrounding environment collected by the camera, ultrasonic distance data between the mobile robot and the surrounding environment collected by the ultrasonic sensor, and environmental sound data at multiple predetermined time points collected by the sound sensor;

[0012] Extracting a surrounding truck tracking feature vector and an environment multimodal association feature vector from the multi-region images of the surrounding environment of the mobile robot collected by the camera, the ultrasonic distance data between the mobile robot and the surrounding environment collected by the ultrasonic sensor, and the environmental sound data at multiple predetermined time points collected by the sound sensor;

[0013] Based on the surrounding environment truck tracking feature vector and the environment multimodal association feature vector, it is determined whether the mobile robot needs to perform an avoidance operation.

[0014] Compared with the prior art, the present application provides a mobile robot mobile truck recognition system and method, which first obtains multi-area images of the mobile robot's surrounding environment collected by a camera, ultrasonic distance data between the mobile robot and the surrounding environment collected by an ultrasonic sensor, and environmental sound data at multiple predetermined time points collected by a sound sensor, and then uses deep learning technology to perform feature extraction and correlation analysis on the three, and finally uses a classifier to determine whether the mobile robot needs to perform an avoidance operation, thereby improving the mobile robot's ability to understand the surrounding environment in a complex and dynamic industrial environment, and ensuring efficient and safe operation of mobile trucks in industrial environments. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] By describing the embodiments of the present application in more detail in conjunction with the accompanying drawings, the above and other purposes, features and advantages of the present application will become more apparent. The accompanying drawings are used to provide a further understanding of the embodiments of the present application and constitute a part of the specification. Together with the embodiments of the present application, they are used to explain the present application and do not constitute a limitation of the present application. In the accompanying drawings, the same reference numerals generally represent the same components or steps.

[0016] Figure 1 4 is a block diagram of a mobile truck identification system of a mobile robot according to an embodiment of the present application.

[0017] Figure 2 4 is a block diagram of a mobile truck identification data extraction module in a mobile truck identification system of a mobile robot according to an embodiment of the present application.

[0018] Figure 3 It is a block diagram of a surrounding environment multi-region feature extraction unit in a mobile truck recognition system of a mobile robot according to an embodiment of the present application.

[0019] Figure 4 It is a block diagram of a mobile robot avoidance operation judgment module in a mobile robot's mobile truck recognition system according to an embodiment of the present application.

[0020] Figure 5 The present invention is a flowchart of a mobile truck identification method of a mobile robot according to an embodiment of the present application.

[0021] Figure 6 is a block diagram of an electronic device according to an embodiment of the present application. DETAILED DESCRIPTION

[0022] Below, the exemplary embodiments according to the present application will be described in detail with reference to the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments of the present application, and it should be understood that the present application is not limited to the exemplary embodiments described here.

[0023] Figure 1 FIG. 1 is a block diagram of a mobile truck identification system of a mobile robot according to an embodiment of the present application. Figure 1As shown, according to the mobile truck identification system 100 of the mobile robot according to the embodiment of the present application, it includes: a mobile truck identification data acquisition module 110, which is used to obtain multi-region images of the mobile robot's surrounding environment collected by a camera, ultrasonic distance data between the mobile robot and the surrounding environment collected by an ultrasonic sensor, and environmental sound data at multiple predetermined time points collected by a sound sensor; a mobile truck identification data extraction module 120, which is used to extract the surrounding environment truck tracking feature vector and the environmental multimodal association feature vector from the multi-region images of the mobile robot's surrounding environment collected by the camera, the ultrasonic distance data between the mobile robot and the surrounding environment collected by the ultrasonic sensor, and the environmental sound data at multiple predetermined time points collected by the sound sensor; a mobile robot avoidance operation judgment module 130, which is used to judge whether the mobile robot needs to perform an avoidance operation based on the surrounding environment truck tracking feature vector and the environmental multimodal association feature vector.

[0024] In the mobile truck identification system 100 of the mobile robot, the mobile truck identification data acquisition module 110 is used to obtain multi-region images of the mobile robot's surrounding environment collected by the camera, ultrasonic distance data between the mobile robot and the surrounding environment collected by the ultrasonic sensor, and environmental sound data at multiple predetermined time points collected by the sound sensor. It should be understood that the mobile robot is an intelligent device that can act and perform tasks autonomously in a variety of environments, and is widely used in industries such as industry, logistics, and medical care. With the increasing application of these robots in industrial environments, a major challenge is how to enable robots to effectively identify and avoid mobile trucks in complex industrial environments. In an industrial environment, the driving path, speed changes, and occasional docking of the truck may interfere with the normal operation of the robot. At present, most mobile robots rely on a single camera for truck detection. However, the viewing angle of a single camera is limited, and visual blind spots are prone to occur, which leads to a limited recognition range and difficulty in fully monitoring dynamic changes in the environment. Therefore, in the technical solution of the present application, by acquiring multi-region images of the mobile robot's surroundings collected by the camera, ultrasonic distance data between the mobile robot and the surroundings collected by the ultrasonic sensor, and environmental sound data at multiple predetermined time points collected by the sound sensor, and combining deep learning technology, the mobile robot's ability to understand the surrounding environment in a complex and dynamic industrial environment is improved to determine whether to perform avoidance operations. This will ensure that the robot can safely and efficiently deal with moving trucks in industrial environments and avoid potential conflicts.

[0025] Specifically, the data collected by the camera, ultrasonic sensor and sound sensor are obtained to achieve a comprehensive perception of the environment around the mobile robot, thereby improving the robot's navigation and obstacle avoidance capabilities in complex environments. Among them, the camera provides visual information of the robot's surroundings and can capture various objects and obstacles in the environment. The use of multi-region images can cover different angles and areas around the robot, allowing the system to identify and track trucks or other dynamic objects in the environment. The ultrasonic sensor measures the distance between the object and the sensor by emitting and receiving ultrasonic waves. These distance data can help the system understand the spatial relationship between the robot and the surrounding environment, including the distance and relative position of obstacles. The environmental sound data recorded by the sound sensor can provide additional environmental information, such as whether there is a sound source (such as horns, engine noise, etc.), which may be related to the moving truck. The combination of these three types of data provides a multimodal perception system that can more comprehensively identify and track surrounding trucks and other obstacles. This comprehensive perception capability enables the mobile robot to make avoidance decisions more accurately in complex environments, improving the safety and reliability of navigation.

[0026] In the mobile truck identification system 100 of the mobile robot, the mobile truck identification data extraction module 120 is used to extract the surrounding truck tracking feature vector and the environment multimodal association feature vector from the multi-region images of the mobile robot's surrounding environment collected by the camera, the ultrasonic distance data between the mobile robot and the surrounding environment collected by the ultrasonic sensor, and the environmental sound data at multiple predetermined time points collected by the sound sensor. It should be understood that through multimodal feature fusion, a comprehensive description of the environment can be obtained, including the location of the truck, the state of motion, the distance of obstacles, and the source of sound. The purpose of this is to improve the accuracy and comprehensiveness of environmental perception by comprehensively utilizing data from different sensors.

[0027] Figure 2 FIG. 1 is a block diagram of a mobile truck identification data extraction module in a mobile truck identification system of a mobile robot according to an embodiment of the present application. Figure 2As shown, in a specific embodiment of the present application, the mobile truck identification data extraction module 120 includes: a surrounding environment multi-region feature extraction unit 121, which is used to perform feature extraction on the multi-region image of the mobile robot's surrounding environment collected by the camera to obtain the surrounding environment truck tracking feature vector; an ultrasonic distance feature extraction unit 122, which is used to perform feature extraction on the ultrasonic distance data between the mobile robot and the surrounding environment collected by the ultrasonic sensor to obtain an obstacle distance semantic feature vector; an environmental sound feature extraction unit 123, which is used to perform feature extraction on the environmental sound data at multiple predetermined time points collected by the sound sensor to obtain an environmental sound spatial state feature vector; an environmental multimodal feature fusion unit 124, which is used to fuse the obstacle distance semantic feature vector and the environmental sound spatial state feature vector to obtain the environmental multimodal association feature vector.

[0028] It should be understood that the environmental images captured by the camera contain a lot of visual information, including the color, shape, and texture of the scene. Feature extraction can extract key features related to the truck, such as the outline, color, and logo of the truck body, by converting these image data into structured feature vectors. In a dynamic environment, the truck may be constantly moving. Through feature extraction, the generated truck tracking feature vector contains information about the truck's position, speed, and direction. These vectors can not only help the system identify the truck in the image, but also track its movement trajectory. This is crucial to achieve continuous monitoring and tracking of targets, especially when the truck is moving, so that it can keep tracking the target.

[0029] Furthermore, feature extraction of ultrasonic distance data between the mobile robot and the surrounding environment collected by ultrasonic sensors is to improve the accuracy of environmental perception and the decision-making ability of the system. Ultrasonic sensors can provide real-time distance data from obstacles. These data reflect the spatial layout around the robot, including the location and distance of obstacles. Through feature extraction, these raw distance data can be converted into structured feature vectors, helping the system to understand the spatial distribution of the surrounding environment faster and more accurately. This real-time perception is crucial for dynamically adjusting the robot path and avoiding collisions.

[0030] Furthermore, feature extraction of environmental sound data collected by sound sensors at multiple predetermined time points is performed to enhance the perception and decision-making capabilities of mobile robots in complex environments. Although cameras and ultrasonic sensors provide visual and distance data, these sensors cannot fully capture the sound changes in the environment. By extracting features from sound data, complementary information about sound features can be obtained. For example, the spectral characteristics of sound can help identify specific sound sources or change patterns, which may be difficult to detect in visual data. Combining sound features with visual and distance data helps to understand the environmental conditions more comprehensively.

[0031] In particular, the obstacle distance semantic feature vector and the environmental sound spatial state feature vector are fused to obtain the environmental multimodal association feature vector in order to enhance the mobile robot's comprehensive understanding and decision-making ability of the surrounding environment. The information provided by different sensors is complementary. Visual and ultrasonic sensors are mainly used to detect the spatial position and distance of obstacles, while sound sensors provide additional information about environmental sounds.

[0032] Figure 3 FIG. 1 is a block diagram of a surrounding environment multi-region feature extraction unit in a mobile truck recognition system of a mobile robot according to an embodiment of the present application. Figure 3 As shown, in a specific embodiment of the present application, the surrounding environment multi-region feature extraction unit 121 includes: a surrounding environment multi-region feature encoding subunit 1211, which is used to perform feature encoding on the multi-region image of the surrounding environment of the mobile robot collected by the camera to obtain a surrounding environment truck tracking feature map; a surrounding environment truck feature downsampling subunit 1212, which is used to downsample the surrounding environment truck tracking feature map to obtain the surrounding environment truck tracking feature vector.

[0033] It should be understood that multi-region images provide rich visual information about the surrounding environment. Through feature encoding, these high-dimensional image data can be converted into low-dimensional feature maps. Feature encoding extracts the most representative features in the image, such as the shape, color, and texture of the truck, allowing the system to more accurately identify and distinguish the truck from other objects. Feature maps can extract the key visual features of the truck, thereby improving the accuracy of recognition. Raw image data often contains a lot of redundant information, and directly processing this data may result in inefficient computation. Feature encoding compresses image data into smaller feature maps while retaining key visual information. This data compression not only increases processing speed, but also reduces the consumption of computing resources, allowing the system to run more efficiently in real-time or near-real-time application scenarios.

[0034] Furthermore, the surrounding truck tracking feature map is downsampled to improve processing efficiency and system performance while maintaining important information. Feature maps usually contain a large number of pixels and complex visual information, which may require huge computing resources when processing. Downsampling reduces the amount of data by reducing the resolution of the feature map, thereby significantly reducing computational complexity and storage requirements. This enables the system to process and analyze faster and meet the requirements of real-time applications, especially on embedded systems or mobile devices with limited resources.

[0035] In a specific embodiment of the present application, the surrounding environment multi-region feature encoding subunit 1211 includes: passing the multi-region images of the surrounding environment of the mobile robot captured by the camera through a surrounding environment image denoiser based on deep learning to obtain multiple-region surrounding environment denoised and enhanced images; passing the multiple-region surrounding environment denoised and enhanced images through a surrounding environment truck tracking YOLO target detection network as a feature extractor to obtain multiple-region surrounding environment truck tracking interest regions; passing the multiple-region surrounding environment truck tracking interest regions through a surrounding environment truck tracking feature encoder to obtain the surrounding environment truck tracking feature map.

[0036] It should be understood that images captured by cameras under different environmental conditions (such as low light, strong reflections, or high contrast scenes) may contain various noises. These noises can interfere with the details of the image, making the image blurry or distorted. The deep learning-based denoiser can effectively identify and reduce these noises, thereby improving the clarity and quality of the image. This clear image helps the system to more accurately identify and analyze target objects in the surrounding environment, such as trucks, people, or other obstacles. The deep learning denoiser can extract more realistic and useful features from the image by training the network model to learn the difference between noise features and real signals. Through denoising, the system's adaptability to different environmental conditions can be improved, allowing the robot to maintain stable performance in the face of various interferences (such as light changes, shadows, or reflections). The enhanced image data makes the system more robust in complex environments, and can still achieve accurate target recognition and environmental understanding under less than ideal conditions. Specifically, the multi-region images of the mobile robot's surrounding environment captured by the camera are input into the encoder of the surrounding environment image denoiser based on deep learning, wherein the encoder uses a convolution layer to perform explicit spatial encoding on the multi-region images of the mobile robot's surrounding environment captured by the camera to obtain image features; and the image features are input into the decoder of the surrounding environment image denoiser based on deep learning, wherein the decoder uses a deconvolution layer to perform deconvolution processing on the image features to obtain the denoised and enhanced images of the multiple-region surrounding environments.

[0037] Furthermore, the denoised and enhanced images of the surrounding environment in multiple regions are processed through the YOLO target detection network, mainly to improve the accuracy and efficiency of target detection. The YOLO (You Only Look Once) target detection network can detect multiple targets in an image in real time and accurately locate them. By inputting the denoised and enhanced images into the YOLO network, its powerful feature extraction and target detection capabilities can be fully utilized to accurately identify and locate the truck. This is because after the denoised and enhanced images are processed, the noise is significantly reduced, and the features in the image are more prominent, which helps the YOLO network to more effectively identify and locate the target. The original intention of the design of the YOLO network is to achieve efficient real-time target detection. The YOLO network can extract key features from complex images and map them to the task of target detection. The denoised and enhanced images provide clearer visual information, and the YOLO network can better extract and utilize these features on this basis, thereby improving the accuracy of truck tracking.

[0038] Furthermore, the surrounding environment truck tracking regions of interest in multiple regions are processed through feature encoders, mainly to further extract and enhance key features in the image and improve the accuracy and efficiency of target tracking. The role of the feature encoder is to extract deep and recognizable features from the region of interest. The region of interest usually contains important information related to the target, but this information may be interfered by complex background or noise. The feature encoder can extract clearer and more useful features from these regions through technologies such as convolutional layers and pooling layers to form feature maps. In the target tracking task, the input image usually contains a large amount of pixel information. Directly processing these high-dimensional data may result in high computational complexity and low processing efficiency. By converting the original image data into a feature map, the feature encoder can map these high-dimensional data to a lower-dimensional space while retaining key feature information. Specifically, each layer of the surrounding environment truck tracking feature encoder is used to perform convolution processing, mean pooling processing based on the local feature matrix and nonlinear activation processing on the input data in the forward pass of the layer to output the surrounding environment truck tracking feature map by the last layer of the surrounding environment truck tracking feature encoder, wherein the input of the surrounding environment truck tracking feature encoder is the multiple regional surrounding environment truck tracking areas of interest.

[0039] In a specific embodiment of the present application, the ultrasonic distance feature extraction unit 122 includes: passing the ultrasonic distance data between the mobile robot and the surrounding environment collected by the ultrasonic sensor through an obstacle distance text convolutional neural network to obtain a plurality of obstacle distance feature vectors; and splicing the plurality of obstacle distance feature vectors into the obstacle distance semantic feature vector.

[0040] It should be understood that the ultrasonic sensor provides distance information from obstacles in the environment, and these raw data themselves may lack sufficient structured information. Convolutional neural networks (CNNs) can automatically extract recognizable features from these distance data and convert them into feature vectors. These feature vectors contain information such as the location, shape, and size of obstacles, providing detailed and accurate data representation for the robot's environmental perception. The data of ultrasonic sensors are usually high-dimensional, and direct processing may result in high computational complexity. Through convolution and pooling operations, convolutional neural networks can effectively convert high-dimensional data into low-dimensional feature vectors while retaining key feature information. Specifically, the ultrasonic distance data between the mobile robot and the surrounding environment collected by the ultrasonic sensor is segmented to obtain a word sequence; the embedding layer of the obstacle distance text convolutional neural network is used to map each word in the word sequence to a word embedding vector to obtain a sequence of word embedding vectors; the Bert model based on the transformer of the obstacle distance text convolutional neural network is used to encode the sequence of word embedding vectors based on global context semantics to obtain multiple obstacle distance feature vectors.

[0041] Furthermore, multiple obstacle distance feature vectors are spliced ​​into an obstacle distance semantic feature vector, mainly to integrate environmental information from different angles, provide a more comprehensive and rich feature representation, and thus improve the understanding and processing capabilities of the environment. Among them, each obstacle distance feature vector is usually collected from a specific perspective or sensor position, and may only reflect part of the environment. By splicing these multiple feature vectors together, distance data from different sources can be fused to form a more comprehensive obstacle distance semantic feature vector. This integration helps to form a global understanding of the environment and enhance the system's ability to perceive surrounding obstacles.

[0042] In a specific embodiment of the present application, the ambient sound feature extraction unit 123 includes: extracting an ambient sound spectrum graph from the ambient sound data at multiple predetermined time points collected by the sound sensor; and passing the ambient sound spectrum graph through an ambient sound spectrum space focusing feature encoder to obtain the ambient sound space state feature vector.

[0043] It should be understood that the original sound data is usually a time domain signal, and it may be difficult to extract useful information by directly processing these signals. By converting the sound data into a spectrogram, the time domain signal can be converted into a frequency domain representation. The spectrogram shows the intensity distribution of the sound signal at different frequencies, allowing analysts to more clearly understand the frequency characteristics and changing trends of the sound, which is crucial for identifying and classifying sounds. The spectrogram can reveal the frequency composition and intensity changes of the sound, helping to identify different sound sources in the environment. For example, the spectrogram can distinguish different sound patterns in the environment, such as vehicle horns, wind, or human voices. This feature extraction is very important for applications such as acoustic environment monitoring, sound event detection, and noise control. The sound spectrogram can decompose complex sound signals into a series of simple frequency components through techniques such as Fourier transform. This frequency decomposition not only helps improve the accuracy of sound recognition, but also can extract useful information from the spectrum when there is a lot of noise or a weak sound signal.

[0044] Furthermore, the ambient sound spectrogram is processed by the ambient sound spectrum spatial focusing feature encoder, mainly to extract representative spatial features from the spectrogram and convert these features into a more informative and structured feature vector. The ambient sound spectrogram provides the distribution of sound in the frequency domain, but in practical applications, the spectrogram alone may not be sufficient to capture the spatial characteristics of the sound source. The ambient sound spectrum spatial focusing feature encoder can extract key features related to the spatial position of the sound source by performing feature extraction and spatial focusing on the spectrogram. These features help understand the spatial layout and source distribution of the sound, allowing the system to better analyze and process the ambient sound. The data in the spectrogram is high-dimensional, and direct processing may result in high computational complexity and information redundancy. Through spatial focusing, the feature encoder can highlight the spatial characteristics of the sound, helping the system to better understand the source of the sound and its spatial characteristics in the environment. Specifically, the convolutional coding part of the ambient sound spectrum spatial focus feature encoder is used to perform deep convolutional coding on the ambient sound spectrum map to obtain an initial convolutional feature map; the initial convolutional feature map is input into the spatial attention part of the ambient sound spectrum spatial focus feature encoder to obtain a spatial attention map; the spatial attention map is activated by a Softmax function to obtain a spatial attention feature map; and the spatial attention feature map and the initial convolutional feature map are multiplied by the position points to obtain the ambient sound spatial state feature vector. More specifically, the initial convolutional feature map is input into the spatial attention part of the ambient sound spectrum spatial focus feature encoder to obtain the spatial attention map, including: performing average pooling and maximum pooling along the channel dimension on the initial convolutional feature map to obtain an average feature matrix and a maximum feature matrix; cascading and channel-adjusting the average feature matrix and the maximum feature matrix to obtain a channel feature matrix; and using the convolutional layer of the spatial attention feature map to perform convolutional coding on the channel feature matrix to obtain a spatial attention map.

[0045] In the mobile robot mobile truck identification system 100, the mobile robot avoidance operation judgment module 130 is used to judge whether the mobile robot needs to perform an avoidance operation based on the surrounding environment truck tracking feature vector and the environment multimodal association feature vector. It should be understood that the truck tracking feature vector provides information about the specific position, speed, direction of movement, etc. of the truck. These feature vectors help the robot understand the current position and movement trajectory of the truck in the environment, thereby predicting the future position of the truck. This is crucial for real-time avoidance because it allows the robot to dynamically adjust its path to avoid collision with the truck. Based on these features, the robot can calculate the movement trend of the truck and decide whether to change the route based on its current and expected position. The environment multimodal association feature vector integrates data from different sensors, such as vision, lidar, and sound, providing a more comprehensive understanding of the environment. This multimodal information fusion helps to improve the ability to recognize environmental complexity, enabling the robot to more accurately perceive surrounding obstacles and dynamic targets, including trucks. By combining the information of the two, the robot can evaluate the relationship between the movement of the truck and other environmental factors in real time, such as whether there is enough space for safe avoidance, whether there are other obstacles affecting the avoidance path, etc. The ultimate goal of this comprehensive analysis is to ensure the safety and operational efficiency of the robot.

[0046] Figure 4 FIG. 1 is a block diagram of a mobile robot avoidance operation judgment module in a mobile robot mobile truck recognition system according to an embodiment of the present application. Figure 4 As shown, in a specific embodiment of the present application, the mobile robot avoidance operation judgment module 130 includes: an environmental information feature association unit 131, which is used to associate the surrounding environment truck tracking feature vector and the environmental multimodal association feature vector to obtain a robot avoidance judgment classification feature vector; an environmental information feature optimization unit 132, which is used to perform a weight-adapted quasi-coherent interference correction on the robot avoidance judgment classification feature vector to obtain an optimized robot avoidance judgment classification feature vector; an avoidance operation classification judgment unit 133, which is used to pass the optimized robot avoidance judgment classification feature vector through a classifier to obtain a classification result, and the classification result is used to determine whether the mobile robot needs to perform an avoidance operation.

[0047] It should be understood that combining the truck tracking feature vector with the environment multimodal association feature vector can generate a robot avoidance judgment classification feature vector. This feature vector brings together information about the dynamic and static environments, and after processing, it can clearly identify whether there is a situation that needs to be avoided. This approach not only improves the accuracy of decision-making, but also speeds up the response speed, ensuring the safety and operating efficiency of the robot in complex environments.

[0048] In particular, in the technical solution of the present application, the robot avoidance judgment classification feature vector is obtained by fusing multiple data sources, namely, multi-region images of the mobile robot's surrounding environment, ultrasonic distance data between the mobile robot and the surrounding environment, and environmental sound data, which contain complex feature information and have different features and semantics. When the features of these multiple data sources are fused, there may be inconsistencies or redundant information between them. When the classifier processes the fused features, it may be affected by this inconsistency, resulting in class-coherent interference between the weights and the feature vectors. In particular, in the feature fusion process, the scale, distribution, and importance of the features may vary, which may cause the impact of certain features on the classifier to be over-emphasized or ignored. Therefore, in the technical solution of the present application, the robot avoidance judgment classification feature vector is subjected to class-coherent interference correction based on weight adaptation to obtain an optimized robot avoidance judgment classification feature vector.

[0049] Among them, the robot avoidance judgment classification feature vector is subjected to quasi-coherent interference correction based on weight adaptation to obtain an optimized robot avoidance judgment classification feature vector, including: calculating the product between the robot avoidance judgment classification feature vector and its transposed vector to obtain a robot avoidance judgment feature autocorrelation expression matrix; calculating the coherent interference phase between the category decoding weight matrix of the classifier and the robot avoidance judgment feature autocorrelation expression matrix to obtain a robot avoidance judgment feature-weight fine-grained coupling representation matrix; performing probabilistic processing on the robot avoidance judgment feature-weight fine-grained coupling representation matrix based on the sigmoid activation function to obtain a robot avoidance judgment feature-weight fine-grained coupling probabilistic representation matrix; performing back-propagation expression compensation on the robot avoidance judgment classification feature vector based on the robot avoidance judgment feature-weight fine-grained coupling probabilistic representation matrix to obtain a correlation compensation representation vector; and calculating the position-weighted sum between the correlation compensation representation vector and the robot avoidance judgment classification feature vector to obtain the optimized robot avoidance judgment classification feature vector.

[0050] The coherent interference phase between the category decoding weight matrix of the classifier and the robot avoidance judgment feature autocorrelation expression matrix is ​​calculated by the following coupling formula to obtain the robot avoidance judgment feature-weight fine-grained coupling representation matrix;

[0051] Wherein, the coupling formula is:

[0052]

[0053] Among them, V 1i represents the i-th row vector of the class decoding weight matrix of the classifier, V 2jrepresents the j-th row vector of the robot avoidance judgment feature autocorrelation expression matrix, ∑ represents the covariance matrix of the data set to which the i-th row vector of the category decoding weight matrix of the classifier and the j-th row vector of the robot avoidance judgment feature autocorrelation expression matrix belong, ∑ -1 represents the inverse matrix of the covariance matrix, A(V 1i ,V 2j ) represents the eigenvalue of the (i, j)th position of the robot avoidance judgment feature-weight fine-grained coupling probabilistic representation matrix.

[0054] Among them, the robot avoidance judgment classification feature vector is back-propagated and expressed to obtain a related compensation representation vector based on the robot avoidance judgment feature-weight fine-grained coupling probabilistic representation matrix according to the following compensation formula;

[0055] Wherein, the compensation formula is:

[0056]

[0057] Among them, V r represents the relevant compensation representation vector, A represents the robot avoidance judgment feature-weight fine-grained coupling probabilistic representation matrix, V c represents the robot avoidance judgment classification feature vector, Represents matrix multiplication.

[0058] In the technical solution of the present application, when the robot avoidance judgment classification feature vector is classified by the classifier, since the weight of the classifier also needs to be adapted to the robot avoidance judgment classification feature vector, class coherent interference with the robot avoidance judgment classification feature vector may occur. Based on this, in the technical solution of the present application, the robot avoidance judgment classification feature vector is corrected by class coherent interference based on weight adaptation, which first represents the autocorrelation interference feature spectrum with the autocorrelation matrix of the robot avoidance judgment classification feature vector, and then simulates the fine-grained feature-class label coherent interference phase between the robot avoidance judgment feature and the weight matrix of the classifier by calculating the correlation measure between the autocorrelation matrix of the robot avoidance judgment classification feature vector and the class decoding weight matrix of the classifier, and then performs back-propagation expression compensation on the robot avoidance judgment classification feature vector with the robot avoidance judgment feature-weight fine-grained coupling representation matrix, and restores the equivalent probability intensity representation of the robot avoidance judgment classification feature vector without interference by reverse compensation, and obtains the optimized robot avoidance judgment classification feature vector to improve the accuracy of the classification result.

[0059] Furthermore, the optimized robot avoidance judgment classification feature vector integrates the truck tracking feature vector and the environment multimodal association feature vector to provide a comprehensive view of the environment. These feature vectors contain the dynamic relationship and static information between the robot and the truck and other environmental factors, covering important factors such as distance, speed, and relative motion. By integrating these features, the classifier is able to process and analyze these complex data and generate effective classification results. The classifier can be trained to recognize the two categories of "need to avoid" or "no need to avoid". This real-time processing capability is crucial to ensure the safety of the robot in a dynamic environment. The classifier provides fast and automatic decision support, reducing the reliance on human intervention.

[0060] In summary, the embodiment of the present application first obtains multi-area images of the mobile robot's surrounding environment collected by the camera, ultrasonic distance data between the mobile robot and the surrounding environment collected by the ultrasonic sensor, and environmental sound data at multiple predetermined time points collected by the sound sensor, and then uses deep learning technology to perform feature extraction and correlation analysis on the three. Finally, a classifier is used to determine whether the mobile robot needs to perform avoidance operations, thereby improving the mobile robot's ability to understand the surrounding environment in a complex and dynamic industrial environment, and ensuring efficient and safe operation of mobile trucks in industrial environments.

[0061] As described above, the mobile truck identification system 100 of the mobile robot according to the embodiment of the present application can be implemented in various terminal devices. In one example, the mobile truck identification system 100 of the mobile robot can be integrated into the terminal device as a software module and / or a hardware module. For example, the mobile truck identification system 100 of the mobile robot can be a software module in the operating system of the terminal device, or can be an application developed for the terminal device; of course, the mobile truck identification system 100 of the mobile robot can also be one of the many hardware modules of the terminal device.

[0062] Alternatively, in another example, the mobile robot's mobile truck identification system 100 and the terminal device may also be separate devices, and the mobile robot's mobile truck identification system 100 may be connected to the terminal device via a wired and / or wireless network and transmit interactive information in accordance with an agreed data format.

[0063] Figure 5 FIG. 1 is a flow chart of a mobile truck identification method of a mobile robot according to an embodiment of the present application. Figure 5As shown, the mobile truck identification method of the mobile robot according to the embodiment of the present application includes: S110, acquiring multi-region images of the surrounding environment of the mobile robot collected by a camera, ultrasonic distance data between the mobile robot and the surrounding environment collected by an ultrasonic sensor, and environmental sound data at multiple predetermined time points collected by a sound sensor; S120, extracting a surrounding environment truck tracking feature vector and an environmental multimodal association feature vector from the multi-region images of the surrounding environment of the mobile robot collected by the camera, the ultrasonic distance data between the mobile robot and the surrounding environment collected by the ultrasonic sensor, and the environmental sound data at multiple predetermined time points collected by the sound sensor; S130, judging whether the mobile robot needs to perform an avoidance operation based on the surrounding environment truck tracking feature vector and the environmental multimodal association feature vector.

[0064] Here, those skilled in the art will appreciate that the specific operations of each step in the above-mentioned mobile robot mobile truck identification method have been described in detail above. Figures 1 to 4 The mobile truck identification system of the mobile robot has been described in detail, and therefore, its repeated description will be omitted.

[0065] Below, reference Figure 6 To describe an electronic device according to an embodiment of the present application.

[0066] like Figure 6 As shown, the electronic device 10 includes an input device 11, an input interface 12, a central processing unit 13, a memory 14, an output interface 15, an output device 16, and a bus 17. The input interface 12, the central processing unit 13, the memory 14, and the output interface 15 are connected to each other through the bus 17, and the input device 11 and the output device 16 are connected to the bus 17 through the input interface 12 and the output interface 15, respectively, and then connected to other components of the electronic device 10.

[0067] Specifically, the input device 11 receives input information from the outside and transmits the input information to the central processing unit 13 through the input interface 12; the central processing unit 13 processes the input information based on the computer executable instructions stored in the memory 14 to generate output information, temporarily or permanently stores the output information in the memory 14, and then transmits the output information to the output device 16 through the output interface 15; the output device 16 outputs the output information to the outside of the electronic device 10 for user use.

[0068] In one embodiment, Figure 6The electronic device 10 shown can be implemented as a network device, which may include: a memory configured to store programs; a processor configured to run the programs stored in the memory to execute any one of the mobile robot mobile truck identification methods described in the above embodiments.

[0069] According to an embodiment of the present application, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present application includes a computer program product, which includes a computer program tangibly contained on a machine-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network, and / or installed from a removable storage medium.

[0070] It will be appreciated by those skilled in the art that all or some of the steps, systems, and functional modules / units in the methods disclosed above may be implemented as software, firmware, hardware, and appropriate combinations thereof. In hardware implementations, the division between the functional modules / units mentioned in the above description does not necessarily correspond to the division of physical components; for example, a physical component may have multiple functions, or a function or step may be performed by several physical components in cooperation. Some or all physical components may be implemented as software executed by a processor, such as a central processing unit, a digital signal processor, or a microprocessor, or implemented as hardware, or implemented as an integrated circuit, such as an application-specific integrated circuit. Such software may be distributed on a computer-readable medium, which may include a computer storage medium (or non-transitory medium) and a communication medium (or temporary medium). As known to those skilled in the art, the term computer storage medium includes volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing information (such as computer-readable instructions, data structures, program modules, or other data). Computer storage media include, but are not limited to, RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital versatile disks (DVD) or other optical disk storage, magnetic cassettes, magnetic tapes, magnetic disk storage or other magnetic storage devices, or any other medium that can be used to store the desired information and can be accessed by a computer. In addition, it is well known to those skilled in the art that communication media typically contain computer-readable instructions, data structures, program modules, or other data in a modulated data signal such as a carrier wave or other transport mechanism, and may include any information delivery media.

[0071] It is to be understood that the above embodiments are merely exemplary embodiments used to illustrate the principles of the present application, but the present application is not limited thereto. For those skilled in the art, various modifications and improvements can be made without departing from the spirit and substance of the present application, and these modifications and improvements are also considered to be within the scope of protection of the present application.

Claims

1. A mobile truck identification system for a mobile robot, characterized in that: include: A mobile truck identification data acquisition module is used to acquire multi-region images of the mobile robot's surrounding environment collected by the camera, ultrasonic distance data between the mobile robot and the surrounding environment collected by the ultrasonic sensor, and environmental sound data at multiple predetermined time points collected by the sound sensor; A mobile truck identification data extraction module is used to extract a surrounding truck tracking feature vector and an environment multimodal association feature vector from the multi-region images of the mobile robot's surrounding environment collected by the camera, the ultrasonic distance data between the mobile robot and the surrounding environment collected by the ultrasonic sensor, and the environmental sound data at multiple predetermined time points collected by the sound sensor; The mobile robot avoidance operation judgment module is used to judge whether the mobile robot needs to perform an avoidance operation based on the surrounding environment truck tracking feature vector and the environment multimodal association feature vector.

2. The mobile truck identification system of the mobile robot according to claim 1, characterized in that: The mobile truck identification data extraction module includes: A surrounding environment multi-region feature extraction unit, used for extracting features from the multi-region images of the surrounding environment of the mobile robot collected by the camera to obtain the surrounding environment truck tracking feature vector; An ultrasonic distance feature extraction unit, used for extracting features from the ultrasonic distance data between the mobile robot and the surrounding environment collected by the ultrasonic sensor to obtain an obstacle distance semantic feature vector; An environmental sound feature extraction unit, used for extracting features from the environmental sound data at a plurality of predetermined time points collected by the sound sensor to obtain an environmental sound space state feature vector; The environmental multimodal feature fusion unit is used to fuse the obstacle distance semantic feature vector and the environmental sound space state feature vector to obtain the environmental multimodal association feature vector.

3. The mobile truck identification system of the mobile robot according to claim 2, characterized in that: The surrounding environment multi-region feature extraction unit comprises: A surrounding environment multi-region feature encoding subunit, used for performing feature encoding on the multi-region images of the surrounding environment of the mobile robot collected by the camera to obtain a surrounding environment truck tracking feature map; The surrounding truck feature downsampling subunit is used to downsample the surrounding truck tracking feature map to obtain the surrounding truck tracking feature vector.

4. The mobile truck identification system of the mobile robot according to claim 3, characterized in that: The surrounding environment multi-region feature encoding subunit includes: Passing the multi-region images of the mobile robot's surrounding environment collected by the camera through a surrounding environment image denoiser based on deep learning to obtain denoised and enhanced images of the surrounding environment in multiple regions; Passing the plurality of regional surrounding environment denoised and enhanced images through a surrounding environment truck tracking YOLO target detection network as a feature extractor to obtain a plurality of regional surrounding environment truck tracking regions of interest; The plurality of regions of interest of surrounding truck tracking are passed through a surrounding truck tracking feature encoder to obtain the surrounding truck tracking feature map.

5. The mobile truck identification system of the mobile robot according to claim 4, characterized in that: The ultrasonic distance feature extraction unit comprises: The ultrasonic distance data between the mobile robot and the surrounding environment collected by the ultrasonic sensor is passed through an obstacle distance text convolutional neural network to obtain a plurality of obstacle distance feature vectors; The multiple obstacle distance feature vectors are concatenated into the obstacle distance semantic feature vector.

6. The mobile truck identification system of the mobile robot according to claim 5, characterized in that: The environmental sound feature extraction unit comprises: Extracting an ambient sound spectrogram from the ambient sound data at a plurality of predetermined time points collected by the sound sensor; The ambient sound spectrum graph is passed through an ambient sound spectrum spatial focusing feature encoder to obtain the ambient sound spatial state feature vector.

7. The mobile truck identification system of the mobile robot according to claim 6, characterized in that: The mobile robot avoidance operation judgment module includes: An environmental information feature association unit, used to associate the surrounding environment truck tracking feature vector with the environmental multimodal association feature vector to obtain a robot avoidance judgment classification feature vector; An environmental information feature optimization unit, used for performing a quasi-coherent interference correction based on weight adaptation on the robot avoidance judgment classification feature vector to obtain an optimized robot avoidance judgment classification feature vector; The avoidance operation classification judgment unit is used to pass the optimized robot avoidance judgment classification feature vector through a classifier to obtain a classification result, and the classification result is used to judge whether the mobile robot needs to perform an avoidance operation.

8. The mobile truck identification system of the mobile robot according to claim 7, characterized in that: The environmental information feature optimization unit includes: Calculating the product between the robot avoidance judgment classification feature vector and its transposed vector to obtain a robot avoidance judgment feature autocorrelation expression matrix; Calculating the coherent interference phase between the category decoding weight matrix of the classifier and the robot avoidance judgment feature autocorrelation expression matrix to obtain a robot avoidance judgment feature-weight fine-grained coupling representation matrix; The robot avoidance judgment feature-weight fine-grained coupling representation matrix is ​​subjected to probabilistic processing based on a Sigmoid activation function to obtain a robot avoidance judgment feature-weight fine-grained coupling probabilistic representation matrix; Based on the robot avoidance judgment feature-weight fine-grained coupling probabilistic representation matrix, back-propagation expression compensation is performed on the robot avoidance judgment classification feature vector to obtain a related compensation representation vector; A position-weighted sum between the relevant compensation representation vector and the robot avoidance judgment classification feature vector is calculated to obtain the optimized robot avoidance judgment classification feature vector.

9. A mobile truck identification method of a mobile robot, characterized in that: include: Acquire multi-region images of the mobile robot's surrounding environment collected by the camera, ultrasonic distance data between the mobile robot and the surrounding environment collected by the ultrasonic sensor, and environmental sound data at multiple predetermined time points collected by the sound sensor; Extracting a surrounding truck tracking feature vector and an environment multimodal association feature vector from the multi-region images of the surrounding environment of the mobile robot collected by the camera, the ultrasonic distance data between the mobile robot and the surrounding environment collected by the ultrasonic sensor, and the environmental sound data at multiple predetermined time points collected by the sound sensor; Based on the surrounding environment truck tracking feature vector and the environment multimodal association feature vector, it is determined whether the mobile robot needs to perform an avoidance operation.

10. The mobile truck identification method of the mobile robot according to claim 9, characterized in that: Extracting a surrounding truck tracking feature vector and an environment multimodal association feature vector from the multi-region images of the surrounding environment of the mobile robot collected by the camera, the ultrasonic distance data between the mobile robot and the surrounding environment collected by the ultrasonic sensor, and the environmental sound data at multiple predetermined time points collected by the sound sensor, including: Performing feature extraction on the multi-region images of the mobile robot's surrounding environment captured by the camera to obtain a truck tracking feature vector of the surrounding environment; Performing feature extraction on the ultrasonic distance data between the mobile robot and the surrounding environment collected by the ultrasonic sensor to obtain an obstacle distance semantic feature vector; Performing feature extraction on the ambient sound data at a plurality of predetermined time points collected by the sound sensor to obtain an ambient sound space state feature vector; The obstacle distance semantic feature vector and the ambient sound space state feature vector are fused to obtain the ambient multimodal association feature vector.

Images