Industrial equipment data processing method and system based on radar and video fusion calculation
By using the fusion calculation method of radar and video data in industrial equipment, multiple embedding operations and abnormal identification are carried out, the heterogeneity and redundancy problems in multimodal data fusion are solved, and the accuracy of abnormal detection is improved and the accuracy and freedom of abnormality detection are improved. Great.
Patent Information
- Application Number
- CN202411268208.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-09-11
- Publication Date
- 2024-11-19
- Estimated Expiration
- 2044-09-11
AI Technical Summary
The existing multimodal data fusion method has heterogeneity problems in the abnormal detection of industrial equipment, making it difficult to effectively fuse video and radar data, resulting in loss of feature descriptors or interference with redundant information, affecting the abnormal identification effect, and lack of dynamic adjustment and optimization. Processing strategies limit the accuracy and robustness of exception recognition.
Using a method based on radar and video fusion computing, the video and point cloud descriptors are obtained, and the same representation domain is integrated, multiple embedding operations are performed using integrated embedding networks to obtain embedding descriptors, and abnormal situations are identified in combination with exception classification tags. Dynamically adjust processing strategies to improve identification accuracy.
It realizes the effective fusion of video and radar data, improves the accuracy and robustness of abnormal identification, ensures the focus of key information and the removal of redundant information, and improves the status monitoring and abnormal detection performance of industrial equipment.
Smart Images

Figure CN118799815B_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of data processing, and in particular, to an industrial equipment data processing method and system based on radar and video fusion calculation. Background Art
[0002] With the rapid development of industrial automation and intelligent manufacturing, higher requirements are placed on real-time monitoring and anomaly detection of equipment status in production environments. Traditional single sensor monitoring methods are often limited by monitoring range, accuracy or environmental adaptability, and are difficult to meet the comprehensive monitoring needs in complex industrial scenarios. Therefore, multimodal data fusion technology has gradually become a research hotspot, especially the fusion processing based on radar and video data, because it can combine the three-dimensional spatial perception ability of radar and the high-resolution visual information of video, providing a richer and more accurate data basis for anomaly detection of industrial equipment. However, existing multimodal data fusion methods still face many challenges in dealing with industrial equipment anomaly detection tasks. On the one hand, there is significant heterogeneity between different modal data. How to effectively fuse video data and radar data to extract comprehensive and representative feature descriptors is a key issue in current research. On the other hand, abnormal situations in industrial environments are often complex and changeable, and a single data processing model is difficult to adapt to all types of abnormal situations. Therefore, a method that can dynamically adjust and optimize processing strategies is needed to improve the accuracy and robustness of anomaly recognition. In addition, existing multimodal fusion methods often lack a selective focusing mechanism for different types of feature information during processing, resulting in the loss of key information or interference of redundant information during the fusion process, which in turn affects the effect of anomaly recognition. At the same time, most methods do not fully utilize the complementarity between different modal data when processing multimodal data, and fail to achieve effective integration and utilization of cross-modal information, which limits the improvement of anomaly detection performance. Summary of the invention
[0003] The object of the present invention is to provide an industrial equipment data processing method and system based on radar and video fusion calculation. The embodiment of the present application is implemented as follows:
[0004] In a first aspect, an embodiment of the present application provides an industrial equipment data processing method based on radar and video fusion calculation, including: obtaining an abnormal descriptor in video acquisition data corresponding to a target environment, and obtaining a point cloud descriptor corresponding to the target environment; wherein the target environment is the environment where the target industrial equipment is located; performing descriptor integration processing on the abnormal descriptor and the point cloud descriptor in the same representation domain to obtain a first fusion descriptor, and determining a first integrated embedding network associated with the abnormal descriptor in p integrated embedding networks based on the first fusion descriptor; the p integrated embedding networks are used to embed multiple different types of descriptors according to different interaction strengths, wherein p ≥ 1; in the first integration In the p integrated embedding network, the abnormal descriptor is integrated and embedded according to the point cloud descriptor to obtain a first embedded descriptor; the point cloud descriptor and the first embedded descriptor are integrated in the same representation domain to obtain a second fused descriptor, and according to the second fused descriptor, a second integrated embedding network associated with the point cloud descriptor is determined in the p integrated embedding networks; in the second integrated embedding network, the point cloud descriptor is integrated and embedded according to the first embedded descriptor to obtain a second embedded descriptor; an abnormal classification label is obtained, and abnormal situation identification information corresponding to the target environment is determined according to the abnormal classification label, the first embedded descriptor and the second embedded descriptor.
[0005] In a second aspect, the present application provides a computer system comprising: one or more processors; a memory; one or more computer programs; wherein the one or more computer programs are stored in the memory and configured to be executed by the one or more processors, and when the one or more computer programs are executed by the processors, the method as described above is implemented.
[0006] The beneficial effects of the present application include at least: in the embodiment of the present application, the abnormal descriptor in the video acquisition data corresponding to the target environment and the point cloud descriptor of the target environment can be obtained; the abnormal descriptor is used as the descriptor of the embedded data type, the point cloud descriptor is used as the descriptor of the comparison dimension, and a suitable first integrated embedding network is determined for the embedded data type from p integrated embedding networks. In the first integrated embedding network, the abnormal descriptor is integrated and embedded according to the point cloud descriptor to obtain the first embedded descriptor. The reverse operation can be performed, the point cloud descriptor is used as the descriptor of the embedded data type, the first embedded descriptor is used as the descriptor of the comparison data type, and a suitable second integrated embedding network is determined for the embedded data type. In the second integrated embedding network, the point cloud descriptor is integrated and embedded according to the first embedded descriptor to obtain the first embedded descriptor; in this way, classification can be performed according to the embedded first embedded descriptor, the second embedded descriptor and the abnormal classification label to obtain abnormal situation identification information of the target environment. The present application can perform round-trip embedding of different types of features, and each embedding can determine the most accurate integrated embedding network among multiple integrated embedding networks and perform integrated embedding simultaneously, and can selectively focus on specific types of information, and in different types of embedding links, with the help of information of comparative data types in different forms, anomaly identification is more accurate. BRIEF DESCRIPTION OF THE DRAWINGS
[0007] Figure 1 It is a flowchart of an industrial equipment data processing method based on radar and video fusion calculation provided in an embodiment of the present application.
[0008] Figure 2 It is a schematic diagram of the composition of a computer system provided in an embodiment of the present application. DETAILED DESCRIPTION
[0009] The execution subject of the industrial equipment data processing method based on radar and video fusion calculation in the embodiment of the present application is a computer system, including but not limited to a server, a personal computer, a laptop, a tablet computer, a smart phone, etc. Figure 1 As shown, the method includes:
[0010] Step S10: Obtain anomaly descriptors in the video acquisition data corresponding to the target environment, and obtain a point cloud descriptor corresponding to the target environment; wherein the target environment is the environment where the target industrial equipment is located.
[0011] As an implementation scenario, in step S10, it is assumed that the target environment is an automated assembly line, on which multiple high-definition cameras are installed to capture video data of the production process in real time. The computer system can receive a continuous video stream from these cameras and divide it into a series of video frames. For example, if the video frame rate is 30 frames per second, then 30 video frames will be captured and processed in one second.
[0012] For each video frame, the computer system applies a pre-trained feature extraction model (such as a convolutional neural network CNN) to identify key features in the image. These features may include the shape, edges, textures of objects, and their positional relationships in space. In the scenario of an automated assembly line, the feature extraction model pays special attention to robotic arms, conveyor belts, parts, and any elements that may indicate anomalies (such as dropped parts, stagnant robotic arms, etc.).
[0013] In order to extract descriptors that can represent the abnormal situation of the entire video sequence from this series of video frames, the computer system further adopts an attention mechanism (such as the self-attention layer in Transformer). This mechanism simulates the attention allocation process of the human visual system when observing complex scenes by assigning different weights (focus influence coefficients) to each video frame descriptor. Video frames with higher weights usually contain more significant or more critical abnormal information. Finally, the computer system sums the weighted video frame descriptors to obtain a comprehensive abnormal descriptor. This descriptor not only summarizes the key abnormal features in the video sequence, but also highlights the importance of these features through the attention mechanism, providing strong support for subsequent multimodal data fusion and abnormality recognition.
[0014] At the same time, the computer system also obtains point cloud data of the target environment through radar. Point cloud data is composed of a large number of points in three-dimensional space, each of which contains information such as position (x, y, z coordinates) and possible color, reflectivity, etc. In the scenario of automated assembly lines, point cloud data can provide accurate three-dimensional models of equipment position, posture, and surrounding environment.
[0015] In order to extract useful descriptors from point cloud data, the computer system uses an affine network (such as a fully connected neural network) to process the point cloud tensor. A point cloud tensor is a multidimensional array that contains the coordinates and other attributes of all points. The affine network maps the point cloud tensor to a low-dimensional feature space through a series of linear transformations and nonlinear activation functions, thereby extracting key features that can characterize the entire point cloud data. These key features constitute the point cloud descriptors, which not only reflect the geometric structure of the target environment, but also may imply information related to abnormal events (such as abnormal position of the robot arm, incorrect stacking of parts, etc.). Together with the video anomaly descriptors, the point cloud descriptors provide a rich multimodal data source for subsequent fusion calculations and anomaly recognition tasks.
[0016] Step S20: performing descriptor integration processing on the abnormal descriptor and the point cloud descriptor in the same representation domain to obtain a first fused descriptor, and determining a first integrated embedding network associated with the abnormal descriptor in p integrated embedding networks based on the first fused descriptor; the p integrated embedding networks are used to embed multiple different types of descriptors according to different interaction strengths, where p≥1.
[0017] In step S20, first, the computer system ensures that the anomaly descriptor and the point cloud descriptor are in the same feature space so that they can be compared and integrated with each other. This is usually achieved by applying an appropriate projection function or transformation matrix. For example, two perceptron networks in a mixture of experts model can be used, one for processing the anomaly descriptor and the other for processing the point cloud descriptor. The two perceptron networks can project their respective descriptors into a common, high-dimensional feature space, so that the anomaly descriptor and the point cloud descriptor are comparable in the space.
[0018] During the projection process, the perceptron network processes the input descriptor layer by layer through a series of linear transformations and nonlinear activation functions, and gradually extracts higher-level feature representations. These feature representations not only retain the key information in the original descriptor, but also introduce new combined features through nonlinear transformations, thereby enhancing the expressive power of the descriptor.
[0019] After the projection is completed, the computer system integrates the abnormal projection descriptor and the point cloud projection descriptor. The integration operation may include simple feature concatenation, weighted summation, or more complex feature fusion algorithms (such as attention mechanism, bilinear pooling, etc.). In this example, it is assumed that a weighted summation method based on attention weights is adopted to dynamically adjust the contribution of each feature according to its importance, so as to obtain the first fusion descriptor. After obtaining the first fusion descriptor, the computer system determines a first integrated embedding network that best matches the abnormal descriptor among the p integrated embedding networks. These integrated embedding networks are a collection of pre-trained models that can embed multiple types of descriptors according to different interaction strengths (i.e., fusion strengths). Each integrated embedding network has its own specific architecture and parameter settings to adapt to different types of fusion tasks and scenario requirements.
[0020] In order to select the best first integrated embedding network, the computer system can use the category output network (such as a softmax classifier) in the expert mixture model. The network performs a category mapping operation on the first fusion descriptor and outputs a probability distribution vector indicating the probability of each integrated embedding network being selected. The computer system selects the integrated embedding network with the highest probability as the first integrated embedding network to ensure that the subsequent embedding processing can maximize the retention and highlighting of key information in the abnormal descriptor.
[0021] For example, suppose that in an automated warehouse, a computer system monitors the handling and storage process of goods through video cameras and radar sensors. The video data captures the appearance and position information of the goods, while the radar data provides the three-dimensional shape and motion trajectory of the goods. In step S20, the computer system first projects the abnormal descriptors (such as falling goods, position displacement, etc.) in the video data and the point cloud descriptors in the radar data into the same feature space. Then, a first fusion descriptor is obtained through integration processing, which contains both the appearance features of the goods and their three-dimensional shape and motion information. Next, the computer system uses the category output network to select a network from multiple integrated embedding networks that is most suitable for the current fusion task. For example, if the abnormal descriptor mainly focuses on the position displacement problem of the goods, then the system can select an integrated embedding network that focuses more on spatial position information. Once the first integrated embedding network is determined, the system can further integrate and embed the abnormal descriptors and point cloud descriptors in the network to provide more accurate feature representation for subsequent abnormal recognition tasks.
[0022] Step S30: In the first integrated embedding network, the abnormal descriptor is integrated and embedded according to the point cloud descriptor to obtain a first embedded descriptor.
[0023] In step S30, first, the computer system determines a first integrated embedding network, which is the best matching network selected from a plurality of pre-trained integrated embedding networks according to the first fusion descriptor. This network is specially designed to handle the fusion task between descriptors of different types (modalities), and can adjust its internal parameters and structure according to data characteristics and task requirements to achieve the best fusion effect.
[0024] Inside the first integrated embedding network, the integrated embedding operation usually involves complex nonlinear transformations and attention mechanisms to ensure that the three-dimensional spatial information in the point cloud descriptor and the visual information in the anomaly descriptor can be effectively combined. For example, if the first integrated embedding network is an Attention Fusion Network, it will first map the anomaly descriptor and the point cloud descriptor to a common high-level feature space through a series of transformation layers (such as fully connected layers or convolutional layers). In this feature space, the weighted focused integration network uses the attention mechanism to dynamically adjust the importance of different features. Specifically, the network can generate query vectors and key vectors to simulate the attention process, where the query vector comes from the anomaly descriptor and the key vector comes from the point cloud descriptor. By calculating the similarity between the query vector and the key vector, the network can assign a weight to each feature in the anomaly descriptor, which reflects the importance of the corresponding information in the point cloud descriptor to the current feature.
[0025] Next, the computer system performs a weighted summation of the features in the anomaly descriptor according to these weights to obtain a first embedded descriptor that incorporates point cloud information. This first embedded descriptor not only contains the key visual features in the original anomaly descriptor, but also introduces the three-dimensional spatial information and contextual information in the point cloud descriptor through the attention mechanism, thereby forming a richer and more comprehensive feature representation. For example, taking the abnormality detection of a robotic arm on an automated production line as an example, the video camera captures the abnormal motion trajectory or posture of the robotic arm (anomaly descriptor), while the radar sensor provides accurate three-dimensional point cloud data of the robotic arm and its surroundings (point cloud descriptor). In step S30, the computer system first integrates and embeds the data of these two modalities through a first integrated embedding network.
[0026] In this process, the network can pay special attention to key features such as the position offset and posture anomaly of the robot arm, and use the spatial information and obstacle distribution in the point cloud data to dynamically adjust the weights of these features. For example, if the point cloud data shows that the robot arm is about to collide with other parts on the production line, the network can increase the weight of features related to the risk of collision. The final generated first embedding descriptor will comprehensively reflect the abnormal situation of the robot arm and the risk factors of the surrounding environment, providing strong support for subsequent abnormal identification and response decisions.
[0027] Step S40: performing descriptor integration processing on the point cloud descriptor and the first embedding descriptor in the same representation domain to obtain a second fused descriptor, and determining a second integrated embedding network associated with the point cloud descriptor in the p integrated embedding networks based on the second fused descriptor.
[0028] In step S40, the computer system continues to deepen its industrial equipment data processing flow based on radar and video fusion calculation, and further explores the potential connection between multimodal data by integrating the point cloud descriptor and the first embedded descriptor in the same representation domain. This process not only consolidates the information extracted and fused in the previous steps, but also provides a more comprehensive and in-depth data foundation for subsequent analysis.
[0029] The computer system first ensures that the point cloud descriptor and the first embedded descriptor are in the same representation domain, which is a prerequisite for integration processing. If the two are not originally in the same feature space, additional projection or transformation operations may be required to align their feature dimensions and semantic spaces.
[0030] In the same representation domain, the computer system uses an integration method similar to step S20 to integrate the point cloud descriptor and the first embedding descriptor. This integration may involve a variety of techniques such as feature splicing, weighted summation, bilinear pooling, etc. The specific choice depends on the characteristics of the data and the requirements of the task. For example, if the point cloud descriptor focuses on the accurate expression of three-dimensional spatial information, and the first embedding descriptor combines visual information and attention weights, then the integration process can tend to retain the key features of these two aspects and introduce new combined features through nonlinear transformations.
[0031] Through the integration process, the computer system generates a second fused descriptor. This descriptor not only contains the three-dimensional spatial information and shape features in the point cloud descriptor, but also integrates the visual features and attention weighted information in the first embedded descriptor to form a more comprehensive and powerful feature representation. After obtaining the second fused descriptor, the computer system again uses the strategy adopted in step S20 to determine a second integrated embedded network that best matches the current task from p pre-trained integrated embedded networks. This selection process also relies on the category output network in the expert mixture model. By performing category mapping on the second fused descriptor, the probability of each integrated embedded network being selected is output, and the network with the highest probability is selected as the second integrated embedded network.
[0032] It is worth noting that although the selection process of the second integrated embedding network is similar to step S20, the final selected network can be different due to the different input descriptors (i.e., the second fused descriptor) and the context of the task (i.e., the fusion result of the point cloud descriptor and the first embedded descriptor is currently processed). This flexibility ensures that the computer system can dynamically adjust its processing flow according to different data characteristics and task requirements to achieve optimal performance.
[0033] For example, taking robot navigation in a smart manufacturing workshop as an example, the radar sensor provides accurate three-dimensional point cloud data (point cloud descriptor) of obstacles and robots in the workshop, while the video camera captures the robot's motion trajectory and changes in the surrounding environment (the first embedded descriptor generated by step S30). In step S40, the computer system integrates the data of these two modalities to generate a second fusion descriptor.
[0034] This descriptor not only contains the precise position, posture and speed information of the robot (from the point cloud descriptor), but also integrates the dynamic changes and attention weights in the video data (from the first embedded descriptor), so that the computer system can more accurately understand the current state of the robot's environment. Subsequently, by selecting the most appropriate second integrated embedding network, the computer system can further embed the second fusion descriptor to support subsequent high-level tasks such as path planning and obstacle avoidance decisions.
[0035] Step S50: In the second integrated embedding network, an integrated embedding operation is performed on the point cloud descriptor according to the first embedded descriptor to obtain a second embedded descriptor.
[0036] In step S50, the computer system first activates a second integrated embedding network, which is a network carefully selected from multiple pre-trained integrated embedding networks according to the second fusion descriptor, and is particularly good at processing the specific type of data fusion required for the current task. The second integrated embedding network may use a complex neural network structure, such as a deep convolutional network, a recurrent neural network, or a Transformer model, which can capture high-level features and long-term dependencies in the data.
[0037] In the integrated embedding operation, the first embedded descriptor acts as a guiding information and affects the point cloud descriptor. Specifically, the first embedded descriptor may contain key visual features and attention weights extracted from the video data, which guide the second integrated embedding network on how to better understand and process the three-dimensional spatial information in the point cloud descriptor. For example, if the first embedded descriptor highlights the abnormal state of a specific area or object, the second integrated embedding network can increase the attention paid to the processing of the corresponding point cloud area to obtain more detailed spatial information and shape features.
[0038] The integrated embedding operation usually involves a series of complex nonlinear transformations and attention mechanisms. Within the second integrated embedding network, the point cloud descriptor and the first embedded descriptor can be mapped to a higher-dimensional feature space, and the features can be interacted and fused through the interaction layer. The interaction layer may use dot product attention, multi-head attention or other advanced attention mechanisms to dynamically adjust the weights and contributions between different features. Finally, through the integrated embedding operation, the computer system generates a second embedded descriptor, which is a comprehensive feature representation that combines video visual information and radar three-dimensional spatial information. The second embedded descriptor not only contains the three-dimensional spatial information and shape features in the original point cloud descriptor, but also integrates the dynamic changes, attention weights and other key visual information in the video data through the guidance of the first embedded descriptor. This cross-modal deep fusion makes the second embedded descriptor have stronger expression ability and higher analysis accuracy, and can more accurately reflect the current status of industrial equipment and environmental changes.
[0039] For example, taking the container lifting operation in the automated port as an example, the radar sensor provides accurate three-dimensional point cloud data (point cloud descriptor) of the lifting area and the container, while the video camera captures the dynamic changes and potential risks in the lifting process (through the first embedding descriptor generated in the previous step). In step S50, the computer system uses the second integrated embedding network to integrate and embed the data of these two modalities.
[0040] Specifically, the first embedding descriptor may highlight the offset or unstable state of a container during the lifting process, while the point cloud descriptor provides the precise positional relationship between the container and the lifting equipment. In the second integrated embedding network, this information is deeply fused and processed to generate the second embedding descriptor. This descriptor not only contains the three-dimensional spatial information of the container and the lifting equipment, but also incorporates the visual features of dynamic changes and potential risks during the lifting process, allowing the computer system to more accurately assess the safety and efficiency of the lifting operation and take necessary intervention measures in a timely manner.
[0041] Step S60: Acquire an abnormal classification label, and determine abnormal situation identification information corresponding to the target environment based on the abnormal classification label, the first embedded descriptor, and the second embedded descriptor.
[0042] In step S60, first, the computer system obtains anomaly classification labels from a data source or an external system. These labels are usually predefined classifications of known abnormal situations, which are used to guide the training and reasoning process of the anomaly recognition model. In the scenario of industrial equipment data processing, anomaly classification labels may include multiple types such as mechanical failure, operational error, environmental interference, etc., each type corresponds to specific abnormal manifestations and characteristics.
[0043] After obtaining the abnormal classification labels, the computer system combines these labels with the first embedding descriptor and the second embedding descriptor to identify the abnormal situation. This process may involve one or more machine learning models (such as classifiers, regressors, or clustering algorithms) that have been trained to learn and understand the complex relationship between different abnormal situations and multimodal data features.
[0044] Specifically, the computer system may use a fusion classifier that can simultaneously process the information in the first embedded descriptor and the second embedded descriptor and make a decision in combination with the abnormal classification label. The fusion classifier may use a variety of fusion strategies, such as feature-level fusion (directly concatenating or weighted summing descriptor features in the feature space), decision-level fusion (first making a classification decision for each descriptor separately, and then fusing the decision results) or hybrid fusion (combining the advantages of feature-level and decision-level fusion).
[0045] Taking feature-level fusion as an example, the computer system first concatenates the first embedded descriptor and the second embedded descriptor into a longer feature vector, and then inputs this feature vector into the fusion classifier. The classifier may contain multiple hidden layers (such as fully connected layers, convolutional layers, or recurrent layers) to extract high-level features and perform nonlinear transformations. Finally, the classifier outputs a probability distribution vector, which indicates the probability that the input data belongs to each abnormal classification label. Based on this probability distribution vector, the computer system selects the label with the highest probability as the abnormal situation identification information of the target environment.
[0046] For example, taking the production line monitoring in the intelligent manufacturing workshop as an example, the radar and video sensors collect data of each device and workpiece on the production line in real time, and generate the first embedded descriptor and the second embedded descriptor through the above steps. In step S60, the computer system obtains a predefined abnormal classification label library, which covers various abnormal situations that may occur on the production line (such as equipment shutdown, missing parts, product quality problems, etc.). Subsequently, the computer system inputs these labels and the first embedded descriptor and the second embedded descriptor into the fusion classifier. The classifier identifies the abnormal situation on the current production line through a comprehensive analysis of multimodal data and outputs the corresponding abnormal situation identification information. For example, if the classifier determines that there is an abnormal equipment shutdown at present, the computer system will immediately issue an alarm and notify relevant personnel to conduct inspections and repairs to ensure the normal operation of the production line. Through this process, the computer system not only improves the accuracy and efficiency of abnormal identification, but also provides strong support for the intelligent management and maintenance of industrial equipment.
[0047] As an implementation manner, in step S10, obtaining anomaly descriptors in the video acquisition data corresponding to the target environment may specifically include:
[0048] Step S11: obtaining video acquisition data corresponding to the target environment, and dividing the video acquisition data into q video frames; wherein q≥2;
[0049] Step S12: performing descriptor representation on the q video frames to obtain video frame embedding descriptors corresponding to the q video frames;
[0050] Step S13: performing a weighted focusing operation on the video frame embedding descriptors corresponding to the q video frames to obtain focusing influence coefficients corresponding to the q video frames;
[0051] Step S14: multiplying the video frame embedding descriptors corresponding to the q video frames and the focus influence coefficients corresponding to the q video frames and summing them up to obtain an abnormal descriptor corresponding to the video acquisition data.
[0052] In the embodiment of the present application, the high-definition camera on the automated production line continuously captures the video images of the production area, and the video data is transmitted to the computer system in real time for processing. In order to efficiently analyze the data, the computer system first needs to divide it into a series of video frames.
[0053] The camera captures the video stream at a fixed frame rate (such as 30 frames per second), and each frame is a static image of the production line at a certain moment. After receiving the video stream, the computer system divides it into q video frames in chronological order. The specific value of q depends on the processing requirements and system resources. For example, if a higher temporal resolution is required, the q value will be relatively large and the time interval between each frame will be short; conversely, if the system processing capacity is limited, the q value can be appropriately reduced to reduce the computational burden.
[0054] For example, assuming the frame rate is 30 frames per second, and the computer system chooses to process 10 frames of data per second to maintain a balance between real-time and processing efficiency, then q = 10. This means that the video stream per second is divided into 10 independent video frames, each of which represents the status of the production line at one-tenth of the time point within that second.
[0055] After dividing the video stream into video frames, the computer system performs feature extraction on each video frame to convert it into a numerical form understandable by the computer (i.e., embedded descriptors). These descriptors should contain enough information to characterize the key elements and events in the video frame.
[0056] The computer system uses a pre-trained deep learning model (such as a convolutional neural network (CNN)) to extract features from video frames. CNN extracts low-level to high-level features such as edges, textures, shapes, and objects from video frames layer by layer through multiple layers of convolution, pooling, and activation functions. The video frames processed by CNN are converted into a series of feature vectors (i.e., embedded descriptors). Each embedded descriptor is a high-dimensional vector whose dimension depends on the size of the output layer of CNN. These vectors compactly represent the key information of the video frame in the feature space. For example, suppose that the CNN model converts each video frame into a 1024-dimensional feature vector. For a sequence of video frames with q=10, the computer system will generate 10 1024-dimensional embedded descriptors, each corresponding to one video frame. For example, the first embedded descriptor may represent the initial state of the production line when it starts, while the last embedded descriptor reflects the state of the production line at the end of that second.
[0057] Since a video frame sequence may contain a lot of redundant information or insignificant details, the computer system needs a mechanism to highlight the video frames that are most valuable for anomaly recognition. The attention mechanism is such a tool, which can dynamically adjust the importance weights of different video frame descriptors.
[0058] The computer system uses an attention model (such as the self-attention mechanism in Transformer) to calculate the focusing influence coefficient (i.e., weighted focusing operation) of each video frame descriptor. The attention model compares the similarities between different descriptors and assigns a weight to each descriptor, which reflects the importance of the descriptor in the current task.
[0059] Specifically, the attention model can generate query vectors, key vectors, and value vectors. Each video frame descriptor is converted into a key vector and a value vector, and the query vector may be a learnable parameter or dynamically generated according to the task context. By calculating the similarity (such as dot product) between the query vector and all key vectors, the attention model assigns a weight to each value vector, which is the focus influence coefficient of the video frame. For example, suppose that the attention model generates a focus influence coefficient between 0 and 1 for each video frame descriptor. The sum of these coefficients is 1, ensuring the normalization of the weights. For example, if a video frame contains obvious signs of equipment failure (such as sparks, abnormal sounds, or motion trajectories), the focus influence coefficient of the video frame can be much higher than that of other frames, indicating that it has a higher importance in abnormal situation recognition.
[0060] Finally, the embedded descriptor of each video frame is multiplied by its corresponding focus influence coefficient, and all weighted descriptors are summed to generate a comprehensive anomaly descriptor. This anomaly descriptor not only contains the key information in the video frame sequence, but also highlights the most important parts through the attention mechanism. The computer system multiplies the embedded descriptor of each video frame by its focus influence coefficient to obtain a weighted descriptor. Then, all weighted descriptors are added to obtain a single, high-dimensional anomaly descriptor. This descriptor is used as a compact representation of the entire video acquisition data for subsequent anomaly recognition tasks. For example, suppose that 10 weighted 1024-dimensional video frame descriptors have been obtained (each descriptor is multiplied by its corresponding focus influence coefficient), and now these descriptors are added to obtain the final anomaly descriptor. This anomaly descriptor is also a 1024-dimensional vector, but the values on each dimension are obtained by weighted summation according to the importance of all video frames. Therefore, it is more likely to highlight the feature dimensions related to the anomaly and suppress insignificant details.
[0061] Through the above steps S11 to S14, the computer system extracts the abnormal descriptor from the video acquisition data of the automated production line. This process not only involves basic operations such as video frame division, feature extraction, and generation of embedded descriptors, but also cleverly uses the attention mechanism to dynamically adjust the importance weights of different video frames. The resulting abnormal descriptor provides strong support for subsequent abnormal situation identification tasks, enabling the computer system to more accurately identify potential problems on the production line and take timely measures to solve them.
[0062] This industrial equipment data processing method based on radar and video fusion computing not only improves the accuracy and efficiency of abnormality identification, but also opens up new ways for the intelligent management and maintenance of industrial equipment. By continuously optimizing and expanding this method, it is expected to achieve a more efficient, safe and sustainable industrial production environment in the future.
[0063] As an implementation manner, in step S10, obtaining a point cloud descriptor corresponding to the target environment may specifically include:
[0064] Step S15: obtaining r environment point cloud matrices corresponding to the target environment, and combining the r environment point cloud matrices into an environment point cloud tensor; r≥1;
[0065] Step S16: Load the environment point cloud tensor into the affine network, perform an affine transformation operation on the environment point cloud tensor based on the affine network, and obtain a point cloud descriptor corresponding to the target environment.
[0066] In one scenario of the present application, for example, in an automated warehouse, a radar sensor scans the surrounding environment at a certain frequency to generate a series of environmental point cloud matrices. Each point cloud matrix represents a set of three-dimensional coordinate points of objects (such as shelves, goods, robots, etc.) inside the warehouse at a certain moment or perspective. These point cloud matrices are the basic data for subsequent analysis and processing.
[0067] Radar sensors measure information such as the distance, angle, and speed of surrounding objects by emitting electromagnetic waves and receiving their reflected signals. This information is converted into coordinate points in three-dimensional space and stored in the form of a point cloud matrix. Each point cloud matrix usually contains the coordinates (x, y, z) of multiple points and possible other attributes such as intensity and reflectivity. In order to improve the integrity and accuracy of point cloud data, radar sensors can scan from multiple perspectives or positions. In this way, for the same object in the warehouse, multiple point cloud matrices can be generated, each of which reflects the three-dimensional shape and position of the object from different perspectives.
[0068] After receiving these point cloud matrices, the computer system needs to combine them into a larger environment point cloud tensor. The environment point cloud tensor is a multidimensional array whose dimensions depend on the number of point cloud matrices (r), the number of points in each matrix, and the number of attributes of each point. By combining multiple point cloud matrices, the computer system is able to build a more comprehensive 3D model of the warehouse.
[0069] For example, suppose a radar sensor scans a warehouse from three different perspectives over a period of time, generating three environmental point cloud matrices (r=3). Each matrix contains coordinate information for 1,000 points. The computer system combines these three matrices into an environmental point cloud tensor with a shape of [3, 1000, 3], where 3 represents the number of matrices (or the number of perspectives), 1000 represents the number of points in each matrix, and 3 represents the coordinate dimension (x, y, z) of each point.
[0070] After obtaining the environment point cloud tensor, the computer system converts it into a more compact and easier to process form, namely the point cloud descriptor. The point cloud descriptor should be able to capture the key features in the point cloud data, such as shape, position, distribution, etc., so as to perform subsequent tasks such as abnormal situation recognition or path planning. An affine network is a neural network that can perform linear transformation and nonlinear activation operations on input data. In point cloud processing, a fully connected network (FCN) is a common choice of affine network because it can process input data of any dimension and extract high-level features through connections between multiple layers of neurons. The computer system takes the environment point cloud tensor as the input of the affine network and processes it layer by layer through multiple layers of neurons in the network. Each layer of neurons performs linear transformation (i.e., weighted summation) and nonlinear activation (such as ReLU activation function) on the environment point cloud tensor to extract higher-level feature representations. These feature representations gradually transition from low-level geometric shape and position information to high-level semantic information and structural information. After layer-by-layer processing by the affine network, a fixed-length vector is finally generated as the point cloud descriptor. This vector contains the key information in the environment point cloud tensor and represents it in a compact form. The dimension of the point cloud descriptor is usually much smaller than that of the original point cloud tensor, but it can more effectively support subsequent processing and analysis tasks.
[0071] For example, suppose the selected affine network is a neural network with three fully connected layers. The first layer has 1024 neurons, the second layer has 512 neurons, and the third layer (output layer) has 128 neurons. The environment point cloud tensor (shape [3, 1000, 3]) is first flattened into a one-dimensional vector (length 9000) and then used as the input of the first fully connected layer. After processing by three fully connected layers, a 128-dimensional vector is finally output as the point cloud descriptor. This descriptor not only contains the three-dimensional shape and position information of the objects in the warehouse, but also may imply high-level features such as the spatial relationship and distribution pattern between objects.
[0072] Through the above steps S15 and S16, the computer system extracts point cloud descriptors from the radar data of the automated warehouse. This process first constructs a comprehensive warehouse 3D model by combining the environmental point cloud matrices under multiple perspectives; then uses an affine network (such as a fully connected network) to process the point cloud tensor layer by layer, extract key feature information, and express it in a compact vector form. As the basic data for subsequent processing and analysis, the point cloud descriptor will support the computer system to perform more efficient tasks such as abnormal situation identification and path planning. This industrial equipment data processing method based on radar and video fusion computing not only improves the accuracy and efficiency of data processing, but also provides new technical support for the intelligent management and operation of automated warehouses.
[0073] As an implementation manner, the step S20, performing descriptor integration processing on the anomaly descriptor and the point cloud descriptor in the same representation domain to obtain a first fused descriptor, may specifically include:
[0074] Step S21: loading the anomaly descriptor and the point cloud descriptor into an expert mixture model, and performing representation domain projection on the anomaly descriptor based on a first perceptron network in the expert mixture model to obtain an anomaly projection descriptor;
[0075] Step S22: Based on the second perceptron network in the expert mixture model, the point cloud descriptor is projected into a representation domain to obtain a point cloud projection descriptor; the abnormal projection descriptor and the point cloud projection descriptor correspond to the same representation domain;
[0076] Step S23: performing descriptor integration processing on the abnormal projection descriptor and the point cloud projection descriptor to obtain a first fusion descriptor.
[0077] The core goal of step S20 is to integrate the anomaly descriptor and the point cloud descriptor in the same representation domain to generate a comprehensive first fusion descriptor. This step is the key to achieving multimodal data fusion, which requires the computer system to intelligently convert data from different sources (video and radar) into the same feature space for subsequent analysis and processing.
[0078] For example, in a smart manufacturing workshop, the video data captured by the high-definition camera generates an abnormal descriptor after the pre-processing step (as described in step S10). At the same time, the point cloud data collected by the radar sensor is also converted into a point cloud descriptor. These two descriptors represent the visual features and three-dimensional spatial information of the workshop environment respectively.
[0079] The Mixture of Experts Model is a machine learning architecture that can dynamically select different processing paths based on different inputs. It usually contains multiple "expert" networks (i.e., sub-models), each of which is good at processing a specific type of data or task. In this embodiment, the Mixture of Experts Model contains at least two Perceptron Networks, which are used to process anomaly descriptors and point cloud descriptors respectively. The first Perceptron Network is responsible for processing anomaly descriptors. It is a multi-layer feedforward neural network that projects the anomaly descriptor from the original feature space to a new representation domain (i.e., feature space) through a series of linear transformations and nonlinear activation functions. This new representation domain is designed to align with the point cloud projection descriptor for subsequent integration processing. Similar to the first Perceptron Network, the second Perceptron Network processes the point cloud descriptor. It also uses the structure of a multi-layer feedforward neural network to project the point cloud descriptor into the same representation domain as the anomaly projection descriptor.
[0080] For example, suppose the anomaly descriptor is a 1024-dimensional vector containing key visual features extracted from video data. The first perceptron network may contain three hidden layers, with 512, 256, and 128 neurons in each layer, respectively. Through layer-by-layer processing, the anomaly descriptor is projected into a 128-dimensional anomaly projection descriptor space. Meanwhile, the point cloud descriptor may be a two-dimensional array of shape [N, 3], where N is the number of points in the point cloud and 3 represents the x, y, and z coordinates of each point. The second perceptron network uses a special point cloud processing structure (such as PointNet), which first applies a shared multi-layer perceptron (MLP) to each point, and then aggregates the features of all points through a maximum pooling layer to finally obtain a global point cloud projection descriptor, which also has a dimension of 128.
[0081] After the processing of step S21, the anomaly descriptor and the point cloud descriptor are projected into the same representation domain, forming an anomaly projection descriptor and a point cloud projection descriptor respectively. The two projection descriptors are now in the same feature space and can be directly compared and integrated.
[0082] Representation domain alignment is a prerequisite for multimodal data fusion. In this embodiment, through the carefully designed first and second perceptron networks, the computer system ensures that the representations of the anomaly projection descriptor and the point cloud projection descriptor in the same feature space are comparable. This alignment requires not only the same dimension, but also the semantic consistency and metric consistency of the feature space. The projection descriptor not only retains the key information in the original descriptor, but also introduces new combined features through nonlinear transformation. These combined features may be more helpful for subsequent tasks (such as anomaly recognition) because they capture the intrinsic connections and interactions between different modal data.
[0083] After obtaining the abnormal projection descriptor and the point cloud projection descriptor, the computer system integrates them together to form a comprehensive first fusion descriptor. This fusion descriptor will contain key information in both video data and radar data, providing strong support for subsequent abnormal situation identification. Descriptor integration processing can adopt a variety of strategies, such as feature splicing, weighted summation, bilinear pooling, etc. In this embodiment, the specific integration strategy depends on the task requirements and data characteristics. However, no matter which strategy is adopted, the goal is to effectively combine the useful information in the abnormal projection descriptor and the point cloud projection descriptor. A simple integration strategy is feature splicing. That is, the abnormal projection descriptor and the point cloud projection descriptor are directly spliced into a longer vector. For example, if both are 128-dimensional vectors, then the first fusion descriptor after splicing is a 256-dimensional vector. This method retains all the original features, but may face the problem of dimensional disaster and increased computational complexity.
[0084] Another strategy is weighted summation. That is, different weights are assigned according to the importance of each projection descriptor, and then the weighted descriptors are added together to obtain the fused descriptor. The weight assignment can be obtained through learning (such as through the attention mechanism) or can be manually set based on prior knowledge.
[0085] Bilinear pooling is a more complex integration strategy that takes into account second-order interactions between different projection descriptors. By computing the outer product of two descriptors and applying a low-rank approximation (such as tensor decomposition), bilinear pooling is able to generate a compact and highly expressive fusion descriptor. For example, assume that feature splicing is used as an integration strategy in this scenario. Then, the computer system splices the 128-dimensional anomaly projection descriptor and the 128-dimensional point cloud projection descriptor into a 256-dimensional first fusion descriptor. This descriptor contains both visual features from video data (such as shape, texture, motion pattern, etc.) and three-dimensional spatial information from radar data (such as position, orientation, size, etc.). Through subsequent processing and analysis, the computer system can use this comprehensive information to more accurately identify abnormalities in the workshop.
[0086] Through the above steps S21~S23, the computer system integrates the anomaly descriptor and the point cloud descriptor in the same representation domain to generate the first fusion descriptor. This process not only involves the application of complex machine learning models (such as expert mixture models and perceptron networks), but also fully considers the inherent connection and interaction between multimodal data. The generation of the first fusion descriptor provides strong data support for subsequent abnormal situation identification tasks, enabling the computer system to more accurately monitor and analyze the operating status of the intelligent manufacturing workshop. This industrial equipment data processing method based on radar and video fusion computing not only improves the accuracy and efficiency of data processing, but also provides a new technical path for the intelligent management and maintenance of intelligent manufacturing.
[0087] As an implementation manner, the step S23, performing descriptor integration processing on the abnormal projection descriptor and the point cloud projection descriptor to obtain a first fusion descriptor, may specifically include:
[0088] Step S231: obtaining a first intensity effective value corresponding to the abnormal projection descriptor, and performing a normalization operation on the abnormal projection descriptor according to the first intensity effective value to obtain an abnormal normalized descriptor;
[0089] Step S232: performing nonlinear transformation on the abnormal normalized descriptor to obtain an abnormal activated descriptor, and performing mean calculation on the abnormal activated descriptor to obtain a first mean descriptor;
[0090] Step S233: obtaining a second intensity effective value corresponding to the point cloud projection descriptor, and performing a standardization operation on the point cloud projection descriptor according to the second intensity effective value to obtain a point cloud standardized descriptor;
[0091] Step S234: performing a nonlinear transformation on the point cloud normalization descriptor to obtain a point cloud activation descriptor, and performing mean calculation on the point cloud activation descriptor to obtain a second mean descriptor;
[0092] Step S235: summing the first mean descriptor and the second mean descriptor to obtain a mean sum descriptor, and performing an affine transformation operation on the mean sum descriptor according to a parameter matrix corresponding to the dense network in the expert mixture model to obtain the first fusion descriptor.
[0093] Step S23 is a part of step S20, which aims to effectively integrate the abnormal projection descriptor and the point cloud projection descriptor to generate a comprehensive first fusion descriptor. This process not only requires retaining the key information in the two descriptors, but also requires appropriate transformation and processing so that the fused descriptor can more accurately reflect the actual situation of the production line and provide strong support for subsequent abnormal situation identification.
[0094] In a scenario, such as on a car production line, the video data captured by a high-definition camera is processed to generate an abnormal projection descriptor, which contains the visual features of abnormal conditions that may exist in the production line. However, due to the diversity and complexity of video data, the feature values in the abnormal projection descriptor may have different dimensions and ranges, which will affect the effect of subsequent integration processing. In order to evaluate the strength or importance of the feature values in the abnormal projection descriptor, the computer system first calculates its first intensity effective value. This value can be the root mean square value (RMS), L2 norm, or other statistics that can reflect the strength or distribution characteristics of the feature. The root mean square value is a common choice because it can balance the impact of large and small values and give a comprehensive strength indicator.
[0095] For example, assuming that the abnormal projection descriptor is a 128-dimensional vector A, its root mean square value can be calculated by the formula RMS(A) = sqrt(sum(A.^2) / length(A)), where ^ represents element-wise multiplication, sum represents summation, sqrt represents square root, and length represents the vector length.
[0096] Standardization operation: After obtaining the first intensity effective value, the computer system standardizes the abnormal projection descriptor according to the value to eliminate the dimension difference and range inconsistency between different eigenvalues. Standardization usually includes subtracting the mean and dividing by the standard deviation, but here, a simpler maximum-minimum normalization method can be used to scale the eigenvalues to a uniform range (such as [0, 1]).
[0097] For example, assuming that the maximum value of the abnormal projection descriptor is maxA and the minimum value is minA, the normalized abnormal normalized descriptor A_norm can be calculated by the formula A_norm= (A-minA) / (maxA-minA). In this way, each element in A_norm will be located in the interval [0, 1]. After standardization, the feature values in the abnormal normalized descriptor are already in the same dimension and range. However, in order to further enhance its expressive power and discrimination, the computer system performs a nonlinear transformation on it in step S232, and calculates the mean of the transformed features to obtain a comprehensive first mean descriptor.
[0098] Nonlinear transformation is a commonly used technique in neural networks, which can enhance the expressive power of the model by introducing nonlinear factors. Here, the nonlinear transformation can be a simple activation function (such as ReLU, Sigmoid, or Tanh) or a more complex neural network layer (such as a convolutional layer, a recurrent layer, or a fully connected layer). However, considering the computational complexity and real-time requirements, a lightweight activation function may be a more appropriate choice. For example, suppose that the ReLU activation function is used to perform a nonlinear transformation on the abnormal normalization descriptor A_norm. The ReLU function is defined as f(x) = max(0, x), that is, for each element in the input A_norm, if it is greater than 0, it remains unchanged; otherwise, it is set to 0. In this way, the abnormal activation descriptor A_act is obtained.
[0099] After obtaining the abnormal activation descriptor, the computer system calculates its mean to obtain a single first mean descriptor. This mean descriptor not only contains the key information in the abnormal projection descriptor, but also introduces new feature combinations and interactions through nonlinear transformations. For example, assuming that the abnormal activation descriptor A_act is a 128-dimensional vector, the first mean descriptor mean_A can be calculated by the formula mean_A = mean(A_act), where mean represents the mean of the calculated vector.
[0100] Similar to the abnormal projection descriptor, the point cloud projection descriptor also needs to go through normalization, nonlinear transformation and mean calculation to obtain the second mean descriptor. The purpose of these steps is to ensure that the key information in the point cloud data can be integrated into the first fusion descriptor in an appropriate form.
[0101] Similar to step S231, the computer system first calculates the second intensity effective value (such as the root mean square value) of the point cloud projection descriptor, and performs standardization processing on the point cloud projection descriptor according to the value to obtain the point cloud standardized descriptor. Subsequently, the point cloud standardized descriptor is subjected to a nonlinear transformation (such as using a ReLU activation function) to obtain a point cloud activation descriptor, and its mean is calculated to obtain a second mean descriptor. These steps are similar to the operations in step S232, but the object of processing is point cloud data rather than video data.
[0102] In step S235, the first mean descriptor and the second mean descriptor are summed and an affine transformation operation is performed. After obtaining the first mean descriptor and the second mean descriptor, the computer system integrates them together to generate a first fusion descriptor. This process requires that the information in the two descriptors can be effectively combined and the key features thereof can be retained.
[0103] First, the computer system sums the first mean descriptor and the second mean descriptor to obtain a preliminary fused descriptor. This descriptor contains comprehensive information from the video data and the radar data, but may still be in a lower-level feature space. For example, assume that the first mean descriptor mean_A and the second mean descriptor mean_P are both scalars (in actual situations, they may be vectors, but here, for simplicity, they are assumed to be scalars). Then the preliminary fused descriptor fused_desc can be calculated by the formula fused_desc = mean_A + mean_P.
[0104] In order to map the preliminary fusion descriptor into a more advanced and expressive feature space, the computer system uses a dense network (i.e., fully connected layer) in the expert mixture model to perform an affine transformation operation on it. Affine transformation usually includes two parts: linear transformation (i.e., weighted summation) and nonlinear activation, but here, we may only focus on the linear transformation part because the nonlinear activation has been completed in the previous step.
[0105] For example, assuming that the parameter matrix corresponding to the dense network is W (a weight matrix) and b (a bias vector), the first fused descriptor first_fused_desc can be calculated by the formula first_fused_desc = W * fused_desc +b. Here, * represents matrix multiplication (if fused_desc is a vector, it is actually a weighted sum operation), and first_fused_desc is the final first fused descriptor.
[0106] It should be noted that in practical applications, fused_desc is likely to be a high-dimensional vector rather than a scalar. Therefore, the dimensions of the parameter matrix W and the bias vector b need to match fused_desc to ensure the correct execution of the affine transformation operation. In addition, in order to enhance the expressiveness and flexibility of the model, the dense network may contain multiple hidden layers, each of which performs a similar affine transformation operation and may be followed by a nonlinear activation function. However, here, in order to simplify the explanation, only a simple affine transformation layer is considered.
[0107] Through the above steps S231~S235, the computer system integrates the abnormal projection descriptor and the point cloud projection descriptor to generate the first fusion descriptor. This process not only involves basic data preprocessing techniques such as data standardization, nonlinear transformation and mean calculation, but also makes full use of the dense network in the expert mixture model for advanced feature extraction and fusion. The generation of the first fusion descriptor provides strong data support for subsequent abnormal situation identification tasks, enabling the computer system to more accurately monitor and analyze the operation status of the automobile production line. This industrial equipment data processing method based on radar and video fusion calculation not only improves the accuracy and efficiency of data processing, but also provides a new technical path for intelligent management and maintenance of intelligent manufacturing. In practical applications, the computer system can adjust the integration strategy and processing flow according to specific scenarios and needs to achieve optimal performance and effects.
[0108] As an implementation manner, in the step S20, determining the first integrated embedding network associated with the abnormal descriptor in p integrated embedding networks according to the first fusion descriptor includes:
[0109] Step S24: performing category mapping on the first fusion descriptor based on the category output network in the expert mixture model to obtain network scores corresponding to p integrated embedding networks;
[0110] Step S25: taking the integrated embedding network corresponding to the maximum network score as the first integrated embedding network associated with the anomaly descriptor.
[0111] In step S20, the computer system has completed the integration of the anomaly descriptor and the point cloud descriptor and generated a first fused descriptor. The core task of this step is to select the most suitable network, i.e., the first integrated embedding network, from p pre-trained integrated embedding networks based on this fused descriptor for subsequent embedding processing. The selection of the integrated embedding network is crucial to ensure the accuracy and efficiency of anomaly recognition.
[0112] In a practical application scenario, such as in a smart manufacturing factory, the first fused descriptor, as a result of multimodal data fusion, contains key information from video and radar. In order to determine which integrated embedding network is most suitable for processing this fused descriptor, the computer system uses the category output network (a trained classifier) in the expert mixture model to evaluate the processing capabilities of each network. The category output network is an important component in the expert mixture model. It is usually a multi-layer neural network, especially a convolutional neural network (CNN) or a fully connected network (FCN), which is used to classify the input data. In this scenario, the task of the category output network is not to divide the input data into predefined category labels, but to assign a network score to each integrated embedding network to reflect how suitable they are for processing the current fused descriptor.
[0113] The category output network extracts and transforms the first fused descriptor through a series of hidden layers, and finally outputs a p-dimensional vector, where each element corresponds to a network score for the integrated embedding network. These scores can be calculated based on the network's past processing performance on similar data, internal parameter configuration, or any other relevant indicators. Importantly, these scores should reflect the network's potential ability to handle the current task.
[0114] Assuming there are p=3 integrated embedding networks, the category output network may perform a series of complex nonlinear transformations and decision boundary calculations on the first fusion descriptor, and finally output a vector [score1, score2, score3], where score1, score2, and score3 represent the evaluation scores of the three networks for the processing capabilities of the current fusion descriptor. For example, taking a specific smart manufacturing factory as an example, assume that the category output network has been trained based on a large amount of historical data and has learned how to evaluate the processing capabilities of different integrated embedding networks based on the characteristics of the fusion descriptor. When the computer system receives a new first fusion descriptor, it inputs the descriptor into the category output network. After a series of processing and transformations, the network outputs a score vector [0.7, 0.85, 0.6]. This vector indicates that under the current fusion descriptor, the second integrated embedding network (score 0.85) is considered to be the most suitable processing network.
[0115] After obtaining the network scores of the p integrated embedding networks, the computer system determines which network has the highest score and uses the corresponding integrated embedding network as the first integrated embedding network. This selection process is based on a direct comparison of the scores, aiming to select the network with the strongest processing capability for the current fusion descriptor. The computer system traverses the score vector output by the category output network and compares the value of each element. This process is a simple linear scan and does not require complex algorithms or additional computing resources. After finding the maximum network score, the computer system uses the integrated embedding network corresponding to the score as the first integrated embedding network. This selection is deterministic, that is, the network with the highest score is always selected, regardless of the scores of other networks. Once the first integrated embedding network is determined, the computer system can use the network in subsequent embedding processing steps. The first integrated embedding network will generate higher-level feature representations based on the anomaly descriptor and the point cloud descriptor (or their fusion form), which will be used for the final anomaly recognition.
[0116] For example, continuing the previous example, the computer system traverses the score vector [0.7, 0.85, 0.6] and finds that the second element (0.85) is the largest. Therefore, it selects the second integrated embedding network as the first integrated embedding network. In the next processing step, the computer system will use this network to further embed the anomaly descriptor and the point cloud descriptor (or their first fusion descriptor) to generate a more accurate feature representation and support the subsequent anomaly recognition task.
[0117] Through the above steps S24 and S25, the computer system successfully selects the most suitable first integrated embedding network from p integrated embedding networks according to the first fusion descriptor. This process not only relies on the accurate evaluation of the fusion descriptor by the category output network in the expert mixture model, but also reflects the powerful application capabilities of multimodal data fusion and machine learning in complex industrial scenarios. The selection of the first integrated embedding network lays a solid foundation for subsequent anomaly recognition tasks, enabling the computer system to more accurately monitor and analyze the production status in the intelligent manufacturing factory, and promptly detect and handle abnormal situations. This industrial equipment data processing method based on radar and video fusion computing not only improves the automation level and intelligence of the production line, but also provides strong technical support for the advent of the Industrial 4.0 era.
[0118] As a first implementation manner, the first integrated embedding network includes a weighted focusing integrated network; based on this, the step S30, in the first integrated embedding network, performs an integrated embedding operation on the abnormal descriptor according to the point cloud descriptor to obtain a first embedded descriptor, which may specifically include:
[0119] Step SA31: combining the anomaly descriptor and the point cloud descriptor to obtain a first combined descriptor;
[0120] Step SA32: performing a self-weighted focusing operation on the first combined descriptor based on the weighted focusing integration network to obtain a first internal focusing descriptor;
[0121] Step SA33: Use the first q rows of descriptors in the first internal focus descriptor as the first embedded descriptor; q is the number of rows of the abnormal descriptor.
[0122] In a real-world scenario, such as in a port automation environment, the video data captured by a high-definition camera is processed to generate an anomaly descriptor, which contains the visual features of possible anomalies in the port. At the same time, the point cloud data collected by the radar sensor is also converted into a point cloud descriptor, which provides the three-dimensional spatial information and precise location of the cargo. In order to effectively combine the data of these two modalities, the computer system first needs to combine them into a unified descriptor, namely the first combined descriptor.
[0123] Before combining, the computer system preprocesses the anomaly descriptor and the point cloud descriptor to ensure their consistency in dimension, dimension, and feature space. For example, if the anomaly descriptor is a high-dimensional vector and the point cloud descriptor is a two-dimensional array (point cloud matrix), it may be necessary to flatten the point cloud matrix into a long vector or map both into the same feature space through some transformation. However, in this embodiment, it is assumed that the anomaly descriptor and the point cloud descriptor have been preprocessed in some way into a format that can be directly spliced.
[0124] The computer system concatenates the anomaly descriptor and the point cloud descriptor along a certain dimension (usually the feature dimension). This concatenation operation is direct and linear, and does not involve complex transformations or calculations. The result of the concatenation is a new, higher-dimensional vector or array, namely the first combined descriptor. For example, suppose the anomaly descriptor is a 128-dimensional vector A, which represents the anomaly features extracted from the video data; the point cloud descriptor is a two-dimensional array P of shape [N, 3], where N is the number of points in the point cloud and 3 represents the x, y, and z coordinates of each point. In order to concatenate the two descriptors, the computer system may first flatten the point cloud descriptor into a 3N-dimensional vector P_flat. Then, A and P_flat are concatenated along the feature dimension to obtain a (128 + 3N)-dimensional first combined descriptor C = [A; P_flat] (here; represents the vertical concatenation of vectors). However, in practical applications, since point cloud data usually contains a large number of points, direct flattening and concatenation can lead to dimensionality explosion. Therefore, a more common approach is to first reduce the dimensionality of the point cloud data (such as through PCA, t-SNE, etc.) to obtain a low-dimensional point cloud embedding descriptor, and then concatenate it with the abnormal descriptor. However, here, in order to simplify the explanation, it is still assumed that the flattened point cloud descriptor is directly used for concatenation.
[0125] After obtaining the first combined descriptor, the computer system further processes it using a weighted focusing integration network (an attention fusion network) to highlight the features most relevant to the abnormal situation and suppress insignificant details. This process is achieved through a self-weighted focusing operation, that is, the network automatically learns the importance weights of each feature and performs a weighted sum of the features based on these weights. A weighted focusing integration network is a special neural network architecture that can dynamically adjust the importance weights of different features when processing input data. This network usually contains multiple attention layers, each of which can calculate the similarity or correlation between input features and generate a weight matrix based on this. Each element in the weight matrix represents the importance of a feature relative to other features. In the weighted focusing integration network, the self-weighted focusing operation refers to the process in which the network automatically calculates the importance weights of the input features through an internal mechanism (such as a self-attention mechanism) and performs a weighted sum of the features. This process does not rely on external labels or supervisory signals, but is completely learned by the network based on the intrinsic characteristics of the input data. In step SA32, the computer system inputs the first combined descriptor into the weighted focusing integration network. The network first extracts and transforms the input data through a series of transformation layers (such as fully connected layers, convolutional layers, etc.), and then applies the self-attention mechanism to calculate the similarity or correlation matrix between the features. Next, the network generates a weight matrix based on the similarity matrix and applies the weight matrix to the input features for weighted summation. The final output is a first internal focused descriptor that has been processed by self-weighted focusing, in which each feature is weighted according to its importance to the abnormal situation. For example, assume that the weighted focusing integration network contains a self-attention layer that can calculate the dot product similarity between the input features and generate a weight matrix based on it. When the first combined descriptor C is input into the network, the self-attention layer first calculates the dot product similarity between all feature pairs in C to obtain a similarity matrix S. Then, the network performs softmax normalization on the similarity matrix to obtain a weight matrix W, in which each element w_ij represents the importance weight of the i-th feature relative to the j-th feature. Finally, the network applies the weight matrix W to the input feature C and performs a weighted sum operation to obtain the first internal focus descriptor F_int = WC (the bias term and activation function are ignored here to simplify the explanation). It should be noted that in actual applications, the weighted sum operation may be more complicated, involving multi-dimensional matrix operations and the application of activation functions.
[0126] After the self-weighted focusing operation, the computer system obtains a first internal focused descriptor in which each feature has been assigned an appropriate importance weight. However, since the weighted focused integration network can output a higher-dimensional descriptor than the original input (especially when the network contains multiple hidden layers), the computer system extracts a fixed-length first embedded descriptor from it for subsequent abnormal situation recognition tasks.
[0127] In order to maintain consistency with the original anomaly descriptor (especially when it may be necessary to concatenate with the anomaly classification label or other descriptors in subsequent processing steps), the computer system selects a portion of the first internal focused descriptor with the same dimension as the original anomaly descriptor as the first embedded descriptor. The first q rows of the descriptor are selected here, where q is the number of rows of the anomaly descriptor.
[0128] By selecting the first q rows in the first internal focused descriptor as the first embedded descriptor, the computer system is actually retaining the features that are most important for anomaly recognition. These features may have been transformed and weighted many times by the network, but their relative positions and order of importance are maintained to a certain extent. For example, suppose the first internal focused descriptor F_int is a matrix of dimension (M, d) (where M may be much larger than q, and d is the depth or number of channels of the feature), and the original anomaly descriptor A is a matrix of dimension (q, d). To obtain the first embedded descriptor, the computer system simply extracts the first q rows from F_int to form a new matrix F_emb = F_int(1:q, :). In this way, F_emb has the same dimension as the original anomaly descriptor A and contains the most important features after the self-weighted focusing process. These features will serve as the basic input data for subsequent anomaly recognition tasks.
[0129] Through the above steps SA31~SA33, the computer system realizes the integrated embedding operation of the anomaly descriptor and the point cloud descriptor in the weighted focusing integration network, and generates the first embedded descriptor. This process not only makes full use of the complementary advantages of multimodal data, but also highlights the feature information most relevant to the abnormal situation through the self-weighted focusing operation. The generation of the first embedded descriptor provides strong support for the subsequent abnormal situation identification tasks, enabling the computer system to more accurately monitor and analyze various abnormal situations in the port automation environment. This industrial equipment data processing method based on radar and video fusion calculation not only improves the automation and intelligence level of port operations, but also provides a useful reference for multimodal data fusion processing in other similar scenarios.
[0130] As an implementation manner, the step SA32, performing a self-weighted focusing operation on the first combined descriptor based on the weighted focusing integration network to obtain a first internal focusing descriptor, may specifically include:
[0131] Step SA321: converting the first combined descriptor into a first search array, a first anchor array and a first result array according to the conversion parameter matrix corresponding to the weighted focusing integration network;
[0132] Step SA322: generating a first candidate array according to the multiplication result between the first search array and the abnormal descriptor, and the multiplication result between the first search array and the point cloud descriptor;
[0133] Step SA323: generating a second candidate array according to the flip array of the multiplication result between the first anchor array and the abnormal descriptor, and the flip array of the multiplication result between the first anchor array and the point cloud descriptor;
[0134] Step SA324: generating a third candidate array according to the multiplication result between the first result array and the abnormal descriptor, and the multiplication result between the first result array and the point cloud descriptor;
[0135] Step SA325: taking the multiplication result of the first candidate array and the second candidate array as the fourth candidate array, and obtaining the number of columns of the abnormal descriptor;
[0136] Step SA326: Standardize the ratio between the fourth candidate array and the square root of the number of columns to obtain an internal focus influence coefficient array, and use the multiplication result between the internal focus influence coefficient array and the third candidate array as the first internal focus descriptor.
[0137] Step SA32 involves using a weighted focus integration network to perform a self-weighted focus operation on the first combined descriptor (composed of the anomaly descriptor and the point cloud descriptor). This process simulates the attention mechanism in the human visual system to automatically learn and adjust the importance weights of different features, thereby highlighting the feature information most relevant to the anomaly. The resulting first internally focused descriptor will serve as input data for subsequent processing steps to support more accurate anomaly identification.
[0138] For example, in a specific implementation scenario, such as an automated manufacturing plant, the first combined descriptor, as the result of multimodal data fusion, contains key information from video and radar. In order to perform self-weighted focusing operations, the computer system first needs to convert this information into a format suitable for attention mechanism processing, namely, search array (query), anchor array (key) and result array (value).
[0139] The weighted focused integration network includes a series of pre-trained transformation parameter matrices, which are used to map input data to different feature spaces or dimensions. In step SA321, the computer system uses these transformation parameter matrices to perform a linear transformation on the first combined descriptor to generate a search array, an anchor array, and a result array.
[0140] For example, suppose the first combined descriptor is a matrix C of dimension (N, D), where N is the number of data points (i.e., the number of feature vectors) and D is the dimension of the feature. The weighted focused integration network contains three transformation parameter matrices W_q, W_k, and W_v, which are used to generate the search array Q, the anchor array K, and the result array V, respectively. The transformation process can be expressed as:
[0141] Q=C*W_q (generates the search array); K=C*W_k (generates the anchor array); V=C*W_v (generates the result array), where * represents the matrix multiplication operation.
[0142] For example, suppose the first combined descriptor C is a two-dimensional array of shape [100, 256] (indicating that there are 100 data points and each data point has 256 features). The transformation parameter matrices W_q, W_k, and W_v in the weighted focused integration network are all two-dimensional arrays of shape [256, 128] (indicating that the dimension of the original features is reduced from 256 to 128). After the transformation, the search array Q, anchor array K, and result array V will all be two-dimensional arrays of shape [100, 128].
[0143] After obtaining the search array, anchor array and result array, the computer system calculates the similarity or correlation between them to generate candidate arrays for subsequent processing. The generation of the first candidate array involves the interaction between the search array and the anomaly descriptor and the point cloud descriptor.
[0144] In the self-attention mechanism, similarity is usually calculated by dot product operation. However, since the anomaly descriptor and point cloud descriptor may not exist in the same format or dimension originally, it is assumed here that the computer system has converted them into a format compatible with the search array, anchor array and result array during internal processing.
[0145] Assume that the computer system extracts the abnormal descriptor part and the point cloud descriptor part in the first combined descriptor respectively, and performs a dot product operation with the search array Q.
[0146] In other implementations, the search array Q and the anchor array K may be directly used to perform a dot product operation to calculate a similarity matrix therebetween.
[0147] The generation of the second candidate array involves the interaction between the anchor array and the anomaly descriptors and point cloud descriptors, but unlike the first candidate array, the transpose of the anchor array (i.e., flipped array) is used here. In matrix operations, the transpose operation refers to swapping the rows and columns of a matrix. Here, the transposition of the anchor array K is to change the order of the dimensions of the matrix in the subsequent dot product operation so that it can be compatible with the anomaly descriptor or point cloud descriptor (represented in some way).
[0148] As a second implementation manner, the first integrated embedding network includes a weighted focused integrated network; based on this, the step S30, in the first integrated embedding network, performs an integrated embedding operation on the abnormal descriptor according to the point cloud descriptor to obtain a first embedded descriptor, which may specifically include:
[0149] Step SB31: Based on the weighted focusing integration network, the anomaly descriptor is converted into a second search array, and the point cloud descriptor is converted into a second anchor array and a second result array;
[0150] Step SB32: multiplying the multiplication result between the second search array and the abnormal descriptor and the flip array of the multiplication result between the second anchor array and the point cloud descriptor to obtain a fifth candidate array, and obtaining the number of columns of the abnormal descriptor;
[0151] Step SB33: normalizing the ratio between the fifth candidate array and the square root of the number of columns to obtain a cross-focus influence coefficient array, and multiplying the cross-focus influence coefficient array and the multiplication result between the second result array and the point cloud descriptor to obtain a cross-focus descriptor;
[0152] Step SB34: Use the first q rows of descriptors in the cross-focus descriptor as the first embedded descriptor, where q is the number of rows of the exception descriptor.
[0153] In a real-world scenario example, consider a complex automated warehousing system equipped with high-definition cameras and radar sensors to monitor the handling, storage, and robotic operations of goods in the warehouse in real time. The computer system is responsible for fusing and processing data from video and radar to detect potential anomalies, such as misplaced goods, robot collisions, or blocked paths. In this scenario, the anomaly descriptor is extracted from the video data and contains visual features of possible anomalies in the warehouse, while the point cloud descriptor is extracted from the radar data and provides the three-dimensional spatial information and precise location of the goods.
[0154] In step SB31, the computer system uses a specific conversion parameter matrix in the weighted focusing integration network to convert the anomaly descriptor and the point cloud descriptor into a format suitable for subsequent processing. Specifically, the anomaly descriptor is converted into a second search array (query), while the point cloud descriptor is converted into a second anchor array (key) and a second result array (value). This conversion process aims to map data of different modalities to the same feature space or dimension for cross-modal attention calculation.
[0155] For example, assume that the anomaly descriptor is a matrix A of shape [m, d1] (m is the number of anomaly descriptors, d1 is the feature dimension), and the point cloud descriptor is a matrix P of shape [n, d2] (n is the number of points in the point cloud, d2 is the feature dimension. To simplify the explanation, it is assumed that the point cloud descriptor has undergone some form of dimensionality reduction so that it can interact directly with the anomaly descriptor). The weighted focused integration network contains three transformation parameter matrices W_q, W_k, and W_v, whose shapes are [d1, e], [d2, e], and [d2, f], respectively, where e is the shared dimension after transformation (for the search array and anchor array), and f is the dimension of the result array. The transformation process can be expressed as:
[0156] The second search array Q=A*W_q, the shape becomes [m,e]. The second anchor array K=P*W_k, the shape becomes [n,e]. The second result array V=P*W_v, the shape becomes [n,f].
[0157] In this way, the computer system obtains the second search array Q, the second anchor array K and the second result array V for subsequent attention calculations.
[0158] In step SB32, the computer system first calculates the similarity between the second search array Q and the anomaly descriptor A. The similarity between the second search array Q and the second anchor array K is calculated, which represents the cross-modal interaction between the anomaly descriptor and the point cloud descriptor. By performing matrix multiplication on the transpose of Q and K and applying the softmax function for normalization, the computer system obtains an array of cross-focus influence coefficients (i.e., a cross-modal attention weight matrix).
[0159] At the same time, in order to maintain consistency with the anomaly descriptor, the computer system also obtains the number of columns of the anomaly descriptor (i.e., the feature dimension d1), but this information may not be used directly in subsequent standardization operations, but is used to understand the data structure and dimensional relationship. Assume that the similarity matrix S between Q and the transpose K^T of K is directly calculated, where S[i, j] represents the similarity between the i-th anomaly descriptor feature and the j-th point cloud descriptor feature. Then, the softmax function is applied to each row of the similarity matrix S to obtain the normalized cross-focus influence coefficient array W_att, whose shape is [m, n].
[0160] It should be noted here that since the softmax function normalizes each row, the sum of each row element of W_att is 1, which represents the importance weight of each abnormal descriptor feature relative to all point cloud descriptor features. Next, the computer system performs a matrix multiplication operation on the cross-focus influence coefficient array W_att and the second result array V to calculate the weighted result representation. But before that, since the dimension of V is related to K (that is, the converted form of the point cloud descriptor), it needs to be kept consistent with the number of abnormal descriptors (m) for subsequent processing, so in actual operation, some form of aggregation or transformation of V (such as average pooling, maximum pooling or weighted summation, etc.) may be involved.
[0161] The calculation of the cross-focus descriptor can be expressed as: F_cross = W_att * V (note that the multiplication operation here may need to be adjusted according to the actual data shape and dimension). However, in a strict sense, since the shape of W_att is [m, n] and the shape of V is [n, f], direct multiplication will result in a matrix of shape [m, f], which is itself a weighted result representation and can be regarded as a cross-focus descriptor. But in subsequent steps, it may be necessary to extract the part directly related to the abnormal descriptor from this result.
[0162] The cross-focus influence coefficient array W_att calculated in step SB32 has actually been normalized by the softmax function, so no additional normalization operation is required. The multiplication result of W_att and the second result array V generates the cross-focus descriptor F_cross, which contains the feature information of the abnormal descriptor after weighting by the point cloud descriptor. These weighted features not only reflect the importance of the abnormal descriptor itself, but also incorporate the three-dimensional spatial information and precise location information provided by the point cloud descriptor, thereby realizing cross-modal feature fusion.
[0163] In step SB34, the computer system extracts the part directly related to the abnormal descriptor from the cross-focus descriptor F_cross as the first embedded descriptor. Since the shape of the cross-focus descriptor may not be completely consistent with the original shape of the abnormal descriptor (especially when multiple points of the point cloud descriptor are involved), the computer system may need to extract the first q rows of descriptors through some form of indexing, slicing or aggregation operation, where q is the number of rows of the abnormal descriptor. In actual situations, if the shape of the cross-focus descriptor F_cross is already [m, d] (where m is the number of abnormal descriptors and d is a certain feature dimension) and is consistent with the original number of abnormal descriptors, then the first q rows (actually all rows here, because m=q) can be directly extracted as the first embedded descriptor. Alternatively, the computer system may need to perform further processing on F_cross (such as dimensionality reduction, feature selection or transformation) to generate the final first embedded descriptor.
[0164] In an embodiment of the present application, it is assumed that the cross-focus descriptor F_cross has been processed in some way into a shape that is completely consistent with the number of abnormal descriptors, and it can be directly regarded as the first embedded descriptor. For example, assume that the shape of the cross-focus descriptor F_cross is [m, d], where m is the number of abnormal descriptors, and d is the feature dimension after weighting and fusion. Since in this embodiment, special attention is paid to the cross-modal interaction between the abnormal descriptor and the point cloud descriptor, and the ultimate goal is to generate a first embedded descriptor that is consistent with the number of abnormal descriptors, F_cross can be directly regarded as the first embedded descriptor without additional extraction or processing steps (unless there are specific requirements or constraints).
[0165] As a third implementation manner, the first integrated embedding network includes a deep weighted focused integrated network; based on this, the step S30, in the first integrated embedding network, performs an integrated embedding operation on the abnormal descriptor according to the point cloud descriptor to obtain a first embedded descriptor, which may specifically include:
[0166] Step SC31: obtaining a deep descriptor, combining the deep descriptor with the point cloud descriptor to obtain a second combined descriptor, wherein the deep descriptor is used to promote the connection of descriptors of different categories;
[0167] Step SC32: Based on the first weighted focusing layer in the deep weighted focusing integration network, the second combined descriptor is subjected to a self-weighted focusing operation to obtain a second internally focused descriptor, and the first e rows of descriptors in the second internally focused descriptor are used as candidate embedded descriptors; e is the number of rows of the deep descriptor;
[0168] Step SC33: combining the abnormal descriptor and the candidate embedded descriptor to obtain a third combined descriptor, and performing a self-weight focusing operation on the third combined descriptor based on the second weight focusing layer in the deep weight focusing integration network to obtain a third internal focusing descriptor;
[0169] Step SC34: Use the first q rows of descriptors in the third internal focus descriptor as the first embedded descriptor, where q is the number of rows of the abnormal descriptor.
[0170] In step SC31, the computer system first obtains a deep descriptor, which can be regarded as a bottleneck feature. It has been extracted in the previous processing step to promote the connection between descriptors of different categories (i.e., cross-modal descriptors). The deep descriptor may be learned from the original data through a pre-trained deep learning model (such as a convolutional neural network CNN or an autoencoder), and it contains highly abstract and generalized feature information. Then, the computer system combines the deep descriptor with the point cloud descriptor to generate a second combined descriptor. This combination process may involve feature concatenation, feature fusion, or more complex feature transformation operations, depending on the data structure and dimension of the deep descriptor and the point cloud descriptor. The combined second combined descriptor not only contains the three-dimensional spatial information in the point cloud descriptor, but also incorporates the cross-modal feature connection in the deep descriptor, providing a rich information basis for the subsequent self-weighted focusing operation.
[0171] For example, suppose the deep descriptor is a matrix D of shape [e, d1] (e is the number of deep descriptors, d1 is the feature dimension), and the point cloud descriptor is also a matrix after some form of dimensionality reduction, but for the sake of simplicity, it is assumed that it is already a shape compatible with the deep descriptor, or has been transformed into the same feature space as the deep descriptor in some way. In this way, the deep descriptor D can be directly concatenated with the point cloud descriptor P_reduced (assuming its shape is [e, d2], but note that in practice, since point cloud data usually contains a large number of points, dimensionality reduction is necessary) to generate the second combined descriptor C2 = [D; P_reduced] (here; represents the vertical concatenation of the matrix, but note that in actual operation, the dimensions may need to be adjusted to ensure the feasibility of concatenation). However, more commonly, the deep descriptor and the point cloud descriptor may have different feature dimensions and numbers. In this case, it may be necessary to map them to the same feature space through additional transformation layers (such as fully connected layers or convolutional layers) and then concatenate them.
[0172] In step SC32, the computer system uses the first weight focusing layer (i.e., attention layer) in the deep weight focusing integration network to perform self-weight focusing operation on the second combined descriptor. This layer dynamically adjusts the importance weights of different features by simulating the attention mechanism, so that the features most relevant to the current task can receive more attention.
[0173] Specifically, the first weighted focus layer may contain multiple self-attention modules, each of which can calculate the similarity or correlation between input features and generate a weight matrix based on it. The weight matrix is then applied to the input features for weighted summation to generate a weighted feature representation. This process is iterative until a predetermined number of layers is reached or a certain convergence condition is met.
[0174] Finally, the first weighted focusing layer outputs a second internal focused descriptor, in which the features have been weighted according to their importance. To be consistent with the subsequent steps, the computer system selects the first e rows of descriptors in the second internal focused descriptor as candidate embedding descriptors. The first e rows are selected here because the number of candidate embedding descriptors needs to be the same as that of deep descriptors.
[0175] Assume that the first weighted focusing layer contains a simple self-attention module that can calculate the similarity matrix between all feature pairs in the second combined descriptor C2 and generate a weight matrix based on it. Then, the weight matrix is applied to C2 for weighted summation to generate the second internal focused descriptor F_int2. Since the specific implementation details of the self-attention mechanism are relatively complex and are not the focus of this paragraph, the specific calculation process is omitted here.
[0176] After the first weighted focusing layer, the features in F_int2 have been reordered and weighted according to their importance for the current task. In order to combine with the abnormal descriptors in the subsequent steps, the computer system selects the first e rows of descriptors in F_int2 as the candidate embedding descriptors E_cand. Here e is the number of deep descriptors, which ensures the consistency of the number of candidate embedding descriptors and deep descriptors.
[0177] In step SC33, the computer system combines the abnormal descriptor with the candidate embedding descriptor to generate a third combined descriptor. This combination process is similar to the combination process in step SC31, but this time it involves the abnormal descriptor and the candidate embedding descriptor after the first self-weighted focusing operation. The combined third combined descriptor not only contains the visual features in the abnormal descriptor, but also incorporates the weighted cross-modal feature information (i.e., the candidate embedding descriptor).
[0178] Next, the computer system uses the second weight focusing layer in the deep weight focusing integration network to perform a second self-weight focusing operation on the third combined descriptor. The operating principle of this layer is the same as that of the first weight focusing layer, which dynamically adjusts the importance weights of features by simulating the attention mechanism. The difference is that the second weight focusing layer may have different parameter settings, number of layers, or attention module configurations to adapt to changes in the current task and data characteristics. Finally, the second weight focusing layer outputs a third internally focused descriptor, in which the features have been further weighted and optimized according to their importance.
[0179] For example, suppose the anomaly descriptor is a matrix A of shape [q, d3] (q is the number of anomaly descriptors, d3 is the feature dimension), and the shape of the alternative embedding descriptor E_cand is [e, d4] (note that in practice, d3 and d4 may not be the same, but for simplicity of explanation, it is assumed that they have been adjusted to the same dimension or feature space in some way). The computer system concatenates A and E_cand to generate a third combined descriptor C3 = [A; E_cand].
[0180] Then, the second weighted focusing layer performs a second self-weighted focusing operation on C3. Similar to the first weighted focusing layer, this layer optimizes the feature representation by calculating the similarity matrix, generating the weight matrix, and performing a weighted sum operation. The final output third internal focused descriptor F_int3 contains the feature information after two weighted adjustments. These features not only reflect the key information in the anomaly descriptor and the point cloud descriptor, but also achieve deep fusion and optimization of cross-modal features through the bottleneck attention mechanism of the deep weighted focusing integration network.
[0181] In step SC34, the computer system extracts the portion directly related to the abnormal descriptor from the third internal focused descriptor as the first embedded descriptor. Since the shape of the third internal focused descriptor may not be completely consistent with the original shape of the abnormal descriptor (especially when it comes to the fusion and weighted adjustment of cross-modal features), the computer system extracts the first q rows of descriptors through some form of indexing, slicing or aggregation operation, where q is the number of rows of the abnormal descriptor.
[0182] The extracted first embedding descriptor not only retains the key visual features in the anomaly descriptor, but also incorporates the 3D spatial information provided by the point cloud descriptor and the cross-modal feature connections promoted by the deep descriptor. This comprehensive information will serve as input data for subsequent processing steps (such as anomaly recognition, classification, or decision support), supporting more accurate and efficient task execution.
[0183] For example, assume that the shape of the third internal focus descriptor F_int3 is [m, d5] (where m may be greater than q because the number of features may increase after two self-weighted focus operations; d5 is the feature dimension after fusion and weighted adjustment). In order to be consistent with the abnormal descriptor, the computer system selects the first q rows of descriptors in F_int3 as the first embedded descriptor E_first. In this way, E_first contains the same number of feature representations as the abnormal descriptor, which have been optimized and fused through the bottleneck attention mechanism of the deep weighted focus integration network, providing strong support for subsequent abnormal situation recognition tasks.
[0184] Through the detailed explanation and example illustration of the above steps, we can see the strong application potential of the deep weighted focused integration network in industrial equipment data processing scenarios based on radar and video fusion calculations. By introducing deep descriptors as a bridge to connect cross-modal features and using two layers of weighted focusing operations to gradually refine feature representations, the computer system is able to generate a more accurate and comprehensive first embedding descriptor. This first embedding descriptor not only contains the key information in the anomaly descriptor and point cloud descriptor, but also realizes the deep fusion and optimization of cross-modal features through the bottleneck attention mechanism of the deep weighted focused integration network, providing strong data support for subsequent abnormal situation recognition and other advanced processing tasks.
[0185] As a fourth implementation manner, the first integrated embedding network includes a point cloud core integration layer; based on this, the step S30, in the first integrated embedding network, performs an integrated embedding operation on the abnormal descriptor according to the point cloud descriptor to obtain a first embedded descriptor, which may specifically include:
[0186] Step SD31: obtaining an abnormal standardized descriptor corresponding to the abnormal descriptor and a point cloud standardized descriptor corresponding to the point cloud descriptor;
[0187] Step SD32: Based on the first coupled oscillation network in the point cloud core integration layer, performing a coupled oscillation operation on the abnormal descriptor to obtain a first coupled oscillation descriptor;
[0188] Step SD33: Based on the second coupled oscillation network in the point cloud core integration layer, performing a coupled oscillation operation on the point cloud descriptor to obtain a second coupled oscillation descriptor;
[0189] Step SD34: perform mean calculation on the second coupled oscillation descriptor to obtain a third mean descriptor, and sum the first coupled oscillation descriptor and the third mean descriptor to obtain a first embedded descriptor.
[0190] In step SD31, the computer system first standardizes the anomaly descriptors and point cloud descriptors to eliminate the dimensional differences and range inconsistencies that may exist between different data sources. Standardization processing usually includes operations such as data normalization, centering or scaling, which aims to convert the original data into a unified format or range for subsequent processing. For the anomaly descriptor, the computer system may standardize by calculating its mean and standard deviation, and then subtracting the mean from each eigenvalue and dividing it by the standard deviation. The anomaly descriptor processed in this way (i.e., the anomaly standardized descriptor) will have zero mean and unit variance.
[0191] For point cloud descriptors, since they usually contain a large number of three-dimensional point coordinates, it may not be practical to directly perform overall normalization. Instead, the computer system can first reduce the dimensionality of the point cloud data (such as through PCA, t-SNE, etc.) to obtain a low-dimensional point cloud embedding representation, and then normalize this embedding representation. Alternatively, if the point cloud descriptor has been aggregated or summarized in some form (such as through voxelization, point cloud feature extraction, etc.), these aggregated features can be directly standardized.
[0192] Regardless of the method used, the ultimate goal is to obtain anomaly normalized descriptors and point cloud normalized descriptors, which will serve as input data for subsequent coupled oscillation operations.
[0193] For example, suppose the anomaly descriptor is a matrix A of shape [q, d] (q is the number of samples, d is the feature dimension), and after normalization, the anomaly normalized descriptor A_norm is obtained. After dimension reduction and aggregation, the point cloud descriptor obtains a matrix P_reduced of shape [p, e] (p is the number of point clouds or regions after aggregation, e is the feature dimension after dimension reduction), and then after normalization, the point cloud normalized descriptor P_norm is obtained.
[0194] In step SD32, the computer system uses the first coupled oscillation network (such as PCNN) in the point cloud core integration layer to perform coupled oscillation operations on the abnormal normalized descriptor. PCNN is a neural network model that simulates the pulse emission mechanism of biological visual cortical neurons. It can automatically adjust the activation threshold and pulse emission time of neurons according to the characteristics of input data, thereby realizing nonlinear transformation and feature extraction.
[0195] In PCNN, each neuron has a coupling connection with its neighboring neurons. When a neuron receives enough input stimulation, it will emit a pulse and activate its neighboring neurons through the coupling connection. This pulse emission process will continue to iterate until a certain stable state is reached or a certain termination condition is met. For the abnormal normalization descriptor A_norm, the computer system can regard it as the state value of the input layer neurons of PCNN. Then, by setting appropriate PCNN parameters (such as connection weights, attenuation coefficients, threshold adjustment functions, etc.), the PCNN network is allowed to iterate until the states of all neurons tend to stabilize. At this point, the state value of the output layer neurons constitutes the first coupled oscillation descriptor F_osc1.
[0196] Assume that the PCNN network has determined the appropriate parameter set through pre-training, and these parameters are effective for the current task. When the abnormal normalization descriptor A_norm is sent to the PCNN network as input, the neurons in the network will perform pulse emission and coupled oscillation operations according to the characteristics of the input data. Finally, when the network reaches a stable state, the state value of the output layer neuron constitutes the first coupled oscillation descriptor F_osc1. This descriptor not only contains the key feature information in the abnormal descriptor, but also introduces new feature combinations and interactions through the nonlinear transformation and feature extraction mechanism of PCNN.
[0197] Step SD33 is similar to step SD32, but the object of processing is the point cloud normalization descriptor P_norm. The computer system uses the second coupled oscillating network (which can also be a PCNN or other type of coupled oscillating network) in the point cloud core integration layer to perform coupled oscillation operations on P_norm. This second coupled oscillating network may have the same structure or parameter settings as the first coupled oscillating network, or it may be adjusted according to different task requirements and data characteristics.
[0198] After the coupled oscillation operation, the point cloud normalized descriptor is converted to the second coupled oscillation descriptor F_osc2. This descriptor also contains the key feature information in the point cloud descriptor, and introduces new feature combinations and interactions through nonlinear transformation. Unlike the first coupled oscillation descriptor, the second coupled oscillation descriptor focuses more on reflecting the three-dimensional spatial structure and shape characteristics of the point cloud data.
[0199] When the point cloud normalization descriptor P_norm is sent as input to the second coupled oscillator network, the neurons in the network will also perform pulse emission and coupled oscillation operations. The final output second coupled oscillator descriptor F_osc2 contains the point cloud feature information after nonlinear transformation and feature extraction.
[0200] In step SD34, the computer system first performs mean calculation on the second coupled oscillation descriptor F_osc2 to obtain a third mean descriptor M_mean. Mean calculation is a simple data aggregation method that obtains a comprehensive feature representation by averaging all feature values. This mean descriptor reflects the average level or central trend of all features in the second coupled oscillation descriptor.
[0201] Then, the computer system performs a sum operation on the first coupled oscillation descriptor F_osc1 and the third mean descriptor M_mean to obtain the first embedded descriptor E_first. The sum operation is a direct linear combination method that adds the eigenvalues of the two descriptors to generate a new descriptor. This new descriptor not only contains the key feature information in the anomaly descriptor and the point cloud descriptor, but also introduces new feature combinations and interactions through coupled oscillation operations and mean calculations, thereby achieving effective integration and embedding of cross-modal data. For example, assume that the first coupled oscillation descriptor F_osc1 is a matrix with a shape of [q, f1] (where f1 is the feature dimension after PCNN transformation), and the third mean descriptor M_mean is a vector with a shape of [1, f2] (note that it is assumed here that the mean calculation results in a vector form compatible with the feature dimension of F_osc1; in practice, it may be necessary to expand the mean descriptor to the same shape as F_osc1 or perform other forms of transformation in some way). The computer system then broadcasts or replicates M_mean along a certain dimension (such as the sample dimension) to make its shape [q, f2] so that it can be summed with F_osc1. However, to simplify the explanation and avoid introducing unnecessary complexity, it is assumed that f1 and f2 are already equal or can be matched in some way.
[0202] Finally, the summation operation can be expressed as E_first = F_osc1 + M_mean (note that the details of dimension matching are omitted here). The summed first embedding descriptor E_first is a matrix of shape [q, f1] (or [q, f2], depending on the dimension matching method), which contains the comprehensive feature information after coupled oscillation operation and mean calculation, and will serve as input data for subsequent processing steps to support abnormal situation recognition and other advanced tasks.
[0203] Through the detailed explanation and example illustration of the above steps, we can understand the unique application value of the point cloud core integration layer and its internal coupled oscillating network (especially PCNN) in the data processing scenario of industrial equipment based on radar and video fusion calculation. By simulating the pulse emission mechanism in the biological vision system and using the coupled oscillating network to perform nonlinear transformation and feature extraction on the input data, the computer system can generate a more accurate and comprehensive first embedding descriptor. This descriptor not only contains the key feature information in the anomaly descriptor and the point cloud descriptor, but also introduces new feature combinations and interactions through coupled oscillation operations and mean calculations to achieve effective integration and embedding of cross-modal data. This data processing method based on bio-inspired algorithms and deep learning technology provides new ideas and methods for the development of intelligent manufacturing, smart logistics and other fields.
[0204] As a fifth implementation, the first integrated embedding network includes a control type discarding layer; based on this, the step S30, in the first integrated embedding network, performs an integrated embedding operation on the abnormal descriptor according to the point cloud descriptor to obtain a first embedded descriptor, including: discarding the point cloud descriptor and using the abnormal descriptor as the first embedded descriptor output by the control type discarding layer.
[0205] Consider a smart manufacturing environment that includes a highly automated production line and a complex robotic arm system. In order to monitor the operating status of the production line and detect potential problems in a timely manner, high-definition cameras and radar sensors are deployed around the production line. The camera captures video data to identify the movements of the robotic arm and the visual features of the surrounding environment, while the radar provides accurate 3D point cloud data for analyzing the shape, position, and movement trajectory of objects. However, in certain specific situations, such as when a specific area on the production line (such as the working area of the robotic arm) is known to be a focus of attention, and the video data already reflects the status of the area clearly and accurately enough, the radar data may become relatively less important.
[0206] In this context, the fifth implementation of step S30 is particularly reasonable. The computer system decides to use a contrast type dropout layer in the integration embedding process, the core idea of which is to directly discard the point cloud descriptor and use the abnormal descriptor as the output of the integration embedding network, that is, the first embedding descriptor.
[0207] The computer system does not perform any form of processing or integration on the point cloud descriptors. Instead, it simply ignores this part of the data, perhaps because it determines, based on prior knowledge or real-time analysis, that the point cloud data does not contain information that is of substantial help in anomaly identification in the current task, or that the information it provides has been fully covered by the anomaly descriptors. Since the point cloud descriptors are discarded, the computer system directly uses the anomaly descriptors as the output of the control type discard layer, that is, the first embedded descriptor. This step does not actually involve complex feature transformations or fusion algorithms, but it embodies an effective strategy in data processing: when resources are limited or information is redundant, the most relevant or important data sources are retained first.
[0208] Assume that the anomaly descriptor is a matrix A of shape [q, d], where q is the number of samples (e.g., the number of video frames) and d is the feature dimension (e.g., the number of key visual features extracted from the video data). The point cloud descriptor is a complex data structure containing a large number of three-dimensional point coordinates, which may have been converted into some form of low-dimensional embedding representation, but in this implementation, we do not need to care about its specific content.
[0209] When the computer system performs the control type dropout layer operation, it directly ignores the point cloud descriptor and outputs the anomaly descriptor A as the first embedded descriptor. This means that in subsequent anomaly recognition or other advanced processing tasks, the computer system will only rely on information in the video data without considering the radar data.
[0210] It reduces the consumption of computing resources, speeds up processing, and avoids the noise that may be introduced by integrating redundant information. However, it is worth noting that the successful application of this strategy is highly dependent on a deep understanding of task requirements and data characteristics. This control type discarding strategy should only be adopted when it is certain that some data sources are indeed unimportant or redundant in the current task.
[0211] In the smart manufacturing environment, if the video data is clear and accurate enough to reflect the abnormal situation on the production line (such as collision of the robot arm, missing parts, etc.), and the radar data mainly provides the 3D shape and position information of the object (this information may not be necessary for the current task), then it is a reasonable choice to discard the point cloud descriptor using the contrast type discard layer. In this way, the computer system can process multimodal data more efficiently and focus on the information source that is most critical to the abnormality recognition task.
[0212] As an implementation manner, the step S60, determining the abnormal situation identification information corresponding to the target environment according to the abnormal classification label, the first embedded descriptor, and the second embedded descriptor, may specifically include:
[0213] Step S61: combining the first embedded descriptor and the second embedded descriptor to obtain a fourth combined descriptor, and adding the abnormal classification label to the head of the fourth combined descriptor to obtain a fused embedded descriptor;
[0214] Step S62: performing a weight focusing operation on the fused embedded descriptor based on the internal weight focusing layer to obtain a weight focused integrated descriptor, performing an affine transformation operation on the weight focused integrated descriptor to obtain abnormal situation identification information corresponding to the target environment.
[0215] In step S61, the computer system first combines the first embedded descriptor and the second embedded descriptor to generate a comprehensive fourth combined descriptor. The two embedded descriptors represent high-level feature representations of video data and radar data after complex processing, respectively, and each captures key information of different aspects of the target environment.
[0216] For example, suppose the first embedding descriptor is a matrix E1 of shape [q, d1] (where q is the number of samples and d1 is the feature dimension of the first embedding), and the second embedding descriptor is a matrix E2 of shape [q, d2]. The computer system concatenates the two matrices along the feature dimension to generate a fourth combined descriptor C4 = [E1; E2] (note that here; indicates vertical concatenation, but in actual operation, the dimension may need to be adjusted to ensure the feasibility of concatenation). The concatenated C4 has a shape of [q, d1+d2], which contains dual information from video and radar.
[0217] Next, the computer system adds anomaly classification labels to the head of C4. These labels are predefined to indicate different types of anomalies. In practice, the anomaly classification labels may be represented in the form of one-hot encoding or added to the front of C4 as an additional feature vector. To simplify the explanation, assume that the anomaly classification label is a one-hot vector L of length c (where c is the total number of anomaly categories), then the fused embedding descriptor F can be expressed as F = [L; C4].
[0218] By combining the embedded descriptors and adding anomaly classification labels, the computer system generates a fused embedded descriptor that not only contains rich multimodal data features, but also clarifies the type of anomaly that needs to be identified, providing a clear goal and rich information basis for subsequent processing steps.
[0219] In step S62, the computer system uses an internal weight focusing layer (i.e., self-attention layer) to perform a weight focusing operation on the fused embedded descriptor. The self-attention layer can simulate the attention mechanism in the human visual system and dynamically adjust the importance weights of different features so that the features most relevant to the current task receive more attention. For example, the self-attention layer receives the fused embedded descriptor F as input and generates a weight matrix through a series of transformations and calculations. Each element in this weight matrix represents the importance of a feature relative to other features. Then, the self-attention layer applies this weight matrix to F, performs a weighted summation operation, and generates a weighted focused integrated descriptor F_att.
[0220] Next, the computer system performs an affine transformation on F_att. Affine transformation usually includes two parts: linear transformation (i.e., weighted summation) and nonlinear activation. Here, we focus on the linear transformation part, which may be implemented through a fully connected layer that contains a series of learnable weights and bias parameters. After the affine transformation, the output is the abnormal situation identification information I corresponding to the target environment.
[0221] The abnormal situation identification information I may be a probability distribution vector, a category label, or a text string containing a detailed abnormal description, depending on the subsequent processing steps and task requirements. If I is a probability distribution vector, then each element represents the probability that the input data belongs to the corresponding abnormal category; if I is a category label, then it directly indicates the type of abnormal situation currently detected; if I is a text string, then it may contain a detailed description of the abnormal situation and recommended measures. Regardless of the specific form of I, it is the result of the computer system based on multimodal data fusion and machine learning model processing, providing warehouse managers with timely and accurate feedback on abnormal situations, helping them to take quick measures to solve problems and ensure the efficient operation and safety of the warehouse.
[0222] The present application embodiment provides a computer system, such as Figure 2 As shown, the computer system 100 includes: a processor 101 and a memory 103. The processor 101 and the memory 103 are connected, such as through a bus 102. Optionally, the computer system 100 may also include a transceiver 104. It should be noted that in actual applications, the transceiver 104 is not limited to one, and the structure of the computer system 100 does not constitute a limitation on the embodiments of the present application.
[0223] An embodiment of the present application provides a computer system. The computer system in the embodiment of the present application includes: one or more processors; a memory; and one or more computer programs, wherein the one or more computer programs are stored in the memory and configured to be executed by the one or more processors, and when the one or more programs are executed by the processor, the above method is implemented.
Claims
1. A method for industrial equipment data processing based on radar and video fusion calculation, characterized in that: include: Obtaining anomaly descriptors in video acquisition data corresponding to a target environment, and obtaining point cloud descriptors corresponding to the target environment; wherein the target environment is an environment where a target industrial device is located; In the same representation domain, the abnormal descriptor and the point cloud descriptor are subjected to descriptor integration processing to obtain a first fused descriptor, and based on the first fused descriptor, a first integrated embedding network associated with the abnormal descriptor is determined in p integrated embedding networks; the p integrated embedding networks are used to embed multiple descriptors of different types according to different interaction strengths, wherein p≥1; wherein the interaction strength is the fusion strength, and based on the first fused descriptor, the first integrated embedding network associated with the abnormal descriptor is determined in p integrated embedding networks, including: performing category mapping on the first fused descriptor based on the category output network in the expert mixture model to obtain network scores corresponding to the p integrated embedding networks; and using the integrated embedding network corresponding to the largest network score as the first integrated embedding network associated with the abnormal descriptor; In the first integrated embedding network, the abnormal descriptor is integrated and embedded according to the point cloud descriptor to obtain a first embedded descriptor; Performing descriptor integration processing on the point cloud descriptor and the first embedding descriptor in the same representation domain to obtain a second fused descriptor, and determining a second integrated embedding network associated with the point cloud descriptor in the p integrated embedding networks according to the second fused descriptor; In the second integrated embedding network, the point cloud descriptor is integrated and embedded according to the first embedded descriptor to obtain a second embedded descriptor; Acquire an abnormality classification label, where the abnormality classification label is predefined and used to indicate different types of abnormal situations; determine abnormal situation identification information corresponding to the target environment based on the abnormality classification label, the first embedded descriptor, and the second embedded descriptor.
2. The method according to claim 1, characterized in that The step of obtaining an abnormal descriptor in the video acquisition data corresponding to the target environment includes: Obtaining video acquisition data corresponding to the target environment, and dividing the video acquisition data into q video frames; wherein q≥2; Performing descriptor representation on the q video frames to obtain video frame embedding descriptors corresponding to the q video frames; Performing a weighted focusing operation on the video frame embedding descriptors corresponding to the q video frames to obtain focusing influence coefficients corresponding to the q video frames; The video frame embedding descriptors corresponding to the q video frames and the focus influence coefficients corresponding to the q video frames are correspondingly multiplied and then summed to obtain an abnormal descriptor corresponding to the video acquisition data; The obtaining of a point cloud descriptor corresponding to the target environment includes: Obtain r environment point cloud matrices corresponding to the target environment, and combine the r environment point cloud matrices into an environment point cloud tensor; r≥1; The environment point cloud tensor is loaded into an affine network, and an affine transformation operation is performed on the environment point cloud tensor based on the affine network to obtain a point cloud descriptor corresponding to the target environment.
3. The method according to claim 1, characterized in that: The step of performing descriptor integration processing on the abnormal descriptor and the point cloud descriptor in the same representation domain to obtain a first fused descriptor includes: The anomaly descriptor and the point cloud descriptor are loaded into an expert mixture model, and the anomaly descriptor is projected into a representation domain based on a first perceptron network in the expert mixture model to obtain an anomaly projection descriptor; Based on the second perceptron network in the expert mixture model, the point cloud descriptor is projected into a representation domain to obtain a point cloud projection descriptor; the abnormal projection descriptor and the point cloud projection descriptor correspond to the same representation domain; The abnormal projection descriptor and the point cloud projection descriptor are subjected to descriptor integration processing to obtain a first fusion descriptor.
4. The method according to claim 3, characterized in that The step of performing descriptor integration processing on the abnormal projection descriptor and the point cloud projection descriptor to obtain a first fusion descriptor includes: Obtaining a first intensity effective value corresponding to the abnormal projection descriptor, and performing a standardization operation on the abnormal projection descriptor according to the first intensity effective value to obtain an abnormal standardized descriptor; the first intensity effective value is a statistic that can reflect feature intensity or distribution characteristics; Performing a nonlinear transformation on the abnormal normalized descriptor to obtain an abnormal activated descriptor, and performing mean calculation on the abnormal activated descriptor to obtain a first mean descriptor; Acquire a second intensity effective value corresponding to the point cloud projection descriptor, and perform a standardization operation on the point cloud projection descriptor according to the second intensity effective value to obtain a point cloud standardized descriptor; Performing a nonlinear transformation on the point cloud normalization descriptor to obtain a point cloud activation descriptor, and performing mean calculation on the point cloud activation descriptor to obtain a second mean descriptor; The first mean descriptor and the second mean descriptor are summed to obtain a mean sum descriptor, and an affine transformation operation is performed on the mean sum descriptor according to a parameter matrix corresponding to a dense network in the expert mixture model to obtain the first fusion descriptor.
5. The method according to claim 1, characterized in that The first integrated embedding network includes a weighted focused integrated network; in the first integrated embedding network, the abnormal descriptor is integrated and embedded according to the point cloud descriptor to obtain a first embedded descriptor, including: Combining the anomaly descriptor and the point cloud descriptor to obtain a first combined descriptor; Performing a self-weighted focusing operation on the first combined descriptor based on the weighted focusing integration network to obtain a first internal focusing descriptor; The first q rows of descriptors in the first internal focus descriptor are used as the first embedded descriptor; q is the number of rows of the abnormal descriptor.
6. The method according to claim 5, characterized in that The step of performing a self-weighted focusing operation on the first combined descriptor based on the weighted focusing integration network to obtain a first internal focusing descriptor includes: According to the conversion parameter matrix corresponding to the weighted focusing integration network, the first combined descriptor is converted into a first search array, a first anchor array and a first result array; wherein the first search array is the query in the attention mechanism, the anchor array is the key in the attention mechanism, and the result array is the value in the attention mechanism; Generate a first candidate array according to a multiplication result between the first search array and the abnormal descriptor, and a multiplication result between the first search array and the point cloud descriptor; Generate a second candidate array according to a flip array of a multiplication result between the first anchor array and the abnormal descriptor, and a flip array of a multiplication result between the first anchor array and the point cloud descriptor; Generate a third candidate array according to the multiplication result between the first result array and the abnormal descriptor, and the multiplication result between the first result array and the point cloud descriptor; Taking the multiplication result of the first candidate array and the second candidate array as a fourth candidate array, and obtaining the number of columns of the abnormal descriptor; The ratio between the fourth candidate array and the square root of the number of columns is normalized to obtain an internal focus influence coefficient array, and the multiplication result between the internal focus influence coefficient array and the third candidate array is used as the first internal focus descriptor.
7. The method according to claim 1, characterized in that The first integrated embedding network includes a weighted focused integrated network; in the first integrated embedding network, the abnormal descriptor is integrated and embedded according to the point cloud descriptor to obtain a first embedded descriptor, including: Based on the weighted focusing integration network, the anomaly descriptor is converted into a second search array, and the point cloud descriptor is converted into a second anchor array and a second result array; Multiplying the multiplication result between the second search array and the abnormal descriptor and the flip array of the multiplication result between the second anchor array and the point cloud descriptor to obtain a fifth candidate array, and obtaining the number of columns of the abnormal descriptor; Standardizing the ratio between the fifth candidate array and the square root of the number of columns to obtain a cross-focus influence coefficient array, and multiplying the cross-focus influence coefficient array and the multiplication result between the second result array and the point cloud descriptor to obtain a cross-focus descriptor; The first q rows of descriptors in the cross-focus descriptor are used as the first embedded descriptor, where q is the number of rows of the abnormal descriptor.
8. The method according to claim 1, characterized in that The first integrated embedding network includes a deep weighted focused integrated network; then in the first integrated embedding network, the abnormal descriptor is integrated and embedded according to the point cloud descriptor to obtain a first embedded descriptor, including: Acquire a deep descriptor, and combine the deep descriptor with the point cloud descriptor to obtain a second combined descriptor, wherein the deep descriptor is used to facilitate the connection of descriptors of different categories; Based on the first weighted focusing layer in the deep weighted focusing integration network, the second combined descriptor is subjected to a self-weighted focusing operation to obtain a second internally focused descriptor, and the first e rows of descriptors in the second internally focused descriptor are used as candidate embedded descriptors; e is the number of rows of the deep descriptor; The abnormal descriptor and the candidate embedded descriptor are combined to obtain a third combined descriptor, and based on the second weight focusing layer in the deep weight focusing integration network, the third combined descriptor is subjected to a self-weight focusing operation to obtain a third internal focusing descriptor; Using the first q rows of descriptors in the third internal focus descriptor as the first embedded descriptor, where q is the number of rows of the abnormal descriptor; Alternatively, the first integrated embedding network includes a point cloud core integration layer; then in the first integrated embedding network, the abnormal descriptor is integrated and embedded according to the point cloud descriptor to obtain a first embedded descriptor, including: Acquire an abnormal standardized descriptor corresponding to the abnormal descriptor, and a point cloud standardized descriptor corresponding to the point cloud descriptor; Based on the first coupled oscillation network in the point cloud core integration layer, performing a coupled oscillation operation on the abnormal descriptor to obtain a first coupled oscillation descriptor; Based on the second coupled oscillation network in the point cloud core integration layer, performing a coupled oscillation operation on the point cloud descriptor to obtain a second coupled oscillation descriptor; Performing mean calculation on the second coupled oscillation descriptor to obtain a third mean descriptor, and summing the first coupled oscillation descriptor and the third mean descriptor to obtain a first embedded descriptor; Alternatively, the first integrated embedding network includes a comparison type discarding layer; then, in the first integrated embedding network, the abnormal descriptor is integrated and embedded according to the point cloud descriptor to obtain a first embedded descriptor, including: The point cloud descriptor is discarded, and the abnormal descriptor is used as the first embedded descriptor output by the comparison type discarding layer.
9. The method according to claim 1, characterized in that: The determining, based on the abnormal classification label, the first embedded descriptor, and the second embedded descriptor, abnormal situation identification information corresponding to the target environment includes: Combining the first embedded descriptor and the second embedded descriptor to obtain a fourth combined descriptor, and adding the abnormal classification label to a head of the fourth combined descriptor to obtain a fused embedded descriptor; Based on the internal weight focusing layer, a weight focusing operation is performed on the fused embedded descriptor to obtain a weight focusing integrated descriptor, and an affine transformation operation is performed on the weight focusing integrated descriptor to obtain abnormal situation identification information corresponding to the target environment.
10. A computer system, characterized in that: include: one or more processors; Memory; one or more computer programs; The one or more computer programs are stored in the memory and configured to be executed by the one or more processors, and when the one or more computer programs are executed by the processors, the method according to any one of claims 1 to 9 is implemented.
Citation Information
Patent Citations
Forest fire detection, early warning and decision-making system based on sky-ground integration technology
CN115348247A
Radar and video fusion target detection method for enhancing feature information
CN116797894A
Point cloud image fusion method and system for vehicle navigation
CN117911829A
Video identification method and device, computer equipment and storage medium
CN118230204A
