Traffic violation event identification method and system based on fusion data

Abstract

The invention provides a traffic violation event identification method and system based on fusion data, and relates to the technical field of computers. The method comprises the following steps: firstly, mining a traffic scene semantic vector corresponding to a traffic scene image and a radar point cloud semantic vector corresponding to radar point cloud data; secondly, performing cross attention mapping on the traffic scene semantic vector and the radar point cloud semantic vector to form a first attention vector, a second attention vector and a third attention vector; then, performing cross attention fusion on the first attention vector, the second attention vector and the third attention vector based on the image shielding parameter and the point cloud shielding parameter to form a traffic scene fusion vector; and finally, illegal event identification is carried out based on the traffic scene fusion vector, and an illegal event identification result is formed. Based on the method, the problem that the reliability of traffic violation event recognition is relatively low in the prior art can be improved.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a method and system for identifying traffic violations based on fused data. Background Art

[0002] Traffic violation identification technology plays a vital role in intelligent transportation systems and is widely used in urban road monitoring and traffic management. However, existing technologies still face numerous challenges in handling complex traffic scenarios. For example, existing technologies typically rely on either image analysis or radar data for violation identification. A single sensing method cannot fully capture key information in traffic scenarios. Image analysis relies on high-resolution images and complex algorithms, while radar data is susceptible to factors such as weather and obstacles, resulting in unstable recognition. Another example is that in complex urban traffic environments, due to occlusions such as pedestrians and vehicles, perception based on a single data source carries a significant risk of misjudgment. Existing technologies often rely on a single data source when handling occlusion, lacking effective occlusion handling methods, resulting in reduced recognition accuracy. Furthermore, existing methods for fusing multimodal data (such as image and radar data) are relatively simplistic and lack systematic research, making it difficult to fully leverage their complementary nature. This impacts overall recognition performance and results in relatively low recognition reliability. Summary of the Invention

[0003] In view of this, an object of the present invention is to provide a method and system for identifying traffic violations based on fused data, so as to improve the problem of relatively low reliability of traffic violation identification in the prior art.

[0004] To achieve the above objectives, the embodiments of the present invention adopt the following technical solutions: A traffic violation event recognition method based on fusion data, comprising: Mining a traffic scene semantic vector corresponding to a traffic scene image of a target traffic scene and a radar point cloud semantic vector corresponding to radar point cloud data, wherein the radar point cloud data is formed by collecting information of the target traffic scene through a millimeter-wave radar; Performing cross-attention mapping on the traffic scene semantic vector and the radar point cloud semantic vector respectively to form a first attention vector of the traffic scene semantic vector, a second attention vector and a third attention vector of the radar point cloud semantic vector; Based on the image masking parameter of the traffic scene image and the point cloud masking parameter of the radar point cloud data, cross-attention fusion is performed on the first attention vector of the traffic scene semantic vector, the second attention vector and the third attention vector of the radar point cloud semantic vector to form a traffic scene fusion vector; Illegal events are identified based on the traffic scene fusion vector to form an illegal event identification result of the target traffic scene.

[0005] In some preferred embodiments, in the above-mentioned method for identifying traffic violations based on fused data, the step of mining the traffic scene semantic vector corresponding to the traffic scene image of the target traffic scene and the radar point cloud semantic vector corresponding to the radar point cloud data includes: Performing a two-dimensional Fourier transform on a traffic scene image of a target traffic scene to form a traffic scene spectrogram, and performing a deep convolution on the traffic scene spectrogram to form a traffic scene semantic vector corresponding to the traffic scene image; Expanding discrete point cloud data in the radar point cloud data of the target traffic scene into a continuous three-dimensional grid to form three-dimensional radar point cloud data, wherein in the three-dimensional radar point cloud data, data at a position corresponding to the point cloud data is 1, and data at other positions is 0; Performing a three-dimensional Fourier transform on the three-dimensional radar point cloud data to form a radar point cloud spectrum graph; Deep convolution is performed on the radar point cloud spectrogram to form a radar point cloud semantic vector corresponding to the radar point cloud data.

[0006] In some preferred embodiments, in the above-mentioned traffic violation event identification method based on fused data, the step of performing deep convolution on the radar point cloud spectrogram to form a radar point cloud semantic vector corresponding to the radar point cloud data includes: Performing channel-wise convolution on the radar point cloud spectrum to form multiple channel convolution vectors; Splicing the multiple channel convolution vectors to form a channel splicing vector; According to the dynamic adjustment parameters, the channel splicing vector is subjected to an association semantic mining operation for M time steps to form a radar point cloud semantic vector, wherein the dynamic adjustment parameters are used to adjust the association mining parameters at different time steps, and the association mining parameters are used to adjust the mined association semantic distribution in the association semantic mining operation.

[0007] In some preferred embodiments, in the above-mentioned traffic violation event identification method based on fused data, the step of performing an association semantic mining operation on the channel splicing vector for M time steps based on the dynamic adjustment parameter to form a radar point cloud semantic vector includes: For the a-th time step among the M time steps, concatenate the associated semantic vector of the a-1-th time step and the dynamic adjustment parameter of the a-th time step to form the semantic vector to be mined of the a-th time step, where a is a positive integer less than or equal to M. When a=1, the associated semantic vector of the a-1-th time step is the channel splicing vector; Performing an associated semantic mining operation on the semantic vector to be mined at the a-th time step to form an associated semantic vector at the a-th time step, wherein the associated semantic mining operation sequentially includes an attention operation, a first vector compression operation, a nonlinear mapping operation, and a second vector compression operation; If a is less than M, then a=a+1 is determined, and the step of concatenating the associated semantic vector of the a-1th time step and the dynamic adjustment parameter of the ath time step for the ath time step among the M time steps is performed in reverse order to form the semantic vector to be mined of the ath time step; If a is equal to M, the associated semantic vector of the a-th time step is determined to be the radar point cloud semantic vector corresponding to the radar point cloud data.

[0008] In some preferred embodiments, in the above-mentioned traffic violation event identification method based on fused data, the step of performing an association semantic mining operation on the channel splicing vector for M time steps based on the dynamic adjustment parameter to form a radar point cloud semantic vector also includes the step of determining the dynamic adjustment parameter for the ath time step, which step includes: Determining configuration association mining parameters, wherein the configuration association mining parameters are formed by learning a mapping relationship between traffic scene image samples, radar point cloud data samples, and violation event labels; For an a-th time step among the M time steps, when a is less than a first preset value, using the configuration association mining parameter as a dynamic adjustment parameter for the a-th time step; When a is greater than or equal to the first preset value and less than a second preset value, performing a sum calculation on the configuration association mining parameter and the first adjustment parameter to form the dynamic adjustment parameter of the a-th time step; When a is greater than or equal to the second preset value and less than or equal to M, the configuration association mining parameter and the second adjustment parameter are summed to form the dynamic adjustment parameter of the ath time step, wherein the second adjustment parameter is greater than the first adjustment parameter.

[0009] In some preferred embodiments, in the above-mentioned traffic violation event identification method based on fused data, the step of performing cross-attention fusion on the first attention vector of the traffic scene semantic vector, the second attention vector and the third attention vector of the radar point cloud semantic vector based on the image masking parameters of the traffic scene image and the point cloud masking parameters of the radar point cloud data to form a traffic scene fusion vector includes: fusing image masking parameters of the traffic scene image and point cloud masking parameters of the radar point cloud data to form fused masking parameters, wherein the image masking parameters and the point cloud masking parameters are formed by learning a mapping relationship between traffic scene image samples, radar point cloud data samples, and violation event labels; Based on the fusion masking parameter, cross-attention-fusing the first attention vector of the traffic scene semantic vector, the second attention vector and the third attention vector of the radar point cloud semantic vector to form an attention fusion vector; The attention fusion vector is mapped into a vector space to form a traffic scene fusion vector.

[0010] In some preferred embodiments, in the above-mentioned traffic violation event identification method based on fused data, the step of fusing the image masking parameters of the traffic scene image and the point cloud masking parameters of the radar point cloud data to form fused masking parameters includes: Performing a bitwise OR operation on an image masking parameter of the traffic scene image and a point cloud masking parameter of the radar point cloud data to form a first fused masking parameter, wherein the image masking parameter is a matrix composed of 0s and 1s, and the point cloud masking parameter is a matrix composed of 0s and 1s; performing a bitwise AND operation on the image masking parameter of the traffic scene image and the point cloud masking parameter of the radar point cloud data to form a second fused masking parameter; The first fusion masking parameter and the second fusion masking parameter are used as fusion masking parameters.

[0011] In some preferred embodiments, in the above-mentioned traffic violation event recognition method based on fused data, the step of performing cross-attention fusion on the first attention vector of the traffic scene semantic vector, the second attention vector and the third attention vector of the radar point cloud semantic vector based on the fusion masking parameter to form an attention fusion vector includes: Performing matrix multiplication on a first attention vector of the traffic scene semantic vector and a second attention vector of the radar point cloud semantic vector to form an attention parameter distribution; Performing a bitwise addition operation on the attention parameter distribution and the first fused masking parameter and the second fused masking parameter included in the fused masking parameter to form a first adjustment parameter distribution and a second adjustment parameter distribution; performing a weighted sum calculation on the third attention vector of the radar point cloud semantic vector based on the first adjustment parameter distribution and the second adjustment parameter distribution, respectively, to form a first fusion vector and a second fusion vector; An attention fusion vector is formed based on the first fusion vector and the second fusion vector.

[0012] In some preferred embodiments, in the above-mentioned traffic violation event identification method based on fused data, the step of performing cross-attention fusion on the first attention vector of the traffic scene semantic vector, the second attention vector and the third attention vector of the radar point cloud semantic vector based on the image masking parameters of the traffic scene image and the point cloud masking parameters of the radar point cloud data to form a traffic scene fusion vector includes: Segmenting the first attention vector of the traffic scene semantic vector based on the length of the image masking parameter of the traffic scene image to form a local first attention vector, and segmenting the second attention vector and the third attention vector of the radar point cloud semantic vector based on the point cloud masking parameter of the radar point cloud data to form a local second attention vector and a local third attention vector; Performing a bitwise multiplication operation on the image masking parameter and the local first attention vector to form a local first adjustment vector, and performing a bitwise multiplication operation on the point cloud masking parameter and the local second attention vector and the local third attention vector respectively to form a local second adjustment vector and a local third adjustment vector; performing a weighted sum calculation on the local third adjustment vector based on the attention parameter distribution between the local first adjustment vector and the local second adjustment vector to form a local attention fusion vector; Based on each local attention fusion vector, an attention fusion vector is formed.

[0013] An embodiment of the present invention also provides a traffic violation event recognition system based on fused data, including a processor and a memory, wherein the memory is used to store a computer program, and the processor is used to execute the computer program to implement the above-mentioned traffic violation event recognition method based on fused data.

[0014] The traffic violation event identification method and system based on fused data provided by the embodiment of the present invention first mine the traffic scene semantic vector corresponding to the traffic scene image and the radar point cloud semantic vector corresponding to the radar point cloud data; secondly, cross-attention mapping is performed on the traffic scene semantic vector and the radar point cloud semantic vector respectively to form a first attention vector, a second attention vector and a third attention vector; then, based on the image masking parameters and the point cloud masking parameters, the first attention vector, the second attention vector and the third attention vector are cross-attention fused to form a traffic scene fusion vector; finally, violation event identification is performed based on the traffic scene fusion vector to form a violation event identification result. Based on the above method, since corresponding masking will be performed based on the image masking parameters and the point cloud masking parameters during the cross-attention fusion process, the cross-fusion of the corresponding semantic vectors can be effectively controlled, thereby achieving attention to the effective semantic information in the corresponding semantic vectors and avoiding interference from invalid semantic information, so that the semantic representation of the formed traffic scene fusion vector is more accurate, thereby ensuring the reliability of the violation event recognition results obtained based on the traffic scene fusion vector, that is, compared with conventional technical means such as using data from a single data source for recognition and analysis or simply fusing data from multiple data sources, it has higher reliability. Therefore, it can improve the problem of relatively low reliability of traffic violation event recognition in the existing technology.

[0015] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, preferred embodiments are given below and described in detail with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 This is a structural block diagram of a traffic violation event recognition system based on fused data provided by an embodiment of the present invention.

[0017] Figure 2 A schematic diagram of the modules included in the traffic violation event identification device based on fusion data provided in an embodiment of the present invention.

[0018] Figure 3 A flow chart of the steps of a method for identifying traffic violations based on fused data provided in an embodiment of the present invention.

[0019] Figure 4 A schematic diagram of semantic mining provided by an embodiment of the present invention.

[0020] Figure 5 A schematic diagram of depthwise convolution provided by an embodiment of the present invention.

[0021] Figure 6 A schematic diagram of masking parameter fusion provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0022] To make the objectives, technical solutions, and advantages of the embodiments of the present invention more clear, the technical solutions of the embodiments of the present invention will be clearly and completely described below in conjunction with the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Generally, the components of the embodiments of the present invention described and shown in the drawings herein can be arranged and designed in various different configurations.

[0023] Therefore, the following detailed description of the embodiments of the present invention provided in the accompanying drawings is not intended to limit the scope of the invention as claimed, but rather merely represents selected embodiments of the present invention. All other embodiments derived by persons of ordinary skill in the art based on the embodiments of the present invention without creative effort shall fall within the scope of protection of the present invention.

[0024] like Figure 1 As shown, an embodiment of the present invention provides a traffic violation event recognition system based on fused data. The traffic violation event recognition system based on fused data may include a memory and a processor.

[0025] Specifically, the memory and the processor are electrically connected directly or indirectly to achieve data transmission or interaction. For example, the electrical connection between them can be achieved through one or more communication buses or signal lines. The memory can store at least one software function module (such as a software module) in the form of software or firmware. Figure 2 The processor may be configured to execute the executable computer program stored in the memory, thereby implementing the method for identifying traffic violations based on fused data provided in an embodiment of the present invention (as described below).

[0026] Optionally, the traffic violation event identification device based on fused data may include: a semantic mining module for mining a traffic scene semantic vector corresponding to a traffic scene image of a target traffic scene and a radar point cloud semantic vector corresponding to radar point cloud data, wherein the radar point cloud data is formed by collecting information of the target traffic scene using a millimeter-wave radar; an attention mapping module, configured to perform cross-attention mapping on the traffic scene semantic vector and the radar point cloud semantic vector, respectively, to form a first attention vector for the traffic scene semantic vector, a second attention vector for the radar point cloud semantic vector, and a third attention vector; an attention fusion module, configured to perform cross-attention fusion on the first attention vector of the traffic scene semantic vector, the second attention vector and the third attention vector of the radar point cloud semantic vector based on the image masking parameter of the traffic scene image and the point cloud masking parameter of the radar point cloud data, to form a traffic scene fusion vector; The illegal event recognition module is used to identify illegal events based on the traffic scene fusion vector to form an illegal event recognition result of the target traffic scene.

[0027] Optionally, the memory may be, but is not limited to, a random access memory (RAM), a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), etc.

[0028] Optionally, the processor may be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), a system on chip (SoC), etc.; it may also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.

[0029] and, Figure 1 The structure shown is for illustration only. The traffic violation event recognition system based on fusion data may also include Figure 1 More or fewer components than shown, or with Figure 1 The different configurations shown, for example, may include a communication unit for interacting with other devices (such as image sensors, millimeter-wave radars, and other traffic scene information collection devices).

[0030] In an alternative example, the traffic violation event recognition system based on fused data may be a server with data processing capabilities.

[0031] Combine Figure 3The embodiment of the present invention also provides a method for identifying traffic violations based on fused data, which can be applied to the above-mentioned system for identifying traffic violations based on fused data. The method steps defined in the process related to the method for identifying traffic violations based on fused data can be implemented by the system for identifying traffic violations based on fused data (hereinafter referred to as the identification system). Figure 3 The specific process shown is explained in detail.

[0032] Step S110 , mining a traffic scene semantic vector corresponding to the traffic scene image of the target traffic scene and a radar point cloud semantic vector corresponding to the radar point cloud data.

[0033] In an embodiment of the present invention, the recognition system can mine a traffic scene semantic vector corresponding to a traffic scene image of a target traffic scene and a radar point cloud semantic vector corresponding to radar point cloud data. That is, semantic mining can be performed on the traffic scene image to form a corresponding traffic scene semantic vector, that is, potential semantic information in the traffic scene image is mined and represented in the form of a vector. Furthermore, semantic mining can be performed on the radar point cloud data to form a corresponding radar point cloud semantic vector, that is, potential semantic information in the radar point cloud data is mined and represented in the form of a vector. The radar point cloud data is formed by collecting information about the target traffic scene (such as a traffic intersection) using a millimeter-wave radar. The traffic scene image can be formed by collecting information using a camera installed at the target traffic scene.

[0034] Step S120 , cross-attention mapping is performed on the traffic scene semantic vector and the radar point cloud semantic vector to form a first attention vector of the traffic scene semantic vector, a second attention vector and a third attention vector of the radar point cloud semantic vector.

[0035] In an embodiment of the present invention, after mining the traffic scene semantic vector and the radar point cloud semantic vector, the recognition system may perform cross-attention mapping on the traffic scene semantic vector and the radar point cloud semantic vector, respectively, to form a first attention vector for the traffic scene semantic vector, a second attention vector for the radar point cloud semantic vector, and a third attention vector. For example, the first attention vector may also be referred to as a query vector, the second attention vector may also be referred to as a key vector, and the third attention vector may also be referred to as a value vector.

[0036] Step S130, based on the image masking parameters of the traffic scene image and the point cloud masking parameters of the radar point cloud data, the first attention vector of the traffic scene semantic vector, the second attention vector and the third attention vector of the radar point cloud semantic vector are cross-attended and fused to form a traffic scene fusion vector.

[0037] In an embodiment of the present invention, when forming the first attention vector, the second attention vector, and the third attention vector, the recognition system may perform cross-attention fusion on the first attention vector of the traffic scene semantic vector, the second attention vector of the radar point cloud semantic vector, and the third attention vector of the radar point cloud semantic vector based on the image masking parameters of the traffic scene image and the point cloud masking parameters of the radar point cloud data to form a traffic scene fusion vector. The image masking parameters and the point cloud masking parameters may be formed by learning the mapping relationship between traffic scene image samples, radar point cloud data samples, and violation event labels. That is, by learning the mapping relationship between traffic scene image samples, radar point cloud data samples, and violation event labels, it is possible to determine which semantic information is valid and which is invalid. This semantic information relationship learned from the samples and labels is then applied to the semantic fusion of the traffic scene image and the radar point cloud data, enabling focus on valid semantic information during the fusion process and masking or discarding invalid semantic information, thereby ensuring that the formed traffic scene fusion vector has a high semantic representation accuracy.

[0038] Step S140 , performing illegal event recognition based on the traffic scene fusion vector to form an illegal event recognition result of the target traffic scene.

[0039] In an embodiment of the present invention, after forming the traffic scene fusion vector, the recognition system may perform violation event recognition based on the traffic scene fusion vector to generate a violation event recognition result for the target traffic scene. For example, classification output may be performed based on the traffic scene fusion vector, such as binary classification or multi-classification. The binary classification result may be used to indicate whether the target traffic scene contains a traffic violation, and the multi-classification result may be used to indicate the specific type of traffic violation present in the target traffic scene (including no traffic violation and at least one other traffic violation, such as speeding, running a red light, or crossing a line).

[0040] Based on the above method, since corresponding masking will be performed based on the image masking parameters and the point cloud masking parameters during the cross-attention fusion process, the cross-fusion of the corresponding semantic vectors can be effectively controlled, thereby achieving attention to the effective semantic information in the corresponding semantic vectors and avoiding interference from invalid semantic information, so that the semantic representation of the formed traffic scene fusion vector is more accurate, thereby ensuring the reliability of the violation event recognition results obtained based on the traffic scene fusion vector, that is, compared with conventional technical means such as using data from a single data source for recognition and analysis or simply fusing data from multiple data sources, it has higher reliability. Therefore, it can improve the problem of relatively low reliability of traffic violation event recognition in the existing technology.

[0041] In the first part, it should be explained about step S110 that the specific process of mining the traffic scene semantic vector corresponding to the traffic scene image of the target traffic scene and the radar point cloud semantic vector corresponding to the radar point cloud data is not restricted and can be configured according to actual needs.

[0042] For example, in one feasible implementation, convolution processing can be performed on the traffic scene image to form a corresponding traffic scene semantic vector. Furthermore, the radar point cloud data can be processed based on a graph neural network (e.g., DGCNN, Dynamic Graph CNN, a deep learning model for point cloud processing) to form a corresponding radar point cloud semantic vector.

[0043] For another example, in another feasible implementation, considering that in the subsequent steps (such as step S130), the semantic vectors corresponding to the data of the two modalities will be cross-attention fused, therefore, in order to ensure the reliability of the cross-attention fusion, the corresponding semantic mining can be used to make the formed semantic vectors in a similar semantic space, thereby ensuring the adaptability between the semantic vectors during fusion. Based on this, the above-mentioned step S110 can further include step S111, step S112, step S113 and step S114. The specific content of each step is as follows (combined with Figure 4 shown).

[0044] Step S111 , performing a two-dimensional Fourier transform on the traffic scene image of the target traffic scene to form a traffic scene spectrum graph, and performing a deep convolution on the traffic scene spectrum graph to form a traffic scene semantic vector corresponding to the traffic scene image.

[0045] In an embodiment of the present invention, a two-dimensional Fourier transform can be performed on the traffic scene image of the target traffic scene to form a traffic scene spectrogram, and a deep convolution can be performed on the traffic scene spectrogram to form a traffic scene semantic vector corresponding to the traffic scene image. In other words, the traffic scene image can be first converted from the spatial domain to the frequency domain to obtain a traffic scene spectrogram, and then the traffic scene spectrogram can be deep convolved to capture high-level, abstract semantic information therein, thereby obtaining a traffic scene semantic vector. The formula for performing a two-dimensional Fourier transform on the traffic scene image can be as follows: ; The size of the traffic scene image is M*N, f(x,y) is the pixel value at position (x,y) in the traffic scene image. F(u,v) is the value of the transformed spectrogram at position (u,v). M and N are the number of rows and columns of the traffic scene image, respectively. j is an imaginary unit. In addition, the network architecture for performing deep convolution on the traffic scene spectrogram may include: 2D convolution layer (convolution kernel 1): The convolution kernel size is 3×3, the number of output channels is 64, and the activation function is ReLU (Rectified Linear Unit); Max pooling layer 1: pooling size is 2x2, stride is 2; 2D convolution layer (convolution kernel 2): convolution kernel size is 3×3, the number of output channels is 128, and the activation function is ReLU; Max pooling layer 2: pooling size is 2x2, stride is 2; 2D convolution layer (convolution kernel 3): The convolution kernel size is 3×3, the number of output channels is 256, and the activation function is ReLU; Max pooling layer 3: pooling size is 2x2, stride is 2; Fully connected layer: The output size is 1x512 and the activation function is ReLU.

[0046] Step S112 : Expanding the discrete point cloud data in the radar point cloud data of the target traffic scene into a continuous three-dimensional grid to form radar point cloud three-dimensional data.

[0047] In an embodiment of the present invention, the discrete point cloud data in the radar point cloud data of the target traffic scene can be expanded into a continuous three-dimensional grid to form three-dimensional radar point cloud data. In the three-dimensional radar point cloud data, the data at the location corresponding to the point cloud data is 1, and the data at other locations is 0. For example, first, the minimum and maximum coordinate ranges of the three-dimensional grid need to be determined, i.e., the minimum and maximum coordinate values of the radar point cloud data. For example, if the x range of the radar point cloud data is [-10, 10], the y range is [-10, 10], and the z range is [-5, 5], then the grid range can be set to: x: -10 to 10, y: -10 to 10, and z: -5 to 5. Then, the grid resolution is set, i.e., the resolution of the grid in each dimension is determined, i.e., the number of grid cells to be divided into on each axis. For example, each dimension can be divided into 100 grid cells, so that the entire grid contains 100×100×100=1,000,000 grid points. The coordinate origin can then be set at the center of the radar point cloud data to reduce errors caused by offset. For example, if the point cloud's x-range is [-10, 10], the origin is set at x=0, and y and z are handled similarly. Thus, for each grid point, if there is point cloud data for that grid point, the value is assigned to 1, and if there is no point cloud data for that grid point, the value is assigned to 0, thus forming the three-dimensional radar point cloud data.

[0048] Step S113: performing a three-dimensional Fourier transform on the three-dimensional radar point cloud data to form a radar point cloud spectrum.

[0049] In an embodiment of the present invention, after forming the three-dimensional radar point cloud data, a three-dimensional Fourier transform can be performed on the three-dimensional radar point cloud data to form a radar point cloud spectrogram. This converts the three-dimensional radar point cloud data in the spatial domain into the frequency domain. The formula for performing the three-dimensional Fourier transform on the three-dimensional radar point cloud data can be as follows: ; The size of the 3D radar point cloud data is M*N*P, where f(x, y, z) is the value at position (x, y, z) in the 3D radar point cloud data. F(k, l, m) is the value at position (k, l, m) in the transformed spectrogram. M, N, and P are the number of rows, columns, and channels of the 3D radar point cloud data, respectively. j is the imaginary unit.

[0050] Step S114: performing deep convolution on the radar point cloud spectrogram to form a radar point cloud semantic vector corresponding to the radar point cloud data.

[0051] In this embodiment of the present invention, after forming a radar point cloud spectrogram, deep convolution can be performed on the radar point cloud spectrogram to form a radar point cloud semantic vector corresponding to the radar point cloud data. Because both the traffic scene semantic vector and the radar point cloud semantic vector are formed by deep convolution of frequency-domain data (i.e., the spectrogram), they reside in similar semantic spaces, improving the reliability of cross-attention fusion in subsequent steps.

[0052] Optionally, in the above-mentioned step S114, the specific process of performing deep convolution on the radar point cloud spectrogram is not limited and can be configured according to actual needs.

[0053] For example, in a feasible implementation, in order to more accurately capture the association between key semantic information, thereby improving the accuracy of the semantic vector, the above step S114 may further include step S114a, step S114b and step S114c, and the specific content of each step is as follows (combined with Figure 5 ).

[0054] Step S114a: performing channel-wise convolution on the radar point cloud spectrogram to form a plurality of channel convolution vectors.

[0055] In an embodiment of the present invention, the radar point cloud spectrogram can be convolved by channel to form multiple channel convolution vectors. As mentioned above, the radar point cloud spectrogram is three-dimensional, i.e., x*y*z, where z represents the number of channels. In this way, the two-dimensional spectrogram of each channel can be convolved separately, thus forming multiple channel convolution vectors. Based on this, by performing convolution operations on different feature layers of the spectrogram separately through channel convolution, the unique features in different channels can be effectively extracted. This helps to capture multi-dimensional information and enrich the diversity of feature representation.

[0056] Step S114b: splicing the multiple channel convolution vectors to form a channel splicing vector.

[0057] In an embodiment of the present invention, after forming multiple channel convolution vectors, the multiple channel convolution vectors can be spliced to form a channel splicing vector. In this way, the multiple channel convolution vectors can be simply and quickly preliminarily fused to obtain a semantically rich channel splicing vector, that is, the features extracted from different channels are integrated together to form a comprehensive semantic description.

[0058] Step S114c: performing an association semantic mining operation on the channel splicing vector for M time steps according to the dynamic adjustment parameters to form a radar point cloud semantic vector.

[0059] In an embodiment of the present invention, after forming the channel splicing vector, the channel splicing vector can be subjected to an association semantic mining operation for M time steps based on a dynamic adjustment parameter to form a radar point cloud semantic vector. The dynamic adjustment parameter is used to adjust the association mining parameters at different time steps, and the association mining parameters are used to adjust the distribution of mined association semantics in the association semantic mining operation. Based on this, by adjusting the association mining parameters, it is possible to flexibly focus on feature associations at different time steps, thereby enhancing the ability to capture complex semantic information and enabling more accurate capture of associations between key features, thereby improving the accuracy of the semantic vector.

[0060] Optionally, in the above step S114c, the specific process of performing the association semantic mining operation on the channel splicing vector for M time steps is not limited. For example, in a feasible implementation, in order to capture effective semantic information during the association semantic mining operation, the above step S114c may further include the following: First, for the ath time step among the M time steps, the associated semantic vector of the a-1th time step and the dynamic adjustment parameter of the ath time step (which can be a matrix or a vector) can be spliced to form the semantic vector to be mined of the ath time step, wherein a is a positive integer less than or equal to M. When a=1, the associated semantic vector of the a-1th time step is the channel splicing vector, that is, in the first time step, the channel splicing vector and the dynamic adjustment parameter of the first time step can be spliced to form the semantic vector to be mined of the first time step. In the second time step, the associated semantic vector of the first time step and the dynamic adjustment parameter of the second time step can be spliced to form the semantic vector to be mined of the second time step. And so on, the semantic vector to be mined of the third time step can be formed. Secondly, an associated semantic mining operation can be performed on the semantic vector to be mined at the a-th time step to form an associated semantic vector at the a-th time step, wherein the associated semantic mining operation sequentially includes an attention operation (for example, which can be implemented by a self-attention network layer to achieve associated mining of different channel convolution vectors and historical semantic information, wherein the dynamic adjustment parameter is related to the learned traffic scene image samples, radar point cloud data samples and violation event labels, that is, characterizing historical semantic information), a first vector compression operation (for example, which can be implemented by a pooling network layer), a nonlinear mapping operation (for example, which can be implemented by a fully connected network layer) and a second vector compression operation (for example, which can be implemented by a pooling network layer); If a is less than M, a=a+1 is determined (i.e., after the first time step is completed, the associated semantic mining operation of the second time step is performed), so as to reversely execute the step of concatenating the associated semantic vector of the a-1 time step and the dynamic adjustment parameter of the a-th time step for the a-th time step among the M time steps to form the semantic vector to be mined of the a-th time step. In this way, the semantic vector to be mined and the associated semantic vector of each time step can be obtained; Furthermore, if a is equal to M, the associated semantic vector of the a-th time step is determined to be the radar point cloud semantic vector corresponding to the radar point cloud data, that is, the associated semantic vector of the last time step is determined to be the radar point cloud semantic vector corresponding to the radar point cloud data.

[0061] Alternatively, in a feasible implementation, the above step S114c may further include a specific process of determining the dynamic adjustment parameter of the a-th time step, such as: First, configuration association mining parameters can be determined. The configuration association mining parameters are formed by learning the mapping relationship between traffic scene image samples, radar point cloud data samples, and violation event labels, that is, representing the semantic information of samples and labels. Secondly, for the ath time step among the M time steps, when a is less than a first preset value, the configuration association mining parameter is used as a dynamic adjustment parameter for the ath time step. That is, in the early stage of the association semantic mining operation, the configuration association mining parameter can be directly used as a dynamic adjustment parameter. In this way, the semantic information in the sample and the label can be fully integrated; Then, when a is greater than or equal to the first preset value and less than the second preset value, the configuration association mining parameter and the first adjustment parameter are summed to form the dynamic adjustment parameter for the a-th time step. That is, in the middle of the association semantic mining operation, the configuration association mining parameter can be adjusted to a certain extent, so that the semantic information in the sample and the label is deformed to a certain extent. Therefore, when performing the association semantic mining operation, the degree of attention is reduced due to the reduction in correlation, thereby achieving attention to the channel splicing vector; Finally, when a is greater than or equal to the second preset value and less than or equal to M, the configuration association mining parameter and the second adjustment parameter are summed to form the dynamic adjustment parameter of the a-th time step, wherein the second adjustment parameter is greater than the first adjustment parameter. That is to say, in the later stage of the association semantic mining operation, the configuration association mining parameter can be adjusted to a large extent, so that the semantic information in the samples and labels is deformed to a large extent. Therefore, when performing the association semantic mining operation, the degree of attention is greatly reduced due to the significant reduction in correlation, thereby achieving further attention to the channel splicing vector.

[0062] For example, M can be equal to 10, based on which the first preset value can be equal to 3, and the second preset value can be equal to 8. Thus, in the first two time steps, the configuration association mining parameter can be directly used as a dynamic adjustment parameter. In the third to seventh time steps, the configuration association mining parameter can be increased to a certain extent. In the eighth to tenth time steps, the configuration association mining parameter can be increased to a greater extent. Furthermore, the first adjustment parameter can be greater than 0 and less than 0.5, and the second adjustment parameter can be greater than 0.5 and less than 1.

[0063] In the second part, it should be noted that for step S120, the specific process of performing cross-attention mapping on the traffic scene semantic vector and the radar point cloud semantic vector is not restricted and can be selected according to actual needs.

[0064] For example, in one feasible embodiment, cross-attention mapping can be implemented by a cross-attention network layer, which can have a first weight matrix, a second weight matrix, and a third weight matrix (formed by learning the mapping relationship between traffic scene image samples, radar point cloud data samples, and violation event labels). Then, the first weight matrix can be multiplied by the traffic scene semantic vector to form a first attention vector of the traffic scene semantic vector. Furthermore, the second weight matrix can be multiplied by the radar point cloud semantic vector to form a second attention vector of the radar point cloud semantic vector, and the third weight matrix can be multiplied by the radar point cloud semantic vector to form a third attention vector of the radar point cloud semantic vector.

[0065] In the third part, it should be noted that for step S130, the specific process of cross-attention fusion of the first attention vector of the traffic scene semantic vector, the second attention vector and the third attention vector of the radar point cloud semantic vector is not restricted and can be selected according to actual needs.

[0066] For example, in a feasible implementation, in order to mine and capture more fine-grained associated information, the above-mentioned step S130 may include the following: In the first step, the first attention vector of the traffic scene semantic vector can be segmented based on the length of the image masking parameter of the traffic scene image to form a local first attention vector (that is, each time a local vector with the same length as the image masking parameter is extracted from the first attention vector to obtain a corresponding local first attention vector, wherein two adjacent local first attention vectors may or may not intersect), and based on the point cloud masking parameter of the radar point cloud data, the second attention vector and the third attention vector of the radar point cloud semantic vector are segmented respectively to form a local second attention vector and a local third attention vector (that is, each Each time, a local vector of equal length to the point cloud masking parameter is extracted from the second attention vector and the third attention vector, and a corresponding local second attention vector and a local third attention vector are obtained, wherein the two adjacent local second attention vectors may or may not cross; the two adjacent local third attention vectors may or may not cross); in addition, the image masking parameter and the point cloud masking parameter are both matrices composed of 0 and 1, and are formed by learning the mapping relationship between traffic scene image samples, radar point cloud data samples and violation event labels. Initially, it can be a random matrix, an all-0 matrix, or an all-1 matrix; In the second step, the image masking parameter and the local first attention vector can be bitwise multiplied to form a local first adjustment vector (so that non-important semantic information in the local first attention vector can be masked), and the point cloud masking parameter is bitwise multiplied with the local second attention vector and the local third attention vector respectively to form a local second adjustment vector and a local third adjustment vector (so that non-important semantic information in the local second attention vector and the local third attention vector can be masked); In a third step, based on the attention parameter distribution between the local first adjustment vector and the local second adjustment vector (for example, the attention parameter distribution may be obtained by multiplying the transposed vectors of the local first adjustment vector and the local second adjustment vector), a weighted summation calculation may be performed on the local third adjustment vector to form a local attention fusion vector. In the second step, an attention fusion vector can be formed based on each local attention fusion vector; for example, each local attention fusion vector (a corresponding local attention fusion vector can be formed based on each segmentation) can be spliced, and then convolution and pooling can be performed to achieve compression, thereby obtaining an attention fusion vector.

[0067] For example, in another feasible implementation, in order to achieve global fusion so that the complementary information of image and point cloud data can be fully utilized through the cross-attention mechanism, the above-mentioned step S130 can further include step S131, step S132 and step S133, the specific contents of which are as follows.

[0068] Step S131 , fusing the image masking parameters of the traffic scene image and the point cloud masking parameters of the radar point cloud data to form fused masking parameters.

[0069] In an embodiment of the present invention, the image masking parameters of the traffic scene image and the point cloud masking parameters of the radar point cloud data can be fused to form fused masking parameters. Thus, the fused masking parameters can be used to jointly characterize the masking parameters of the two dimensions. The image masking parameters and the point cloud masking parameters are formed by learning the mapping relationship between traffic scene image samples, radar point cloud data samples, and violation event labels, i.e., they are formed as parameters in the corresponding neural network model during the model training process.

[0070] Step S132: Based on the fusion masking parameter, the first attention vector of the traffic scene semantic vector, the second attention vector and the third attention vector of the radar point cloud semantic vector are cross-attentively fused to form an attention fusion vector.

[0071] In an embodiment of the present invention, after forming the fusion masking parameters, the first attention vector of the traffic scene semantic vector and the second and third attention vectors of the radar point cloud semantic vector can be cross-attentively fused based on the fusion masking parameters to form an attention fusion vector. In other words, during the cross-attention fusion process, the fusion masking parameters are used to mask local attention information to increase focus on important semantic information, thereby enabling mining of important semantic information.

[0072] Step S133: Perform vector space mapping on the attention fusion vector to form a traffic scene fusion vector.

[0073] In an embodiment of the present invention, after forming the attention fusion vector, the attention fusion vector can be mapped into a vector space to form a traffic scene fusion vector. For example, the attention fusion vector can be processed through a fully connected network layer to capture more complex semantic information, thereby obtaining a corresponding traffic scene fusion vector.

[0074] Optionally, in the above step S131, the specific process of fusing the image masking parameters of the traffic scene image and the point cloud masking parameters of the radar point cloud data is not limited. For example, in a feasible embodiment, in order to fully utilize the masking parameters of the two dimensions to capture complex semantic information, the above step S131 may further include the following (combined with Figure 6 As shown, where black represents 0 and white represents 1): The first step is to perform a bitwise OR operation on the image masking parameters of the traffic scene image and the point cloud masking parameters of the radar point cloud data (i.e., if any one of the same position is 1, the result is 1) to form a first fused masking parameter, wherein the image masking parameters are a matrix composed of 0s and 1s, and the point cloud masking parameters are a matrix composed of 0s and 1s; The second step is to perform a bitwise AND operation on the image masking parameter of the traffic scene image and the point cloud masking parameter of the radar point cloud data (i.e., if any one of the same positions is 0, the result is 0) to form a second fused masking parameter; In the third step, the first fusion masking parameter and the second fusion masking parameter are used as fusion masking parameters, that is, the fusion masking parameters include the first fusion masking parameter and the second fusion masking parameter.

[0075] Optionally, in the above step S132, the specific process of cross-attention fusion of the first attention vector of the traffic scene semantic vector, the second attention vector of the radar point cloud semantic vector, and the third attention vector is not limited. For example, in a feasible implementation, as described above, the fusion masking parameter includes the first fusion masking parameter and the second fusion masking parameter. Based on this, the above step S132 may further include the following content: In the first step, a first attention vector of the traffic scene semantic vector and a second attention vector of the radar point cloud semantic vector may be matrix multiplied to form an attention parameter distribution; Secondly, the attention parameter distribution and the first fused masking parameter and the second fused masking parameter included in the fused masking parameter can be bitwise added to form a first adjustment parameter distribution and a second adjustment parameter distribution; that is, the attention parameter distribution and the first fused masking parameter can be bitwise added to form a first adjustment parameter distribution, and the attention parameter distribution and the second fused masking parameter can be bitwise added to form a second adjustment parameter distribution; Then, a weighted summation calculation can be performed on the third attention vector of the radar point cloud semantic vector based on the first adjustment parameter distribution and the second adjustment parameter distribution to form a first fusion vector and a second fusion vector; that is, a weighted summation calculation can be performed on the third attention vector based on the first adjustment parameter distribution to form a first fusion vector, and a weighted summation calculation can be performed on the third attention vector based on the second adjustment parameter distribution to form a second fusion vector; Finally, an attention fusion vector can be formed based on the first fusion vector and the second fusion vector. For example, the first fusion vector and the second fusion vector can be averaged or summed to form the attention fusion vector. Alternatively, the first fusion vector and the second fusion vector can be concatenated and then subjected to convolution, pooling and other processing to form the corresponding attention fusion vector.

[0076] In the fourth part, it should be noted that for step S140, the specific process of identifying illegal events based on the traffic scene fusion vector is not limited and can be selected according to actual conditions.

[0077] For example, in a feasible implementation, the traffic scene fusion vector can be fully connected so that the traffic scene fusion vector is mapped into a fully connected vector of size 1*b, where b is equal to the number of specific types of traffic violation situations (for example, when it is necessary to identify whether there is a traffic violation situation, b is equal to 2). Then, the fully connected vector can be mapped (such as through functions such as softmax) to obtain a probability distribution, in which each value in the probability distribution is used to represent the probability of a traffic violation situation. Then, the traffic violation situation with the highest probability can be used as the violation event recognition result of the target traffic scene.

[0078] To summarize, the traffic violation event recognition method and system based on fused data provided by the present invention first mine the traffic scene semantic vector corresponding to the traffic scene image and the radar point cloud semantic vector corresponding to the radar point cloud data; secondly, the traffic scene semantic vector and the radar point cloud semantic vector are cross-attention mapped to form a first attention vector, a second attention vector and a third attention vector; then, based on the image masking parameters and the point cloud masking parameters, the first attention vector, the second attention vector and the third attention vector are cross-attention fused to form a traffic scene fusion vector; finally, violation event recognition is performed based on the traffic scene fusion vector to form a violation event recognition result. Based on the above method, since corresponding masking will be performed based on the image masking parameters and the point cloud masking parameters during the cross-attention fusion process, the cross-fusion of the corresponding semantic vectors can be effectively controlled, thereby achieving attention to the effective semantic information in the corresponding semantic vectors and avoiding interference from invalid semantic information, so that the semantic representation of the formed traffic scene fusion vector is more accurate, thereby ensuring the reliability of the violation event recognition results obtained based on the traffic scene fusion vector. Compared with conventional technical means such as using data from a single data source for recognition analysis or simply fusing data from multiple data sources, it has higher reliability. Therefore, it can improve the problem of relatively low reliability of traffic violation event recognition in the existing technology.

[0079] In the several embodiments provided in the embodiments of the present invention, it should be understood that the disclosed devices and methods can also be implemented in other ways. The device and method embodiments described above are merely illustrative. For example, the flowcharts and block diagrams in the accompanying drawings show the possible architectures, functions, and operations of the devices, methods, and computer program products according to multiple embodiments of the present invention. In this regard, each box in the flowchart or block diagram can represent a module, program segment, or part of the code, which contains one or more executable instructions for implementing the specified logical functions. It should also be noted that in some alternative implementations, the functions marked in the boxes can also occur in an order different from that marked in the accompanying drawings. For example, two consecutive boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flowchart, as well as the combination of boxes in the block diagram and / or flowchart, can be implemented using a dedicated hardware-based system that performs the specified functions or actions, or can be implemented using a combination of dedicated hardware and computer instructions.

[0080] In addition, the functional modules in the various embodiments of the present invention may be integrated together to form an independent part, or each module may exist independently, or two or more modules may be integrated to form an independent part.

[0081] If the functions are implemented in the form of software modules and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the portion that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions for causing a computer device (which can be a personal computer, electronic device, or network device, etc.) to perform all or part of the steps of the methods described in various embodiments of the present invention. The aforementioned storage media include various media that can store program code, such as USB flash drives, mobile hard drives, read-only memories (ROMs), random access memories (RAMs), magnetic disks, or optical disks. It should be noted that, in this document, the terms "comprise," "include," or any other variations thereof are intended to encompass non-exclusive inclusion, such that a process, method, article, or device that includes a series of elements includes not only those elements but also other elements not explicitly listed, or also includes elements inherent to such process, method, article, or device. Without further constraints, an element defined by the phrase "comprises a..." does not preclude the existence of additional identical elements in the process, method, article or apparatus that includes the element.

[0082] The foregoing description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Those skilled in the art will readily appreciate that various modifications and variations of the present invention are possible. Any modifications, equivalent substitutions, or improvements made within the spirit and principles of the present invention are intended to be within the scope of protection of the present invention.

Claims

1. A traffic violation event identification method based on fusion data, characterized in that: include: Mining a traffic scene semantic vector corresponding to a traffic scene image of a target traffic scene and a radar point cloud semantic vector corresponding to radar point cloud data, wherein the radar point cloud data is formed by collecting information of the target traffic scene through a millimeter-wave radar; Performing cross-attention mapping on the traffic scene semantic vector and the radar point cloud semantic vector respectively to form a first attention vector of the traffic scene semantic vector, a second attention vector and a third attention vector of the radar point cloud semantic vector; Based on the image masking parameter of the traffic scene image and the point cloud masking parameter of the radar point cloud data, cross-attention fusion is performed on the first attention vector of the traffic scene semantic vector, the second attention vector and the third attention vector of the radar point cloud semantic vector to form a traffic scene fusion vector; Illegal events are identified based on the traffic scene fusion vector to form an illegal event identification result of the target traffic scene.

2. The traffic violation event identification method based on fusion data according to claim 1, characterized in that: The step of mining the traffic scene semantic vector corresponding to the traffic scene image of the target traffic scene and the radar point cloud semantic vector corresponding to the radar point cloud data includes: Performing a two-dimensional Fourier transform on a traffic scene image of a target traffic scene to form a traffic scene spectrogram, and performing a deep convolution on the traffic scene spectrogram to form a traffic scene semantic vector corresponding to the traffic scene image; Expanding discrete point cloud data in the radar point cloud data of the target traffic scene into a continuous three-dimensional grid to form three-dimensional radar point cloud data, wherein in the three-dimensional radar point cloud data, data at a position corresponding to the point cloud data is 1, and data at other positions is 0; Performing a three-dimensional Fourier transform on the three-dimensional radar point cloud data to form a radar point cloud spectrum graph; Deep convolution is performed on the radar point cloud spectrogram to form a radar point cloud semantic vector corresponding to the radar point cloud data.

3. The traffic violation event identification method based on fusion data according to claim 2, characterized in that: The step of performing deep convolution on the radar point cloud spectrogram to form a radar point cloud semantic vector corresponding to the radar point cloud data includes: Performing channel-wise convolution on the radar point cloud spectrum to form multiple channel convolution vectors; Splicing the multiple channel convolution vectors to form a channel splicing vector; According to the dynamic adjustment parameters, the channel splicing vector is subjected to an association semantic mining operation for M time steps to form a radar point cloud semantic vector, wherein the dynamic adjustment parameters are used to adjust the association mining parameters at different time steps, and the association mining parameters are used to adjust the mined association semantic distribution in the association semantic mining operation.

4. The method for identifying traffic violations based on fused data according to claim 3, characterized in that: The step of performing an association semantic mining operation on the channel splicing vector for M time steps based on the dynamic adjustment parameters to form a radar point cloud semantic vector includes: For the a-th time step among the M time steps, concatenate the associated semantic vector of the a-1-th time step and the dynamic adjustment parameter of the a-th time step to form the semantic vector to be mined of the a-th time step, where a is a positive integer less than or equal to M. When a=1, the associated semantic vector of the a-1-th time step is the channel splicing vector; Performing an associated semantic mining operation on the semantic vector to be mined at the a-th time step to form an associated semantic vector at the a-th time step, wherein the associated semantic mining operation sequentially includes an attention operation, a first vector compression operation, a nonlinear mapping operation, and a second vector compression operation; If a is less than M, then a=a+1 is determined, and the step of concatenating the associated semantic vector of the a-1th time step and the dynamic adjustment parameter of the ath time step for the ath time step among the M time steps is performed in reverse order to form the semantic vector to be mined of the ath time step; If a is equal to M, the associated semantic vector of the a-th time step is determined to be the radar point cloud semantic vector corresponding to the radar point cloud data.

5. The method for identifying traffic violations based on fused data according to claim 4, characterized in that: The step of performing an association semantic mining operation on the channel splicing vector for M time steps based on the dynamic adjustment parameter to form a radar point cloud semantic vector also includes the step of determining the dynamic adjustment parameter for the ath time step, which step includes: Determining configuration association mining parameters, wherein the configuration association mining parameters are formed by learning a mapping relationship between traffic scene image samples, radar point cloud data samples, and violation event labels; For an a-th time step among the M time steps, when a is less than a first preset value, using the configuration association mining parameter as a dynamic adjustment parameter for the a-th time step; When a is greater than or equal to the first preset value and less than a second preset value, performing a sum calculation on the configuration association mining parameter and the first adjustment parameter to form the dynamic adjustment parameter of the a-th time step; When a is greater than or equal to the second preset value and less than or equal to M, the configuration association mining parameter and the second adjustment parameter are summed to form the dynamic adjustment parameter of the ath time step, wherein the second adjustment parameter is greater than the first adjustment parameter.

6. The method for identifying traffic violations based on fused data according to any one of claims 1 to 5, characterized in that: The step of performing cross-attention fusion on the first attention vector of the traffic scene semantic vector, the second attention vector and the third attention vector of the radar point cloud semantic vector based on the image masking parameter of the traffic scene image and the point cloud masking parameter of the radar point cloud data to form a traffic scene fusion vector includes: fusing image masking parameters of the traffic scene image and point cloud masking parameters of the radar point cloud data to form fused masking parameters, wherein the image masking parameters and the point cloud masking parameters are formed by learning a mapping relationship between traffic scene image samples, radar point cloud data samples, and violation event labels; Based on the fusion masking parameter, cross-attention-fusing the first attention vector of the traffic scene semantic vector, the second attention vector and the third attention vector of the radar point cloud semantic vector to form an attention fusion vector; The attention fusion vector is mapped into a vector space to form a traffic scene fusion vector.

7. The method for identifying traffic violations based on fusion data according to claim 6, characterized in that: The step of fusing the image masking parameters of the traffic scene image and the point cloud masking parameters of the radar point cloud data to form fused masking parameters includes: Performing a bitwise OR operation on an image masking parameter of the traffic scene image and a point cloud masking parameter of the radar point cloud data to form a first fused masking parameter, wherein the image masking parameter is a matrix composed of 0s and 1s, and the point cloud masking parameter is a matrix composed of 0s and 1s; performing a bitwise AND operation on the image masking parameter of the traffic scene image and the point cloud masking parameter of the radar point cloud data to form a second fused masking parameter; The first fusion masking parameter and the second fusion masking parameter are used as fusion masking parameters.

8. The method for identifying traffic violations based on fusion data according to claim 6, characterized in that: The step of performing cross-attention fusion on the first attention vector of the traffic scene semantic vector, the second attention vector and the third attention vector of the radar point cloud semantic vector based on the fusion masking parameter to form an attention fusion vector includes: Performing matrix multiplication on a first attention vector of the traffic scene semantic vector and a second attention vector of the radar point cloud semantic vector to form an attention parameter distribution; Performing a bitwise addition operation on the attention parameter distribution and the first fused masking parameter and the second fused masking parameter included in the fused masking parameter to form a first adjustment parameter distribution and a second adjustment parameter distribution; performing a weighted sum calculation on the third attention vector of the radar point cloud semantic vector based on the first adjustment parameter distribution and the second adjustment parameter distribution, respectively, to form a first fusion vector and a second fusion vector; An attention fusion vector is formed based on the first fusion vector and the second fusion vector.

9. The method for identifying traffic violations based on fused data according to any one of claims 1 to 5, characterized in that: The step of performing cross-attention fusion on the first attention vector of the traffic scene semantic vector, the second attention vector and the third attention vector of the radar point cloud semantic vector based on the image masking parameter of the traffic scene image and the point cloud masking parameter of the radar point cloud data to form a traffic scene fusion vector includes: Segmenting the first attention vector of the traffic scene semantic vector based on the length of the image masking parameter of the traffic scene image to form a local first attention vector, and segmenting the second attention vector and the third attention vector of the radar point cloud semantic vector based on the point cloud masking parameter of the radar point cloud data to form a local second attention vector and a local third attention vector; Performing a bitwise multiplication operation on the image masking parameter and the local first attention vector to form a local first adjustment vector, and performing a bitwise multiplication operation on the point cloud masking parameter and the local second attention vector and the local third attention vector respectively to form a local second adjustment vector and a local third adjustment vector; performing a weighted sum calculation on the local third adjustment vector based on the attention parameter distribution between the local first adjustment vector and the local second adjustment vector to form a local attention fusion vector; Based on each local attention fusion vector, an attention fusion vector is formed.

10. A traffic violation event recognition system based on fusion data, characterized in that: It includes a processor and a memory, the memory is used to store a computer program, and the processor is used to execute the computer program to implement the traffic violation event identification method based on fusion data as described in any one of claims 1 to 9.

Images