Image library construction and recognition method under large-scale incomplete multi-view and multi-modal scene

Abstract

The present application relates to the technical field of behavior recognition, in particular to a large-scale incomplete multi-view, multi-modal scene image library construction and recognition method, comprising the following steps: deploying multiple sensors and cameras in a traffic scene, and unifying multi-modal data to the same space-time coordinate system; fusing the multi-modal data after space-time alignment, predicting the target state and updating the measurement value; based on the fused data, using the adversarial generative network under space-time constraints, dynamically generating multi-view candidate views, combining the target kinematic model to complete the missing views; fusing the target kinematic model and deep learning technology to track the cross-camera target, constructing the traffic regulation ontology library, and combining the image recognition technology to automatically identify and classify the traffic violation behavior. The present application ensures the consistency of different sensor data in time and space dimensions, avoiding data confusion and errors caused by space-time differences.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of behavior recognition, and in particular to an image library construction and recognition method in a large-scale incomplete multi-view and multi-modal scene. BACKGROUND

[0002] With the acceleration of urbanization and the rapid growth of traffic flow, traffic law enforcement is facing unprecedented challenges. The modern urban traffic network is complex and the traffic flow is huge. The traditional monitoring means is difficult to achieve comprehensive coverage of all road sections, especially in some key areas where traffic violations occur frequently. More efficient monitoring means are needed in the process of traffic law enforcement. In traffic law enforcement, data from different perspectives and different modalities often have relevance and complementarity. How to effectively integrate and utilize these data has become a difficult problem.

[0003] Traditional traffic law enforcement methods, such as manual patrol and on-site law enforcement, have been difficult to meet the needs of modern traffic management, especially in the incomplete multi-view and multi-modal scene. The identification, evidence collection and triggering of traffic violations become particularly complex. SUMMARY

[0004] The present application provides an image library construction and recognition method in a large-scale incomplete multi-view and multi-modal scene to solve the technical problems in the prior art.

[0005] The technical solution of the present application to solve the above technical problems is as follows: an image library construction and recognition method in a large-scale incomplete multi-view and multi-modal scene, specifically comprising the following steps:

[0006] S101, deploying multiple sensors and cameras in a traffic scene, collecting their position, orientation and time synchronization information, using these information to construct a space-time alignment matrix, and through matrix operation, unifying multi-modal data into the same space-time coordinate system;

[0007] S102, fusing the multi-modal data after space-time alignment, predicting the target state and updating the measurement value, fusing different sensor data, and enhancing the integrity of image information;

[0008] S103, based on the fused data, using an adversarial generative network under space-time constraints, dynamically generating multi-view candidate views, combining a target kinematic model, simulating the appearance and motion state of the target under different perspectives, and completing the missing views;

[0009] S104, fusing the target kinematic model and deep learning technology, tracking the target across cameras, constructing a traffic regulation ontology library, and combining image recognition technology, automatically identifying and classifying traffic violations;

[0010] In a preferred embodiment, in S101, the geographical position information of each sensor and camera is acquired using GPS, including longitude, latitude and altitude, which is recorded and associated with the corresponding device. Network time protocol servers are equipped for all sensors and cameras to ensure time synchronization between devices and record the timestamp information of each device. For each sensor and camera, the conversion parameters of its own coordinate system to the unified coordinate system O-XYZ are calculated according to its position and orientation information, including translation vector and rotation matrix.

[0011] The translation vector represents the position offset of the sensor and camera in the unified coordinate system. The position coordinates of the sensor and camera are subtracted from the origin coordinates of the unified coordinate system to obtain the translation vector. The calculation of the translation vector is based on the origin coordinates O = (O x , O y , O z ) of the unified coordinate system O-XYZ, the local coordinate system o-xyz of the sensor and camera, and the coordinates of the origin of the sensor and camera in the unified coordinate system P = (P x , P y , P z ). The calculation formula of the translation vector is as follows:

[0012]

[0013] Where T represents the position offset from the origin of the unified coordinate system to the origin of the local coordinate system. T x represents the position offset of the sensor and camera in the unified coordinate system along the x-axis direction relative to the origin, T y represents the position offset of the sensor and camera in the unified coordinate system along the y-axis direction relative to the origin, and T z represents the position offset of the sensor and camera in the unified coordinate system along the z-axis direction relative to the origin. In actual application, if the origin of the unified coordinate system is located at (0, 0, 0), then the translation vector is directly equal to the coordinates of the origin of the sensor and camera in the unified coordinate system, i.e. T = (P x , P y , P z ).

[0014] The rotation matrix represents the angle relationship between the orientation of the sensor and camera and the coordinate axes of the unified coordinate system. According to the horizontal and vertical angles of the sensor and camera, the rotation matrix is obtained using the calculation formula of the rotation matrix. The rotation matrix is used to describe the rotation relationship of the local coordinate system relative to the unified coordinate system, and the rotation angles are respectively the yaw angle ψ, the pitch angle θ and the roll angle φ.

[0015] The translation vector and rotation matrix of each sensor and camera are combined into a 4x4 homogeneous transformation matrix as the space-time alignment matrix of the device, and the general form of the homogeneous transformation matrix is:

[0016]

[0017] wherein R 3×3 represents a rotation matrix, T 3×1 represents a translation vector, 0 1×3 is a 1x3 zero vector.

[0018] Start all sensors and cameras to start collecting multi-modal data, including images, radar data, speed data, while recording the timestamp and corresponding device information at each time point, for each collected data point, use the corresponding space-time alignment matrix to perform coordinate conversion, convert it from the device coordinate system to the unified coordinate system, multiply the coordinate vector of the data point with the space-time alignment matrix to obtain the coordinates in the unified coordinate system of the device, according to the timestamp information of each time point, time align the multi-modal data, use linear interpolation method to unify the data collected by different devices at different times to the same time point, assuming that there are two adjacent time points t1 and t2 when the device is collecting data, the corresponding multi-modal data values are v1 and v2, we want to get the difference data value v at time point t, then the specific calculation formula is as follows:

[0019]

[0020] wherein v represents the difference data value at time point t, v1 represents the multi-modal data value obtained by the sensor and other data collection devices at time point t1, and v2 represents the multi-modal data value obtained by the sensor and other data collection devices at time point t2.

[0021] In a preferred embodiment, in S102, the multi-modal data collected by each sensor and camera is represented in homogeneous coordinate form, and the data is converted from the local coordinate system of the sensor and camera to the unified space-time coordinate system through matrix multiplication operation, according to the motion characteristics of the target in the traffic scene, the state variable is defined, and Kalman filter is applied for data fusion, which includes the following steps:

[0022] S1, define state variables: the state variables of the vehicle target include its position (x, y, z) in the unified space-time coordinate system, speed (v x , v y , v z ), acceleration (a x , a y , a z ), which form a state vector S1, define state variables: according to the motion law of the vehicle target, the state transition equation is established:

[0023]

[0024] wherein, represents the state vector of the vehicle target at time k, F k represents the state transition matrix, which describes the change relationship of the state of the vehicle target from k-1 time to k time, represents the process noise;

[0025] S2, initialize Kalman filter parameters: according to the initial measurement value of a sensor in multi-modal data, the initial state of the vehicle target is estimated, the uncertainty of the initial state estimation is described by the initial covariance matrix P0, and the process noise covariance matrix Q is set to quantify the statistical characteristics of the process noise ;

[0026] S3, prediction: according to the state transition equation and the vehicle target state estimation at the last time, the state prediction value of the vehicle target at the current time is calculated The covariance prediction value at the current time is calculated wherein, represents the prediction of the uncertainty of the state estimation at the current time before new measurement data is obtained;

[0027] S4, update: pre-process each sensor measurement value in the spatio-temporally aligned multi-modal data, including removing noise, outlier processing and data format conversion operation, and then combine these measurement values into a measurement vector According to the characteristics and measurement principle of the sensor, the measurement equation is established wherein, H k represents the measurement matrix, which maps the state vector to the measurement space, represents the measurement noise, according to the covariance prediction value measurement matrix H k and measurement noise covariance matrix R k , calculate the Kalman gain Use the measurement vector Kalman gain K k and the vehicle target state prediction value update the state estimation value at the current time According to the Kalman gain K k and the covariance prediction value update the covariance matrix at the current time wherein I denotes an identity matrix;

[0028] S5, output the updated state estimation value As the fused vehicle target state, it is used for subsequent view generation, target tracking, behavior analysis processing, and the covariance matrix P k is saved as the initial covariance matrix of the next time Kalman filtering, and the prediction and update steps are repeatedly performed to realize continuous fusion of multi-modal data.

[0029] In a preferred embodiment, in S103, the multi-modal data fused by Kalman filtering, including the position, speed and acceleration information of the vehicle target, are used as inputs, the generator generates candidate views under different perspectives according to the inputs, multiple views are generated to cover different observation angles, a network structure is constructed using convolutional layers, deconvolutional layers and fully connected layers, the convolutional layers are used to extract features, the deconvolutional layers are used to generate images, and the fully connected layers are used to integrate information, the discriminator receives the candidate views generated by the generator and the real multi-perspective images, extracts features and classifies the received images, uses the target kinematic model to predict the future state of the target under different perspectives according to the current vehicle target state and historical motion information, if the current position of the target is (x1, y1), the speed is (vx1, vy1), the acceleration is (ax1, ay1), and the time interval is Δt, the position (x2, y2) at the next time can be calculated by the following formula:

[0030]

[0031] wherein (x2, y2) represents the position coordinates of the target at the next time, the predicted target state is input into the trained generator to generate candidate views under different perspectives, for the missing views, a linear interpolation method is used for inference and completion, assuming that the target states under adjacent perspectives δ1 and δ2 (x1, y1) and (x2, y2) are known, the calculation formula of the target state under perspective δ to be inferred is as follows:

[0032]

[0033] wherein (B x ,B y ) represents the position coordinates of the target under perspective δ calculated by the linear interpolation method, according to the inferred target state, the generator is used to generate the corresponding candidate view, and the view library is completed, the screened multi-perspective candidate views are stored in the database, and an index is established for each candidate view, the index information includes view shooting time, perspective parameter, target category and position information.

[0034] In a preferred embodiment, in S104, a target monitoring model based on a convolutional neural network is used. A large amount of multi-view image data containing various types of targets is collected, and the targets are accurately labeled. The labeling information includes the target category and the location box. The labeled data is used to train the convolutional neural network model. The model parameters are continuously adjusted through the back propagation algorithm. The trained deep learning model is used to detect targets in images captured by each camera. For the detected targets, the features of the targets and the location information in the image are compared, and a data association algorithm is used to associate the same target captured by different cameras. A unique identifier is established for each target. The predicted position of the target kinematic model is fused with the actual position of the target detected by the deep learning model. When identifying the target, the target image T A is matched with the images in the multi-view candidate view library T B . The similarity between the images is calculated to retrieve the most similar candidate view to the current target image. The specific calculation formula is as follows:

[0035]

[0036] where sim represents the similarity, m represents the dimension of the feature vector, and represent the i-th element of the feature vector T A and T B , respectively. Comprehensive collection of traffic regulations, extraction of key concepts from regulation clauses, including elements of illegal behavior and judgment criteria, use of different traffic violations, use of deep learning models to extract corresponding image features, collection of video and image data containing traffic violations, labeling of collected data, marking different types of illegal behavior and related features, including signal light status, vehicle position and vehicle speed, inputting multi-view candidate views and real-time monitoring images, using multiple convolutional layers and pooling layers to extract spatial features of the image, flattening the output of the convolutional layer and connecting it to the fully connected layer for classifying the signal light status, outputting the signal light status and the relative position relationship of the vehicle, integrating the trained model into the actual traffic identification, and real-time identification of traffic violations.

[0037] The beneficial effects of the present application are: the present application collects the position, orientation and time synchronization information of the sensor and camera, constructs a space-time alignment matrix, unifies the multi-modal data to the same space-time coordinate system, ensures the consistency of different sensor data in time and space dimensions, provides a solid foundation for subsequent accurate processing and analysis of data, avoids data confusion and errors caused by space-time differences, applies Kalman filtering algorithm to fuse the multi-modal data after space-time alignment, can effectively process the noise and uncertainty in the data, not only enhances the accuracy and integrity of image information, but also improves the quality of subsequent view generation, makes the generated image closer to the real scene, is conducive to more accurately identifying targets and behaviors, dynamically generates multi-view candidate views using the adversarial generative network under the space-time constraint, and combines the target kinematic model to complete the missing views, while screening high-quality candidate views to construct a multi-view candidate view library, which greatly enriches the content of the image library, which makes the image library can cover a wider range of perspectives and scenes, helps to improve the identification accuracy of the target under different views, reduces the identification error caused by the lack of perspective, fuses the target kinematic model and deep learning technology, uses a deep learning model to extract target features, combines the kinematic model to predict the target trajectory, and realizes accurate tracking of cross-camera targets, based on the multi-view candidate view library, provides more abundant information for target identification, so that the system can observe the target from multiple angles, thereby improving the identification accuracy of the target under different views. BRIEF DESCRIPTION OF DRAWINGS

[0038] Figure 1 Flowchart of the present application; DETAILED DESCRIPTION

[0039] The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor are within the scope of protection of the present application.

[0040] In the description of the present application, the terms "first", "second" are used only for description purposes, and cannot be understood as indicating or implying relative importance or implicitly indicating the number of indicated technical features. Therefore, the features defined with "first", "second" can explicitly or implicitly include one or more of the features. In the description of the present application, the meaning of "multiple" is two or more, unless otherwise specifically limited.

[0041] In the description of the present application, the term "for example" is used to indicate "serving as an example, instance, or illustration." Any embodiment described as "for example" in this application is not necessarily to be construed as preferred or advantageous over other embodiments. The following description is presented to enable any person skilled in the art to make and use the application. In the following description, for the purpose of explanation, details are set forth. It should be appreciated that one of ordinary skill in the art can realize and implement the application without using these specific details. In other instances, well-known structures and processes are not elaborated in detail in order not to obscure the description of the application with unnecessary details. Therefore, the present application is not intended to be limited to the embodiments shown, but is to be accorded the widest scope consistent with the principles and features disclosed herein.

[0042] As Figure 1 The embodiment provides a large-scale incomplete multi-view and multi-modal scene image library construction and recognition method, and specifically comprises the following steps:

[0043] S101, deploy multiple sensors and cameras in a traffic scene, collect position, orientation and time synchronization information of the sensors and cameras, use the information to construct a space-time alignment matrix, and perform matrix operation to unify the multi-modal data into a same space-time coordinate system;

[0044] Further, the geographic position information of each sensor and camera is acquired by using a GPS, including longitude, latitude and altitude, the information is recorded and associated with the corresponding device, a network time protocol server is provided for all sensors and cameras to ensure time synchronization between the devices, and the time stamp information of each device is recorded, and for each sensor and camera, the conversion parameters of its own coordinate system to the unified coordinate system O-XYZ are calculated according to the position and orientation information of the sensor and camera, the conversion parameters including a translation vector and a rotation matrix;

[0045] The translation vector represents the position offset of the sensor and camera in the unified coordinate system, and the position coordinates of the sensor and camera are subtracted from the origin coordinates of the unified coordinate system to obtain the translation vector, the calculation of the translation vector is based on the origin coordinates O=(O x ,O y ,O z ) of the unified coordinate system O-XYZ, the local coordinate system o-xyz of the sensor and camera, and the coordinates of the origin of the sensor and camera in the unified coordinate system are P=(P x ,P y ,P z ), and the calculation formula of the translation vector is as follows:

[0046]

[0047] Wherein, T is used to describe the position offset from the origin of the unified coordinate system to the origin of the local coordinate system, T x represents the position offset of the sensor and the camera relative to the origin along the x-axis direction in the unified coordinate system, T y represents the position offset of the sensor and the camera relative to the origin along the y-axis direction in the unified coordinate system, T z represents the position offset of the sensor and the camera relative to the origin along the z-axis direction in the unified coordinate system, in actual application, such as the origin of the unified coordinate system is located at (0, 0, 0), then the translation vector is directly equal to the coordinates of the origin of the sensor and the camera in the unified coordinate system, that is, T = (P x , P y , P z );

[0048] The rotation matrix represents the angle relationship between the orientation of the sensor and the camera and the coordinate axis of the unified coordinate system, according to the horizontal and vertical angles of the sensor and the camera, using the calculation formula of the rotation matrix, the rotation matrix is obtained, the rotation matrix is used to describe the rotation relationship of the local coordinate system relative to the unified coordinate system, and the rotation angles are respectively the yaw angle ψ, the pitch angle θ and the roll angle φ;

[0049] The translation vector and the rotation matrix of each sensor and camera are combined into a 4 × 4 homogeneous transformation matrix, which is used as the space-time alignment matrix of the device, and the general form of the homogeneous transformation matrix is:

[0050]

[0051] Wherein, R 3×3 represents the rotation matrix, T 3×1 represents the translation vector, and 0 1×3 is a 1 × 3 zero vector.

[0052] Start all sensors and cameras to start collecting multi-modal data, including images, radar data, speed data, while recording the time stamp and corresponding device information of each time point, for each collected data point, use the corresponding space-time alignment matrix to perform coordinate conversion, and convert it from the device coordinate system to the unified coordinate system, multiply the coordinate vector of the data point with the space-time alignment matrix to obtain the coordinates in the unified coordinate system of the device, according to the time stamp information of each time point, time align the multi-modal data, adopt linear interpolation method to unify the data collected by different devices at different time points to the same time point, assuming that there are two adjacent time points t1 and t2 when the device is collecting data, the corresponding multi-modal data values are v1 and v2, and we want to obtain the difference data value v at time point t, then the specific calculation formula is as follows:

[0053]

[0054] wherein v represents the difference data value at time point t, v1 represents the multi-modal data value acquired by the sensor and other data acquisition devices at time point t1, and v2 represents the multi-modal data value acquired by the sensor and other data acquisition devices at time point t2.

[0055] It should be noted that the coordinate conversion has a data point in the local coordinate system of the device, and the homogeneous coordinates thereof are represented by a vector containing four elements, which are the coordinate values of the data point in the x-axis direction, the y-axis direction, the z-axis direction of the local coordinate system and the fixed bit 1, respectively. When data conversion from the coordinate system of the device is to be performed, the vector representing the homogeneous coordinates of the data point needs to be multiplied by the space-time alignment matrix.

[0056] It should be noted that the rotation matrix R Z (ψ) is:

[0057]

[0058] The rotation matrix R Y (θ) is:

[0059]

[0060] The rotation matrix R X (φ) is:

[0061]

[0062] S102, the multi-modal data after space-time alignment is fused, the target state is predicted and the measurement value is updated, different sensor data is fused, and the integrity of image information is enhanced;

[0063] Further, for the multi-modal data collected by each sensor and camera, the multi-modal data is represented in the form of homogeneous coordinates, the data is converted from the local coordinate system of the sensor and the camera to the unified space-time coordinate system through matrix multiplication operation, the state variable is defined according to the motion characteristics of the target in the traffic scene, the Kalman filter is applied for data fusion, and the specific steps include the following steps:

[0064] S1, defining a state variable: the state variable of the vehicle target includes its position (x, y, z) in the unified space-time coordinate system, velocity (v x , v y , v z ), acceleration (a x , a y , a z ), and a state vector S1, define state variables: according to the motion law of the vehicle target, establish the state transition equation:

[0065]

[0066] wherein, represents the state vector of the vehicle target at time k, F k represents the state transition matrix, which describes the change relationship of the state of the vehicle target from k-1 time to k time, represents the process noise;

[0067] S2, initialize Kalman filter parameters: according to the initial measurement value of a sensor in multi-modal data, estimate the initial state of the vehicle target , describe the uncertainty of the initial state estimation through the initial covariance matrix P0, set the process noise covariance matrix Q to quantify the statistical characteristics of the process noise ;

[0068] S3, prediction: according to the state transition equation and the vehicle target state estimation at the last time, calculate the state prediction value of the vehicle target at the current time Calculate the covariance prediction value at the current time wherein, represents the prediction of the uncertainty of the state estimation at the current time before new measurement data;

[0069] S4, update: pre-process each sensor measurement value in the spatio-temporally aligned multi-modal data, including removing noise, outlier processing and data format conversion operation, and then combine these measurement values into a measurement vector According to the characteristics and measurement principle of the sensor, establish the measurement equation wherein, H k represents the measurement matrix, which maps the state vector to the measurement space, represents the measurement noise, according to the covariance prediction value measurement matrix H k and measurement noise covariance matrix R k , calculate the Kalman gain Use the measurement vector Kalman gain K k and vehicle target state prediction value update the state estimation value at the current time According to the Kalman gain K k and the covariance prediction value update the covariance matrix at the current time wherein I represents an identity matrix;

[0070] S5, output the updated state estimation value As the fused vehicle target state, it is used for subsequent view generation, target tracking, behavior analysis processing, and the covariance matrix P k is saved as the initial covariance matrix of Kalman filtering at the next moment, and the prediction and update steps are repeatedly performed to realize continuous fusion of multi-modal data.

[0071] S103, based on the fused data, a multi-view candidate view is dynamically generated using the adversarial generation network under the spatio-temporal constraint, and the appearance and motion state of the target under different viewing angles are simulated by combining the target kinematic model to complete the missing view;

[0072] Further, using the multi-modal data fused by Kalman filtering, including the position, velocity and acceleration information of the vehicle target, the fused data and the random noise vector are used as input, the generator generates candidate views under different viewing angles according to the input, multiple views are generated to cover different observation angles, and the network structure is constructed using convolutional layers, deconvolutional layers and fully connected layers. The convolutional layer is used to extract features, the deconvolutional layer is used to generate images, and the fully connected layer is used to integrate information. The discriminator receives the candidate views generated by the generator and the real multi-view images, extracts features and classifies the received images. Using the target kinematic model, the future state of the target under different viewing angles is predicted according to the current vehicle target state and the historical motion information. If the current position of the target is (x1, y1), the velocity is (vx1, vy1), the acceleration is (ax1, ay1), and the time interval is Δt, the position (x2, y2) at the next moment can be calculated by the following formula:

[0073]

[0074] wherein (x2, y2) represents the position coordinates of the target at the next moment, the predicted target state is input into the trained generator to generate candidate views under different viewing angles, and for the missing views, a linear interpolation method is used to infer and complete. Assuming that the target states under adjacent viewing angles δ1 and δ2 (x1, y1) and (x2, y2) are known, the calculation formula of the target state under the viewing angle δ is as follows:

[0075]

[0076] wherein (B x ,B yThe coordinates of the target at the viewpoint δ are calculated by linear interpolation. Based on the inferred target state, a corresponding candidate view is generated by a generator to complete the view library. The selected multi-view candidate views are stored in the database, and an index is created for each candidate view. The index information includes the view capture time, viewpoint parameters, target category, and location information.

[0077] By combining S101 to S103 to construct an image library, and using a multi-sensor spatiotemporal alignment matrix, data from different coordinate systems are converted into a unified world coordinate system, achieving time synchronization of multimodal data, reducing spatial coordinate transformation errors, and solving the spatiotemporal heterogeneity problem of multi-sensor data fusion. The Kalman filter algorithm is used to perform optimal estimation of multi-source data, reducing the error of target position measurement. Virtual view images are generated through spatiotemporal constraints, achieving monitoring capabilities without blind spots. Spatiotemporal joint completion is performed based on kinematic models and linear interpolation methods, making up for the lack of missing view completion in traditional monitoring, and providing a reliable data foundation for subsequent target tracking and behavior recognition.

[0078] S104. Integrating target kinematics models with deep learning technology, the system tracks targets across cameras, constructs a traffic regulations ontology, and combines image recognition technology to automatically identify and classify traffic violations.

[0079] Furthermore, based on a convolutional neural network-based target detection model, a large amount of multi-view image data containing various targets is collected. Targets are accurately labeled, with labeling information including target category and bounding box. The labeled data is used to train the convolutional neural network model, continuously adjusting model parameters through backpropagation. The trained deep learning model is then used to detect targets in images captured by various cameras. For detected targets, by comparing target features and positional information in the images, a data association algorithm is used to associate the same target captured by different cameras, establishing a unique identifier for each target. The position predicted by the target kinematics model is fused with the actual target position detected by the deep learning model. When identifying a target, the target image T captured by the current camera is used as the basis for the identification. A Image T from the multi-view candidate view library B The matching process involves calculating the similarity between images to retrieve the candidate view that is most similar to the current target image. The specific calculation formula is as follows:

[0080]

[0081] Where sim represents the similarity, and m represents the dimension of the feature vector. and T represents the eigenvectors respectively. A and T BThe i-th element of the first element, comprehensively collect traffic regulations, extract key concepts from the provisions of the regulations, contain the constituent elements of illegal behavior and judgment criteria, use different traffic violations, use deep learning model to extract corresponding image features, collect video and image data containing traffic violations, label the collected data, mark different types of illegal behavior and related features, including signal light state, vehicle position and vehicle speed, input multi-view candidate view and real-time monitoring image, use multiple convolution layers and pooling layers to extract spatial features of the image, flatten the output of the convolution layer and connect to the fully connected layer for classifying the signal light state, output the signal light state and the relative position relationship of the vehicle, integrate the trained model into the actual traffic identification, and identify the traffic violations in real time.

[0082] It should be noted that in the above embodiments, the description of each embodiment has its own emphasis, and the parts not described in detail in a certain embodiment can be referred to the related description of other embodiments.

[0083] Those skilled in the art will understand that embodiments of the present application can be provided as methods, systems, or computer program products. Therefore, the present application can take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware aspects. Moreover, the present application can take the form of a computer program product implemented on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0084] The present application is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present application. It should be understood that each flow and / or block in the flowcharts and / or block diagrams, and the combination of flows and / or blocks in the flowcharts and / or block diagrams can be implemented by computer program instructions. These computer program instructions can be provided to the processor of a general-purpose computer, a special-purpose computer, an embedded computer, or other programmable data processing devices to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing devices produce a device that implements the functions specified in the flowcharts and / or block diagrams. Figure 1 The functions specified in one or more flows and / or blocks Figure 1 The functions specified in one or more flows and / or blocks

[0085] These computer program instructions can also be stored in a computer-readable storage medium that can guide the computer or other programmable data processing devices to work in a specific way, so that the instructions stored in the computer-readable storage medium produce a product including instruction devices, which implement the functions specified in the flowcharts and / or block diagrams. Figure 1 The functions specified in one or more flows and / or blocks Figure 1 The functions specified in one or more flows and / or blocks

[0086] These computer program instructions can also be loaded into computer or other programmable data processing devices, so that a series of operations steps are performed on the computer or other programmable data processing devices to generate computer-implemented processes, thus the instructions executed on the computer or other programmable data processing devices provide the function of realizing the processes specified in the flowcharts Figure 1 one flowchart or multiple flowcharts and / or blocks Figure 1 one block or multiple blocks.

[0087] Although the preferred embodiments of the application have been described, those skilled in the art will be able to make additional modifications and variations to these embodiments without departing from the spirit and scope of the application. Accordingly, it is intended that the appended claims be construed to include all such modifications and variations as fall within the scope of the application.

[0088] Obviously, various modifications and changes are possible in the present application without departing from the spirit and scope of the application. It is to be understood that the above description is intended to be illustrative and not restrictive. The scope of the application should be determined by the appended claims and their equivalents.

Claims

1. A method for image library construction and recognition in large-scale incomplete multi-view, multi-modal scenes, characterized in that, Comprise the following steps: S101, deploy multiple sensors and cameras in a traffic scene, collect their positions, orientations and time synchronization information, use these information to construct a space-time alignment matrix, and through matrix operation, unify the multi-modal data into the same space-time coordinate system; S102, fuse the multi-modal data after space-time alignment, predict the target state and update the measurement value, fuse different sensor data, and enhance the integrity of image information; S103, based on the fused data, use the adversarial generative network under space-time constraints to dynamically generate multi-view candidate views, combine the target kinematic model to simulate the appearance and motion state of the target under different views, and complete the missing views; Use the multi-modal data fused by Kalman filtering, including the position, speed and acceleration information of the vehicle target, take the fused data and random noise vector as input, the generator generates candidate views under different views according to the input, generates multiple views to cover different observation angles, uses convolution layer, deconvolution layer and fully connected layer to build network structure, convolution layer is used to extract features, deconvolution layer is used to generate images, and fully connected layer is used to integrate information, the discriminator receives the candidate views generated by the generator and the real multi-view images, extracts features and classifies the received images, uses the target kinematic model to predict the future state of the target under different views according to the current vehicle target state and historical motion information, if the current position of the target is (x1, y1), the speed is (vx1, vy1), the acceleration is (ax1, ay1), and the time interval is Δt, the next time position (x2, y2) can be calculated by the following formula: Wherein, (x2, y2) represents the position coordinates of the target at the next time, the predicted target state is input into the trained generator to generate candidate views under different views, for the missing views, linear interpolation method is used to infer and complete, assuming that the target states under adjacent views δ1 and δ2 (x1, y1) and (x2, y2) are known, the calculation formula of the target state under view δ is as follows: Wherein, (B x ,B y ) represents the position coordinates of the target at the viewing angle δ calculated by the linear interpolation method, according to the inferred target state, the corresponding candidate view is generated by using the generator, the view library is completed, the screened multi-view candidate view is stored in the database, and an index is established for each candidate view, the index information includes view shooting time, viewing angle parameter, target category and position information; S104, fuse the target kinematic model and deep learning technology, track the target across the cameras, construct the traffic regulation ontology library, and combine the image recognition technology to automatically identify and classify traffic violations.

2. The method of claim 1, wherein, In the S101, use GPS to obtain the geographic position information of each sensor and camera, including longitude, latitude and altitude, record these information and associate with the corresponding device, equip all sensors and cameras with network time protocol server to ensure the time synchronization between devices, and record the time stamp information of each device, for each sensor and camera, calculate the conversion parameters from its own coordinate system to the unified coordinate system O-XYZ according to its position and orientation information, the conversion parameters include translation vector and rotation matrix.

3. The method of claim 2, wherein, The translation vector represents the position offset of the sensor and the camera in the unified coordinate system. The position coordinates of the sensor and the camera are subtracted from the origin coordinates of the unified coordinate system to obtain the translation vector. The calculation of the translation vector is based on the origin coordinates O=(O x ,O y ,O z ) of the unified coordinate system O-XYZ, the local coordinate system o-xyz of the sensor and the camera, and the coordinates of the origin of the sensor and the camera in the unified coordinate system P=(P x ,P y ,P z ). The calculation formula of the translation vector is as follows: wherein T is used to describe the position offset from the origin of the unified coordinate system to the origin of the local coordinate system, T x represents the position offset of the sensor and the camera along the x-axis direction relative to the origin in the unified coordinate system, T y represents the position offset of the sensor and the camera along the y-axis direction relative to the origin in the unified coordinate system, T z represents the position offset of the sensor and the camera along the z-axis direction relative to the origin in the unified coordinate system.

4. The method of claim 2, wherein, The rotation matrix represents the angle relationship between the orientation of the sensor and the camera and the coordinate axis of the unified coordinate system. According to the horizontal and vertical angles of the sensor and the camera, the rotation matrix is obtained by using the calculation formula of the rotation matrix. The rotation matrix is used to describe the rotation relationship of the local coordinate system relative to the unified coordinate system. The rotation angles are yaw angle ψ, pitch angle θ and roll angle φ respectively.

5. The method of claim 3, wherein, The translation vector and the rotation matrix of each sensor and camera are combined into a 4*4 homogeneous transformation matrix, which is used as the space-time alignment matrix of the device. The general form of the homogeneous transformation matrix is: where R 3×3 represents a rotation matrix, T 3×1 represents a translation vector, and 0 1×3 is a 1 x 3 zero vector.

6. The method of claim 1, wherein, Start all sensors and cameras to start collecting multi-modal data, including images, radar data, speed data, and record the timestamp and corresponding device information at each time point. For each collected data point, use the corresponding space-time alignment matrix to perform coordinate conversion from the device coordinate system to the unified coordinate system. Multiply the coordinate vector of the data point with the space-time alignment matrix to obtain the coordinates in the unified coordinate system of the device. According to the timestamp information of each time point, the multi-modal data is time-aligned. The linear interpolation method is used to unify the data collected by different devices at different times to the same time point. Assuming that there are two adjacent time points t1 and t2 when the device is collecting data, the corresponding multi-modal data values are v1 and v2 respectively. We want to get the difference data value v at time point t. The specific calculation formula is as follows: Where v represents the difference data value at time point t, v1 represents the multi-modal data value obtained by the sensor and other data collection devices at time point t1, and v2 represents the multi-modal data value obtained by the sensor and other data collection devices at time point t2.

7. The method of claim 1, wherein, In S102, the multi-modal data collected by each sensor and camera is represented in homogeneous coordinate form. Through matrix multiplication operation, the data is converted from the local coordinate system of the sensor and camera to the unified space-time coordinate system. According to the motion characteristics of the target in the traffic scene, the state variable is defined, and the Kalman filter is applied for data fusion. The specific steps include: S1, define state variables: the state variables of the vehicle target include its position (x, y, z), velocity (v x , v y , v z ), acceleration (a x , a y , a z ) in the unified space-time coordinate system, which constitute the state vector S1, define state variables: according to the motion law of the vehicle target, the state transition equation is established: wherein, represents the state vector of the vehicle target at time k, F k represents the state transition matrix, which describes the change relationship of the state of the vehicle target from time k-1 to time k, represents the process noise; S2, initializing Kalman filter parameters: according to the initial measurement value of a certain sensor in the multi-modal data, the initial state of the vehicle target is estimated, the uncertainty of the initial state estimation is described by the initial covariance matrix P0, the process noise covariance matrix Q is set to quantify the statistical characteristics of the process noise ;​ S3, prediction: estimate the vehicle target state at the current time according to the state transition equation and the vehicle target state at the last time calculate the state prediction value of the vehicle target at the current time calculate the covariance prediction value at the current time wherein, indicates the prediction of the uncertainty of the state estimation at the current time before new measurement data S4, updating: preprocessing each sensor measurement value in the spatio-temporally aligned multi-modal data, including noise removal, outlier processing and data format conversion operation, and then combining these measurement values into a measurement vector According to the characteristics and measurement principle of the sensor, a measurement equation is established wherein H k represents a measurement matrix, mapping the state vector to the measurement space, represents measurement noise, and is calculated according to the covariance prediction value The measurement matrix H k and the measurement noise covariance matrix R k , the Kalman gain is calculated The measurement vector z is used to update the state estimation value k and the vehicle target state prediction value at the current time According to the Kalman gain K k and the covariance prediction value , the covariance matrix at the current time is updated wherein I represents an identity matrix; S5, output the updated state estimation value As the fused vehicle target state, it is used for subsequent view generation, target tracking, behavior analysis processing, and covariance matrix P k is saved as the initial covariance matrix of Kalman filtering at the next moment, and the prediction and update steps are repeatedly performed to realize continuous fusion of multi-modal data.

8. The method of claim 1, wherein, In S104, based on the target monitoring model of convolutional neural network, a large number of multi-view image data containing various targets are collected, and the targets are accurately labeled. The labeling information includes target category and location box. The labeled data is used to train the convolutional neural network model. The model parameters are continuously adjusted through the back propagation algorithm. The trained deep learning model is used to detect targets in images captured by each camera. For the detected targets, the features of the targets and the position information in the image are compared, and the data association algorithm is used to associate the same target captured by different cameras. A unique identifier is established for each target. The predicted position of the target kinematics model is fused with the actual position of the target detected by the deep learning model. When identifying the target, the target image T A is matched with the images in the multi-view candidate view library T B . The most similar candidate view to the current target image is retrieved by calculating the similarity between the images. The specific calculation formula is as follows: wherein sim represents similarity, m represents dimension of feature vector, and respectively represent the i-th element of T A and T B , comprehensively collect traffic regulations, extract key concepts from regulation clauses, contain the constitutive elements of illegal behavior and judgment standards, use different traffic violations, use deep learning models to extract corresponding image features, collect video and image data containing traffic violations, label the collected data, mark different types of illegal behavior and related features, including signal light state, vehicle position and vehicle speed, input multi-view candidate views and real-time monitoring images, use multiple convolutional layers and pooling layers to extract spatial features of the image, flatten the output of the convolutional layer and connect it to the fully connected layer for classifying the signal light state, output the signal light state and the relative position relationship of the vehicle, integrate the trained model into the actual traffic identification, and identify traffic violations in real time.

Images