Data processing method, device and readable storage medium

Abstract

The present application discloses a data processing method, device, and readable storage medium, the method comprising: acquiring a driving image associated with a vehicle; the driving image including lane lines on the surface of the road on which the vehicle is traveling; determining a first angular velocity change curve of the vehicle within a first time interval based on the lane lines included in the driving image; determining a second angular velocity change curve of the vehicle within a second time interval based on a set of angular velocities for the vehicle detected by a sensor within the second time interval; and time-calibrating a camera device and a sensor based on a curve correlation between the first angular velocity change curve of the vehicle within the first time interval and the second angular velocity change curve of the vehicle within the second time interval. Using the present application, calibration costs can be reduced and calibration accuracy can be improved in time calibration scenarios. The present application can be applied in the field of transportation.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a data processing method, device, and readable storage medium. Background Art

[0002] Visual-Inertial Odometry (VIO), as a spatial positioning method, is widely used in virtual reality (VR) and augmented reality (AR), autonomous driving, and mobile robotics. Typically, a VIO system consists of two sensors: a visual sensor (called a camera) and an inertial measurement unit (IMU). VIO systems rely on these two sensors for operation.

[0003] Generally, when a VIO system is operating, it assumes that the IMU and camera times are synchronized and aligned. However, due to hardware system issues such as trigger delays, transmission delays, and inaccurately synchronized clocks, there is often a time offset between the IMU and camera. This incorrect assumption of synchronization and alignment can affect the performance of the VIO system. To estimate and correct the time offset between the camera and IMU, thereby effectively improving the performance of the VIO system, existing technologies typically use a calibration plate to measure the camera's rotation, extract corner points, and calculate angular velocity. The camera's angular velocity is then compared with the IMU reading to determine the time difference between the camera and IMU. However, this approach is highly dependent on the calibration plate and a specific calibration scenario, requiring significant manpower and material resources to deploy the calibration scenario, resulting in high costs. Furthermore, it places high demands on hardware quality (such as camera image quality). For scenes with poor camera image quality, the image quality captured by the camera is not high enough, and the calibration plate cannot accurately extract corner points, resulting in inaccurate angular velocity calculations and low accuracy of the compared time difference. Summary of the Invention

[0004] The embodiments of the present application provide a data processing method, device, and readable storage medium, which can reduce calibration costs and improve calibration accuracy in time calibration scenarios.

[0005] On the one hand, an embodiment of the present application provides a data processing method, including:

[0006] Acquire a driving image associated with a vehicle; the driving image is captured by a camera device configured on the vehicle; the driving image includes lane lines on the ground of the road on which the vehicle is traveling;

[0007] determining a first angular velocity change curve of the vehicle within a first time interval based on lane lines included in the driving image;

[0008] determining a second angular velocity change curve of the vehicle in the second time interval based on the angular velocity set of the vehicle detected by the sensor in the second time interval;

[0009] The camera device and the sensor are time-calibrated according to a curve correlation between a first angular velocity change curve of the vehicle in a first time interval and a second angular velocity change curve of the vehicle in a second time interval.

[0010] In one aspect, an embodiment of the present application provides a data processing device, including:

[0011] An image acquisition module is used to acquire a driving image associated with a vehicle; the driving image is captured by a camera device configured on the vehicle; the driving image includes lane lines on the ground of the road on which the vehicle is traveling;

[0012] a curve determination module, configured to determine a first angular velocity change curve of the vehicle within a first time interval based on lane lines included in the driving image;

[0013] The curve determination module is further configured to determine a second angular velocity change curve of the vehicle in the second time interval based on the set of angular velocities of the vehicle detected by the sensor in the second time interval;

[0014] The time calibration module is used to perform time calibration on the camera device and the sensor according to the curve correlation between the first angular velocity change curve of the vehicle in the first time interval and the second angular velocity change curve of the vehicle in the second time interval.

[0015] In one embodiment, the time calibration module includes:

[0016] a correlation matching unit, configured to determine a curve correlation between a first angular velocity change curve within a first time interval and a second angular velocity change curve within a second time interval, and match the curve correlation with a correlation threshold;

[0017] a time updating unit, configured to update the second time interval used for mapping the second angular velocity change curve if the curve correlation is less than a correlation threshold, to obtain an updated time interval for remapping the second angular velocity change curve; the duration of the updated time interval being equal to the duration of the second time interval;

[0018] a time calibration unit, configured to perform time calibration on the camera device and the sensor based on a second angular velocity change curve of the vehicle in the update time interval and a first angular velocity change curve of the vehicle in the first time interval;

[0019] The time calibration unit is further configured to use the time matching result as the time calibration result between the camera device and the sensor if the curve correlation is greater than a correlation threshold.

[0020] In one embodiment, the second angular velocity change curve is in a curvilinear coordinate system, and the curvilinear coordinate system includes a time axis;

[0021] Time update unit, including:

[0022] The curve translation subunit is used to obtain the translation amount and translate the second angular velocity change curve in the second time interval in the curve coordinate system according to the translation amount and the target translation direction; the target translation direction is parallel to the axis direction of the time axis;

[0023] The timestamp acquisition subunit is used to obtain the starting position and the ending position of the second angular velocity change curve after translation in the curve coordinate system, and obtain the starting timestamp corresponding to the starting position of the curve and the ending timestamp corresponding to the ending position of the curve on the time axis;

[0024] The interval determination subunit is used to determine the time interval composed of the start timestamp and the end timestamp as the update time interval.

[0025] In one embodiment, the time calibration unit is further configured to determine an update curve correlation between a second angular velocity change curve within the update time interval and a first angular velocity change curve within the second time interval, and match the update curve correlation with a correlation threshold;

[0026] The time calibration unit is further specifically used to determine the time difference between the update time interval and the second time interval if the update curve correlation is greater than the correlation threshold, and determine the time difference as the time calibration result between the camera device and the sensor.

[0027] In one embodiment, the driving image includes a first driving image pair and a second driving image pair; the first driving image pair includes a first driving sub-image and a second driving sub-image; the second driving image pair includes a third driving sub-image and a fourth driving sub-image;

[0028] Curve determination module, including:

[0029] an angular velocity determination unit, configured to determine a first angular velocity of the vehicle corresponding to a first moment in time based on lane lines respectively included in the first and second driving sub-images; the first moment in time being determined based on a first capturing moment in time and a second capturing moment in time, the first capturing moment in time being the moment in time when the camera device captured the first driving sub-image, and the second capturing moment in time being the moment in time when the camera device captured the second driving sub-image;

[0030] The angular velocity determination unit is further configured to determine a second angular velocity of the vehicle corresponding to a second moment in time based on lane lines respectively included in the third and fourth driving sub-images; the second moment in time being determined based on a third capturing moment in time and a fourth capturing moment in time, the third capturing moment being the moment in time when the camera device captured the third driving sub-image, and the fourth capturing moment being the moment in time when the camera device captured the fourth driving sub-image;

[0031] The curve determining unit is used to determine a first angular velocity change curve corresponding to the vehicle according to the first angular velocity, the second angular velocity, the first moment and the second moment.

[0032] In one embodiment, the angular velocity determination unit includes:

[0033] an offset determination subunit, configured to determine a first offset angle of the vehicle corresponding to the first capturing moment based on the lane line included in the first driving sub-image; the first offset angle being the offset angle between the driving direction of the vehicle at the first capturing moment and the lane line;

[0034] The offset determination subunit is further configured to determine a second offset angle of the vehicle corresponding to the second shooting moment based on the lane line included in the second driving sub-image; the second offset angle is the offset angle between the driving direction of the vehicle at the second shooting moment and the lane line;

[0035] The angular velocity determination subunit is configured to determine a first angular velocity of the vehicle at the first moment according to the first offset angle, the second offset angle, the first shooting moment, and the second shooting moment.

[0036] In one embodiment, the offset determination subunit is further configured to identify lane lines included in the first driving sub-image and determine pixel coordinates of pixel points corresponding to the lane lines in an image coordinate system; the image coordinate system is a coordinate system corresponding to the first driving sub-image;

[0037] The offset determination subunit is further specifically used to perform coordinate conversion on the pixel coordinates according to the initial pitch angle value to obtain the spatial position coordinates of the pixel point in the world coordinate system;

[0038] The offset determination subunit is further specifically used to perform straight line fitting on the spatial position coordinates to obtain a fitting straight line corresponding to the pixel point, and determine the first offset angle corresponding to the vehicle at the first shooting moment based on the fitting straight line.

[0039] In one embodiment, the offset determination subunit is further configured to obtain a vertical distance corresponding to the camera device; the vertical distance refers to a straight-line distance between the camera device and the road surface;

[0040] The offset determination subunit is further specifically used to determine the rotation matrix corresponding to the camera device based on the initial pitch angle value, and to determine the spatial position coordinates of the pixel point in the world coordinate system based on the rotation matrix, vertical distance and pixel coordinates.

[0041] In one embodiment, the lane lines included in the first driving sub-image include N lane lines, and the first driving sub-image corresponds to N fitted straight lines;

[0042] The offset determination subunit is further specifically used to obtain the slope of each of the N fitted straight lines;

[0043] The offset determination subunit is further specifically used to sort the N straight line slopes in order of magnitude to obtain a straight line slope sequence;

[0044] The offset determination subunit is further specifically configured to sequentially obtain a first straight line slope and a second straight line slope in the straight line slope sequence, determine a fitting straight line corresponding to the first straight line slope as a first target fitting straight line, and determine a fitting straight line corresponding to the second straight line slope as a second target fitting straight line;

[0045] The offset determination subunit is further specifically configured to determine a first offset angle corresponding to the vehicle at the first shooting moment based on the first target fitting straight line and the second target fitting straight line.

[0046] In one embodiment, the offset determination subunit is further specifically configured to determine a vertically projected ground position of the camera device on the horizontal ground of the road, and determine the vertically projected ground position as the world coordinate origin of the world coordinate system;

[0047] The offset determination subunit is further specifically used to determine the intersection point between the first target fitting straight line and the second target fitting straight line, and the distance between the intersection point and the world coordinate origin;

[0048] The offset determination subunit is further specifically configured to determine a linear relationship between the first target fitting straight line and the second target fitting straight line according to the distance;

[0049] The offset determination subunit is further specifically configured to determine a first offset angle corresponding to the vehicle at the first shooting moment according to the straight line relationship.

[0050] In one embodiment, the offset determination subunit is further configured to determine, if the straight line relationship is a parallel relationship, an initial pitch angle value as a correct pitch angle value, and determine the angle between the driving direction of the vehicle at the first shooting moment and the first target fitting line as a first offset angle;

[0051] The offset determination subunit is also specifically used to adjust the initial pitch angle value if the straight line relationship is a non-parallel relationship, convert the pixel coordinates according to the adjusted pitch angle value, obtain the updated spatial position coordinates of the pixel point in the world coordinate system, perform straight line fitting on the updated spatial position coordinates, obtain the updated fitting straight line corresponding to the pixel point, and determine the first offset angle corresponding to the vehicle at the first shooting moment based on the updated fitting straight line.

[0052] In one embodiment, the first shooting moment and the second shooting moment are adjacent shooting moments;

[0053] The angular velocity determination subunit is further specifically configured to determine an absolute value of an angle difference between the first offset angle and the second offset angle;

[0054] The angular velocity determination subunit is further specifically configured to obtain an intermediate shooting moment between the first shooting moment and the second shooting moment, and determine the intermediate shooting moment as the first moment;

[0055] The angular velocity determination subunit is further specifically configured to determine the shooting interval between the first shooting moment and the second shooting moment, and determine the first angular velocity of the vehicle corresponding to the first moment according to the absolute value of the angle difference and the shooting interval.

[0056] In one embodiment, the curve determining unit includes:

[0057] an initial curve determining subunit, configured to determine an initial angular velocity change curve corresponding to the vehicle according to the first moment, the first angular velocity, the second moment, and the second angular velocity;

[0058] a mean data determination subunit, configured to determine a mean time between the first time and the second time, and determine the mean angular velocity between the first angular velocity and the second angular velocity as the angular velocity of the vehicle corresponding to the mean time;

[0059] The curve determination subunit is used to determine a first angular velocity change curve corresponding to the vehicle according to the first moment, the first angular velocity, the second moment, the second angular velocity, the average moment and the average angular velocity.

[0060] In one aspect, an embodiment of the present application provides a computer device, including: a processor and a memory;

[0061] The memory stores a computer program, and when the computer program is executed by the processor, the processor executes the method in the embodiment of the present application.

[0062] On one hand, an embodiment of the present application provides a computer-readable storage medium, which stores a computer program. The computer program includes program instructions. When the program instructions are executed by a processor, the method in the embodiment of the present application is executed.

[0063] In one aspect of the present application, a computer program product or computer program is provided. The computer program product or computer program includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the method provided in one aspect of the embodiments of the present application.

[0064] In an embodiment of the present application, a camera device can capture an image of a vehicle traveling on a road surface, wherein the image includes lane markings. Subsequently, a first angular velocity change curve of the vehicle in a first time interval can be determined based on the lane markings in the image. A sensor in the vehicle can also detect a set of angular velocities of the vehicle, and based on this set of angular velocities, a second angular velocity change curve of the vehicle in a second time interval can be determined. The curve correlation degree (i.e., the degree of curve correlation) between the first angular velocity change curve and the second angular velocity change curve can be calculated to perform time calibration between the camera device and the sensor. As can be seen, the present application does not require additional equipment or a special external environment when performing time calibration between the camera device and the sensor, only lane markings are required, which is highly flexible and greatly reduces calibration costs. At the same time, by determining the curve correlation degree between the angular velocity change curve corresponding to the camera device and the angular velocity change curve corresponding to the sensor to perform time calibration, time calibration can be performed when the angular velocity captured by the camera device and the angular velocity detected by the sensor are highly correlated (i.e., the angular velocities are highly correlated), resulting in a highly accurate time calibration result. In summary, this application can improve calibration flexibility and accuracy and reduce calibration costs in scenarios where time calibration of camera devices and sensors is performed. BRIEF DESCRIPTION OF THE DRAWINGS

[0065] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0066] Figure 1 This is a network architecture diagram provided by an embodiment of the present application;

[0067] Figure 2 This is a flow chart of a data processing method provided in an embodiment of the present application;

[0068] Figure 3 is a schematic diagram of an angular velocity change curve provided in an embodiment of the present application;

[0069] Figure 4 This is a schematic diagram of time calibration of a translational angular velocity change curve provided in an embodiment of the present application;

[0070] Figure 5 This is a flowchart of a data processing method provided by an embodiment of the present application;

[0071] Figure 6 This is a schematic diagram of a curve smoothing process provided by an embodiment of the present application;

[0072] Figure 7 This is a schematic diagram of a process for determining an offset angle of a vehicle provided in an embodiment of the present application;

[0073] Figure 8a This is a schematic diagram of a scenario for determining an offset angle provided in an embodiment of the present application;

[0074] Figure 8b This is a schematic diagram of a scenario for determining an offset angle provided in an embodiment of the present application;

[0075] Figure 9 is a structural diagram of a data processing device provided in an embodiment of the present application;

[0076] Figure 10 It is a structural diagram of a computer device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0077] The following will be combined with the drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are only part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0078] This application relates to the field of intelligent transportation. For ease of understanding, the following will first explain intelligent transportation and its related concepts.

[0079] Intelligent Vehicle Infrastructure Cooperative Systems (IVICS), also known as VICS, are a development direction of Intelligent Transportation Systems (ITS). VICS utilizes advanced wireless communications and next-generation internet technologies to implement dynamic, real-time information exchange between vehicles and roads. Based on the collection and integration of dynamic traffic information across time and space, VICS conducts active vehicle safety control and collaborative road management. This fully realizes effective coordination between people, vehicles, and roads, ensuring traffic safety and improving traffic efficiency, resulting in a safe, efficient, and environmentally friendly road transportation system.

[0080] The present application relates to an Advanced Driving Assistance System (ADAS) and a Visual-Inertial Navigation System (VINS) in an intelligent vehicle-road cooperative system.

[0081] Among them, ADAS uses various sensors installed on the car (such as millimeter-wave radar, lidar, monocular / binocular cameras, and satellite navigation, etc.) to sense the surrounding environment at any time during the car's driving, collect data, and identify, detect and track static and dynamic objects. It also combines navigation map data to perform systematic calculations and analysis, so that the driver can be aware of possible dangers in advance, effectively increasing the comfort and safety of car driving.

[0082] Visual-Inertial Odometry (VIO) is a computer vision technology that uses cameras and inertial measurement units (IMUs) as sensors to measure and estimate displacement and attitude changes. It primarily achieves simultaneous localization and mapping (SLAM) by fusing camera and IMU data. The camera sensor captures photons within a certain exposure time, generating electrical signals to produce digital images. This provides rich and accurate information even under slow motion, while the IMU measures the angular velocity and linear acceleration of the vehicle, unaffected by the external environment. Because cameras rely on capturing photons within a certain exposure time, they are susceptible to the effects of exposure time, aperture (which determines the amount of light), and working distance. Furthermore, they cannot accurately estimate position in low-resolution images (such as white walls and snow), in high-speed scenes (which are prone to motion blur), or with high dynamic range. The camera's output frame rate is typically 100 Hz. Relatively speaking, IMUs are more accurate in fast-paced scenarios, but less accurate in slow-paced scenarios due to drift and measurement noise. The IMU's output frequency is much higher than that of a camera, typically around 1000 Hz. Therefore, camera sensors and IMUs complement each other in their respective usage scenarios. By fusing camera and IMU information, more accurate navigation and positioning can be achieved.

[0083] As a spatial positioning method, VIO is widely used in VR / AR, autonomous driving, and mobile robotics for spatial positioning or vehicle navigation. To better integrate camera and IMU sensor information, this application primarily provides a method for time calibration of the camera and IMU in a VIO system.

[0084] See Figure 1 , Figure 1 This is a network architecture diagram provided by the embodiment of this application. Figure 1 As shown, the network architecture may include a service server 1000 and a terminal device cluster. The terminal device cluster may include one or more terminal devices. The number of terminal devices is not limited here. Figure 1 As shown, the plurality of terminal devices may include a terminal device 100a, a terminal device 100b, a terminal device 100c, ..., a terminal device 100n; Figure 1 As shown, the terminal device 100a, the terminal device 100b, the terminal device 100c, ..., the terminal device 100n can respectively establish a network connection with the service server 1000, so that each terminal device can exchange data with the service server 1000 through the network connection.

[0085] In the embodiment of the present application, a terminal device can be selected from multiple terminal devices as the target terminal device. The terminal device can be an intelligent vehicle-mounted device, but is not limited thereto. The intelligent vehicle-mounted device can be deployed in a transportation vehicle (such as a bus, car, truck, etc.). For example, in the embodiment of the present application, Figure 1 The terminal device 100a shown is used as the target terminal device. The target terminal device can be deployed in a vehicle, and the vehicle can also be deployed with a camera device (also called a visual sensor or camera device) and an inertial measurement unit (also called an inertial sensor or IMU). Among them, the camera device can capture images of the vehicle when it is driving on the road surface (also called driving images), and the IMU can also measure the corresponding angular velocity set of the vehicle at different times. The driving image captured by the camera device and the angular velocity set measured by the IMU at different times can be used as business data, and the target terminal device can send the business data to the business server 1000 via a network connection.

[0086] Subsequently, the business server 1000 can determine the first angular velocity change curve of the vehicle in the first time interval based on the lane lines included in the driving image (that is, the lane lines on the road surface in the driving image); similarly, the business server 1000 can also determine the second angular velocity change curve of the vehicle in the second time interval (the time interval composed of different moments corresponding to the angular velocity set measured by the IMU) based on the angular velocity set measured by the IMU at different moments. The business server 1000 can calculate the degree of correlation between the first angular velocity change curve and the second angular velocity change curve (which can be called curve correlation), and can perform time calibration on the camera device (that is, the camera sensor) and the IMU sensor based on the degree of correlation, so that the information of the camera sensor and the information of the IMU sensor can be better integrated to perform more accurate positioning or navigation. Among them, for the specific implementation method of determining the first angular velocity change curve and time calibration of the camera device and the IMU sensor, please refer to the subsequent Figure 2 Description in the corresponding embodiment.

[0087] It is understood that the methods provided in the embodiments of the present application can be executed by computer devices, including but not limited to terminal devices or business servers. The business server can be an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms.

[0088] The terminal device and the service server may be connected directly or indirectly via wired or wireless communication, which is not limited in this application.

[0089] Optionally, it is understood that the aforementioned computer device (such as the aforementioned service server 1000, terminal device 100a, terminal device 100b, etc.) can be a node in a distributed system, wherein the distributed system can be a blockchain system, and the blockchain system can be a distributed system formed by connecting multiple nodes through network communication. The nodes can form a peer-to-peer (P2P) network, and the P2P protocol is an application layer protocol that runs on top of the Transmission Control Protocol (TCP). In a distributed system, any form of computer device, such as a service server, terminal device, or other electronic device, can become a node in the blockchain system by joining the peer-to-peer network. To facilitate understanding, the concept of blockchain will be explained below: Blockchain is a new application model of computer technologies such as distributed data storage, peer-to-peer transmission, consensus mechanism, and encryption algorithm. It is mainly used to organize data in chronological order and encrypt it into a ledger to make it tamper-proof and forgery-proof. It can also verify, store, and update data. When the computer device is a blockchain node, due to the tamper-proof and anti-counterfeiting characteristics of the blockchain, the data in this application (such as driving images, angular velocity change curves, time calibration results, etc.) can be authentic and secure, so that the results obtained after relevant data processing based on these data can be more reliable.

[0090] Further, see Figure 2 , Figure 2 This is a flow chart of a data processing method provided by an embodiment of the present application. The data processing method can be performed by a business server (such as the above Figure 1 The service server 1000 in the corresponding embodiment) can also be executed by a terminal device (such as the above Figure 1 The data processing method is executed by any terminal device in the terminal device cluster in the corresponding embodiment, such as the terminal device 100a); it can also be executed by the terminal device and the service server together. For ease of understanding, the following description is based on the example of the data processing method being executed by the terminal device. Figure 2 As shown, the process may include at least the following steps S101 to S104:

[0091] Step S101, obtaining a driving image associated with a vehicle; the driving image is captured by a camera device configured on the vehicle; the driving image includes lane lines on the ground of the road on which the vehicle is traveling.

[0092] In this application, the camera device may refer to a visual sensor with a shooting function such as a camera or a driving recorder. The camera device may be deployed in a vehicle and may be used to shoot the driving environment of the vehicle during its driving on the road surface to obtain a driving image. The terminal device may include a vehicle-mounted computer, a computer or other device with computing functions. The terminal device may be connected to the camera device of the vehicle. The terminal device may send a request to the camera device so that the camera device can feedback the image (which may be referred to as a driving image) shot during the driving of the vehicle on the road surface (the road surface may include lane lines, which are in a parallel relationship on the road surface). After capturing the driving image, the camera device may also actively send the driving image to the terminal device so that the terminal device can receive the driving image.

[0093] The terminal device may be installed on a vehicle, which may be a vehicle (such as an autonomous vehicle, or a vehicle equipped with an Advanced Driving Assistance System (ADAS)), a drone, or the like.

[0094] It should be understood that the terminal device may also have a perception function. When the vehicle is actually driving on the road, the terminal device can identify objects around the vehicle based on the environmental images collected by the camera device, and then determine the driving strategy of the vehicle.

[0095] It is understood that the images captured by the camera device can be video images, and the terminal device can use each frame of the video image as a driving image (i.e., an image to be processed), or filter out images that need to be processed from the video image as driving images, for example, capturing an image every few frames as a driving image for processing. Of course, in actual applications, the images captured by the camera device can also be pictures. The camera device can capture a picture at a certain time interval (e.g., every few milliseconds, every few seconds, every few minutes, etc.), and the picture can be sent to the terminal device as a driving image.

[0096] It can be understood that, assuming that the driving image collected by the terminal device is a picture for example, the vehicle can be controlled to change lanes (for example, change lanes twice in a row) at a higher speed (such as 40km / s) on the road surface including lane lines. In this process, the camera device is set to take a frame of picture (called a driving image) every 2 seconds, thereby obtaining multiple driving images taken at different times. The camera device can send these multiple driving images taken at different times to the terminal device.

[0097] Step S102 : determining a first angular velocity change curve of the vehicle in a first time interval according to the lane lines included in the driving image.

[0098] In this application, after acquiring multiple driving images captured by a camera device at different times, a terminal device can form a time interval (referred to as a first time interval) based on these different times. Based on the lane lines included in each driving image, the angular velocity of the vehicle at each time within the first time interval can be determined. A coordinate system including a time axis and an angular velocity axis can be established, and an angular velocity change curve (referred to as a first angular velocity change curve) can be plotted in this coordinate system based on the angular velocity of the vehicle at each time within the first time interval.

[0099] To understand the first angular velocity curve, please refer to Figure 3 , Figure 3 Schematic diagram of an angular velocity change curve provided in an embodiment of the present application. Figure 3 As shown, taking the establishment of a plane rectangular coordinate system (a coordinate system composed of two axes perpendicular to each other and having a common origin on the same plane) as an example, the two axes in this plane rectangular coordinate system are placed in horizontal and vertical positions respectively, and the rightward and upward directions are the positive directions of the two axes respectively. Among them, the axis in the horizontal direction can be called the x-axis (or horizontal axis); the axis perpendicular to the x-axis can be called the y-axis (or vertical axis). It should be understood that the x-axis and the y-axis can be collectively referred to as coordinate axes, and either axis can be used as the time axis (at the same time, the other axis can be used as the angular velocity axis). Here, the x-axis can be used as the time axis and the y-axis can be used as the angular velocity axis. The common origin of the x-axis and the y-axis (which can be called the intersection point) is called the origin of the rectangular coordinate system.

[0100] like Figure 3As shown, taking the case where the camera device takes a picture every 10 seconds at a certain moment (for example, the moment when the vehicle starts to change lanes, 10:30:00 on September 14, 2021), then from the 0th to the 50th second, the camera device can take 5 pictures and obtain 5 driving images. The first time interval can be determined as [0, 50] (i.e., from 0th second to 50th second. If corresponding to the time, it is from 10:30:00 on September 14, 2021 to 10:30:50 on September 14, 2021). According to the lane lines respectively included in the 5 driving images taken by the camera device, the angular velocity of the vehicle at each moment in the first time interval [0, 50] can be determined. For example, it can be determined that the angular velocity of the vehicle at 0th second is 0rad / s (assuming that at 0th second, the vehicle is not driving (or has not changed lanes), the angular velocity of the vehicle at 0th second is 0), the angular velocity at 5th second is 0.025 (the unit can be radians / s: rad / s), the angular velocity at 10th second is 0.1rad / s, the angular velocity at 15th second is 0.2rad / s,..., and the angular velocity at 50th second is 0.07rad / s.

[0101] A moment and its corresponding angular velocity can be combined into a coordinate. For example, the 5th second and the corresponding angular velocity of 0.025 rad / s can be combined into a coordinate (5, 0.025). This can yield multiple coordinates, and the points corresponding to these coordinates can be obtained in the plane rectangular coordinate system. These points can then be smoothly connected to form a plot such as Figure 3 The angular velocity change curve shown in FIG. 1 can be referred to as the first angular velocity change curve. Figure 3 The time interval (every 10 seconds), time interval ([0, 50]), angular velocity and angular velocity change curve of the camera shooting images provided in the corresponding embodiment are examples made for ease of understanding and have no practical reference significance.

[0102] For the specific implementation of determining the angular velocity of the vehicle at each moment in the first time interval based on the lane lines included in each driving image, and then determining the first angular velocity change curve, please refer to the subsequent Figure 5 Description in the corresponding embodiment.

[0103] Step S103 : determining a second angular velocity change curve of the vehicle in the second time interval according to the angular velocity set of the vehicle detected by the sensor in the second time interval.

[0104] In the present application, the sensor here may be an inertial sensor, wherein the inertial sensor may be a sensor that detects and measures acceleration, tilt, shock, vibration, rotation and multi-degree-of-freedom (DoF) motion. Inertial sensors are important components for solving navigation, orientation and motion carrier control. Inertial sensors may include accelerometers, angular velocity sensors (also known as gyroscopes), attitude and heading reference systems (AHRS), inertial measurement units (IMUs), and the like. Among them, the accelerometer is a sensor that uses the inertial force of the sensing mass to measure, and is generally composed of a standard mass block (sensing element) and a detection circuit. The gyroscope is made based on the principle of sensing angular velocity by using the Coriolis effect when the vibration mass is rotated by the base (housing). The main forms include frame drive (inner and outer frames), comb drive, electromagnetic drive, etc. An AHRS is primarily composed of a three-axis gyroscope, an accelerometer, and a magnetic sensor. It uses the four-element method to calculate and output the pitch, roll, and heading angles of a moving object. An IMU is primarily composed of three accelerometers, three gyroscopes, and a calculation circuit. It is primarily used to measure the angular velocity and linear acceleration of a body, and is unaffected by external influences. The inertial sensor in this application may refer to an inertial sensor capable of measuring the angular velocity of a body, such as an angular velocity sensor (also known as a gyroscope), an IMU, or the like.

[0105] The inertial sensor can be deployed in a vehicle. When the vehicle changes lanes rapidly, the inertial sensor can measure the angular velocity of the vehicle at various moments. The terminal device can be connected to the inertial sensor to read the angular velocity values ​​measured by the inertial sensor at various moments, thereby obtaining a set of angular velocities corresponding to the vehicle at various moments measured by the inertial sensor. A second time interval can be determined based on these moments (the second time interval can be the same as the first time interval, but can also be different, for example, the second time interval can be slightly smaller than the first time interval). Based on the set of angular velocities corresponding to the vehicle at various moments within the second time interval measured by the inertial sensor, a second angular velocity change curve of the vehicle within the second time interval can also be plotted. The specific implementation method for determining the second angular velocity change curve based on the angular velocity corresponding to the vehicle at various moments measured by the inertial sensor is the same as the method for determining the first angular velocity change curve based on the angular velocity corresponding to the vehicle at various moments determined by the lane markings (i.e., establishing a coordinate system and then plotting within the coordinate system). The method for plotting the second angular velocity change curve will not be further described here.

[0106] It is understandable that the second angular velocity change curve can be plotted in the same coordinate system (e.g., a plane rectangular coordinate system) as the first angular velocity change curve. Thus, the first angular velocity change curve and the second angular velocity change curve can be placed in the same coordinate system, thereby facilitating comparison between the first angular velocity change curve and the second angular velocity change curve.

[0107] Step S104 , time calibration is performed on the camera device and the sensor according to a curve correlation between a first angular velocity change curve of the vehicle in the first time interval and a second angular velocity change curve of the vehicle in the second time interval.

[0108] In this application, after determining the first angular velocity change curve and the second angular velocity change curve, the degree of correlation between the first angular velocity change curve in the first time interval and the second angular velocity change curve in the second time interval can be calculated (which can be understood as the degree of correlation between the angular velocity captured by the camera device (i.e., corresponding to the first angular velocity change curve) and the angular velocity measured by the inertial sensor (i.e., corresponding to the second angular velocity change curve)). This degree of correlation can be called the curve correlation degree. Based on this curve correlation degree, the camera device and the sensor can be time-calibrated.

[0109] Among them, a specific implementation method for time calibration of the camera device and the inertial sensor based on the curve correlation between the first angular velocity change curve in the first time interval and the second angular velocity change curve in the second time interval can be: the curve correlation between the first angular velocity change curve in the first time interval and the second angular velocity change curve in the second time interval can be determined, and then the correlation threshold can be obtained (it can be a manually specified value or a value generated by a machine through an algorithm; the presentation form of the correlation threshold can be a percentage (such as 90%), a fraction (such as 9 / 10), a decimal (such as 0.9), etc.), which There will be no restrictions here), and then the curve correlation can be matched with the correlation threshold; if the curve correlation is less than the correlation threshold, the second time interval used to map the second angular velocity change curve can be time-updated to obtain an updated time interval for remapping the second angular velocity change curve, and the camera device and the sensor are time-calibrated according to the second angular velocity change curve of the vehicle in the updated time interval and the first angular velocity change curve of the vehicle in the first time interval; wherein the duration of the update time interval is equal to the duration of the second time interval; and if the curve correlation is greater than the correlation threshold, the time matching result can be used as the time calibration result between the camera device and the sensor.

[0110] It should be understood that when determining a first angular velocity change curve within a first time interval and a second angular velocity change curve within a second time interval, due to the existence of trigger delay, transmission delay, and inaccurate clock synchronization, the time between the camera device (camera) and the inertial sensor (IMU) is likely to be out of sync, that is, there will be a time difference between the camera device and the inertial sensor (IMU). In this case, the present application can determine whether the time between the camera device and the inertial sensor is synchronized by the curve correlation between the first angular velocity change curve and the second angular velocity change curve; and when the time between the camera device and the inertial sensor is out of sync, the time difference between the camera device and the inertial sensor is determined by the curve correlation, so that the camera device and the inertial sensor can be time-calibrated, and the camera device and the inertial sensor can be better integrated to achieve more accurate positioning.

[0111] Among them, it can be understood that when the first time interval is the same as the second time interval (that is, the first time interval and the second time interval are the same time interval), when the first angular velocity change curve in the first time interval and the second angular velocity change curve in the second time interval have a curve correlation (that is, the degree of correlation) greater than (or equal to) the correlation threshold, then it can be considered that the angular velocity captured by the camera device and the angular velocity measured by the inertial sensor are highly correlated (the degree of overlap is very high), and the camera device and the inertial sensor both measure angular velocities with a very high degree of overlap in the same time interval, then it can be considered that the time between the camera device and the inertial sensor is synchronized (that is, the time calibration result between the camera device and the inertial sensor is a time matching result, and the time difference is 0).

[0112] If the curve correlation (i.e., correlation degree) between the second angular velocity change curves within the second time interval is less than the correlation threshold, then it can be considered that the angular velocity captured by the camera device is not correlated with the angular velocity measured by the inertial sensor (or the correlation degree is not high enough). In this case, the first angular velocity change curve or the second angular velocity change curve can be kept unchanged and moved (moved until they increasingly overlap). By moving, the time interval of the moved angular velocity change curve can be changed to obtain an updated time interval, and the first angular velocity change curve and the second angular velocity change curve can be made to overlap and correlate more closely. Until the curve correlation between the two angular velocity change curves exceeds the correlation threshold, the time change of the moved angular velocity change curve can be checked. The time change can be understood as the time difference between the updated time interval and the original time interval (the time interval corresponding to the moved angular velocity change curve (i.e., one of the first and second time intervals)). This time difference can be used as the time difference between the camera device and the inertial sensor, and this time difference can be used as the time calibration result.

[0113] In this application, to facilitate calculation and comparison, the first angular velocity change curve and the second angular velocity change curve can be plotted and displayed in the same coordinate system. When the curve correlation between the first angular velocity change curve in the first time interval and the second angular velocity change curve in the second time interval is less than a correlation threshold, one of the first and second angular velocity change curves can be moved in the coordinate system to change the time interval of the angular velocity change curve, thereby changing the curve correlation between the two angular velocity change curves.

[0114] Taking the example of a first angular velocity change curve and a second angular velocity change curve being in the same coordinate system (which may be referred to as a curvilinear coordinate system), where the curvilinear coordinate system may be a rectangular coordinate system, with the horizontal axis being the time axis and the vertical axis being the angular velocity axis, as an example, when the curve correlation between the first angular velocity change curve in the first time interval and the second angular velocity change curve in the second time interval is less than a correlation threshold, the position of the first angular velocity change curve may be maintained unchanged, while the second angular velocity change curve may be horizontally translated (parallel to the time axis), thereby updating the time interval of the second angular velocity change curve (i.e., the second time interval). That is to say, for time updating the second time interval used to map the second angular velocity change curve, a specific implementation method for obtaining the updated time interval for remapping the second angular velocity change curve may be: the translation amount may be obtained, and then the second angular velocity change curve in the second time interval in the curve coordinate system may be translated according to the translation amount and the target translation direction; wherein the target translation direction is parallel to the axis direction of the time axis; then, the starting position and the ending position of the translated second angular velocity change curve may be obtained in the curve coordinate system, and the starting timestamp corresponding to the starting position of the curve and the ending timestamp corresponding to the ending position of the curve may be obtained on the time axis; the time interval composed of the starting timestamp and the ending timestamp may be determined as the updated time interval.

[0115] Furthermore, after the second angular velocity change curve is translated, the curve correlation between the second angular velocity change curve in the update time interval and the first angular velocity change curve in the first time interval (which can be called the update curve correlation) can be determined again, and the update curve correlation can be matched with the correlation threshold; if the update curve correlation at this time is greater than (or equal to) the correlation threshold, the time difference between the update time interval and the second time interval can be determined, and the time difference can be determined as the time calibration result between the camera device and the sensor. If the correlation of the updated curve is less than the correlation threshold, the second angular velocity change curve can be translated again (translated in the target translation direction according to the translation amount), and then the new curve correlation between the second angular velocity change curve in the new time interval and the first angular velocity change curve in the first time interval is calculated again until the curve correlation is greater than the correlation threshold. At this time, the time translation corresponding to the second angular velocity change curve can be determined (that is, the time change corresponding to the translation), and the time translation can be used as the time difference between the camera device and the inertial sensor (that is, the time calibration result).

[0116] It should be understood that the above-mentioned translation amount can refer to a single translation amount, which can be the length corresponding to a single time change fixed in the curvilinear coordinate system. Each time the angular velocity change curve is moved, the length corresponding to the single time change amount can be translated. Thus, the angular velocity change curve can be translated to a new position in the curvilinear coordinate system to obtain a new time interval. For example, if the single time change amount is 0.05s, the length corresponding to the word time change amount of 0.05s can be obtained on the time axis of the curvilinear coordinate system. The angular velocity change curve is then translated horizontally (with the horizontal axis as the time axis. If the vertical axis is the time axis, the angular velocity change curve is translated vertically), and the length of the translation is the length corresponding to the 0.05s. It should be understood that when the correlation between the two angular velocity change curves is greater than the correlation threshold due to translation, the number of translations can be obtained. The number of translations is multiplied by the word time change amount, and the product obtained is the time difference between the camera device and the inertial sensor. Of course, the amount of translation each time can also be different. When the correlation between the two angular velocity change curves is greater than the correlation threshold through translation, the time difference between the last time interval and the original time interval (such as the second time interval) can be obtained (for example, the start timestamp of the last time interval minus the start timestamp of the original time interval). This time difference can be used as the time difference between the camera and the inertial sensor (time calibration result). Alternatively, the time change of each translation can be added and summed, and the resulting sum can be used as the time difference between the camera device and the inertial sensor.

[0117] For easier understanding, please refer to Figure 4 , Figure 4 This is a schematic diagram of a time calibration of a translation angular velocity change curve provided by an embodiment of the present application. Figure 4 The curvilinear coordinate system shown can be Figure 3 In the plane rectangular coordinate system of the corresponding embodiment, in the curve coordinate system, the curve 40a can be the above Figure 3 The curve in the corresponding embodiment, curve 40a can be the first angular velocity change curve corresponding to the camera device, and the first time interval corresponding to the first angular velocity change curve is [0, 50]. In this curve coordinate system, the second angular velocity change curve (i.e., curve 40b) can be drawn according to the angular velocity measured by the inertial sensor. The second time interval corresponding to the second angular velocity change curve is also [0, 50].

[0118] Furthermore, a curve correlation between the first angular velocity change curve (i.e., curve 40a) in the first time interval [0, 50] and the second angular velocity change curve (i.e., curve 40b) in the second time interval [0, 50] can be calculated. Subsequently, the curve correlation can be compared with a correlation threshold. If the curve correlation is 0.8 and the correlation threshold is 0.95, then the comparison can determine that the curve correlation is less than the correlation threshold. In this case, the second angular velocity change curve (i.e., curve 40b) can be shifted.

[0119] like Figure 4 As shown, the time variation of the curve can be obtained as 2.5 seconds. Then, the length corresponding to the current time variation of 2.5 seconds can be obtained in the curve coordinate system. Curve 40b can be translated along the positive direction (horizontal direction) of the time axis by the length corresponding to the time variation of 2.5 seconds. At this time, curve 40b will reach a new position and will be in a new time interval of [2.5, 52.5]. At this time, the new curve correlation between curves 40a and 40b can be calculated again as 0.96. After comparing this new curve correlation of 0.96 with the correlation threshold of 0.95, it can be determined that the new curve correlation is greater than the correlation threshold. At this time, it can be considered that curves 40a and 40b have a sufficiently high correlation. The angular velocity captured by the camera device in the time interval [0, 50] is correlated with the angular velocity measured by the inertial sensor in the time interval [2.5, 52.5] (which can be understood as being synchronized). Because the time required for the translation curve 40b to change is 2.5 seconds, the 2.5 seconds can be used as the time difference between the camera device and the inertial sensor, that is, the 2.5 seconds is the time calibration result between the camera device and the inertial sensor.

[0120] In an embodiment of the present application, a camera device can capture an image of a vehicle traveling on a road surface, wherein the image includes lane markings. Subsequently, a first angular velocity change curve of the vehicle in a first time interval can be determined based on the lane markings in the image. A sensor in the vehicle can also detect a set of angular velocities of the vehicle, and based on this set of angular velocities, a second angular velocity change curve of the vehicle in a second time interval can be determined. The curve correlation degree (i.e., the degree of curve correlation) between the first angular velocity change curve and the second angular velocity change curve can be calculated to perform time calibration between the camera device and the sensor. As can be seen, the present application does not require additional equipment or a special external environment when performing time calibration between the camera device and the sensor, only lane markings are required, which is highly flexible and greatly reduces calibration costs. At the same time, by determining the curve correlation degree between the angular velocity change curve corresponding to the camera device and the angular velocity change curve corresponding to the sensor to perform time calibration, time calibration can be performed when the angular velocity captured by the camera device and the angular velocity detected by the sensor are highly correlated (i.e., the angular velocities are highly correlated), resulting in a highly accurate time calibration result. In summary, this application can improve calibration flexibility and accuracy and reduce calibration costs in scenarios where time calibration of camera devices and sensors is performed.

[0121] Further, see Figure 5 , Figure 5 This is a flow chart of a data processing provided by an embodiment of the present application. This flow can correspond to the above Figure 2 The corresponding embodiment is a process of determining the angular velocity of the vehicle at each moment in the first time interval and determining the first angular velocity change curve. Figure 5 As shown, the process may include at least the following steps S501 to S503:

[0122] Step S501: Determine the first angular velocity of the vehicle at the first moment based on the lane lines respectively included in the first driving sub-image and the second driving sub-image; the first moment is determined based on the first shooting moment and the second shooting moment, the first shooting moment is the moment when the camera device shoots the first driving sub-image, and the second shooting moment is the moment when the camera device shoots the second driving sub-image.

[0123] Specifically, the embodiment of the present application takes the driving image including the first driving image pair and the second driving image pair (the first driving image pair includes the first driving sub-image and the second driving sub-image; the second driving image pair includes the third driving sub-image and the fourth driving sub-image) as an example. Figure 2In the corresponding embodiment, the camera device can capture a driving image at regular intervals (or capture at random times) to obtain multiple driving images within a period of time. After obtaining the multiple driving images, the terminal device can group the driving images in pairs according to the shooting time to obtain multiple driving image pairs. The driving images contained in each driving image pair can be called driving sub-images. The first driving image pair here can be any driving image pair among the multiple driving image pairs, and the first driving sub-image and the second driving sub-image can be the driving images contained in the first driving image pair. The second driving image pair can be a driving image pair shot immediately after the first driving image pair, and the third driving sub-image and the fourth driving sub-image can be the driving images contained in the second driving image pair.

[0124] For example, the driving images include driving image 1, driving image 2, driving image 3, driving image 4, driving image 5, and driving image 6, wherein driving image 1 is captured earlier than driving image 2, driving image 2 is captured earlier than driving image 3, driving image 3 is captured earlier than driving image 4, driving image 4 is captured earlier than driving image 5, and driving image 5 is captured earlier than driving image 6. Here, based on adjacent capturing times, driving image 1 and driving image 2 are grouped into a driving image pair (driving image 1, driving image 2), driving image 2 and driving image 3 are grouped into a driving image pair (driving image 2, driving image 3), driving image 3 and driving image 4 are grouped into a driving image pair (driving image 3, driving image 4), driving image 4 and driving image 5 are grouped into a driving image pair (driving image 4, driving image 5), and driving image 5 and driving image 6 are grouped into a driving image pair (driving image 5, driving image 6). The driving images contained in each driving image pair may include driving sub-images. For example, the driving image pair (driving image 1, driving image 2) can be considered the first driving image pair, and driving image 1 contained in this driving image pair (driving image 1, driving image 2) can be called the first driving sub-image, and driving image 2 can be called the second driving sub-image. Because the driving image pair captured after the first driving image pair is the driving image pair (driving image 2, driving image 3) (the earliest captured time in this driving image pair (driving image 2, driving image 3) is the captured time of driving image 2, and the earliest captured time in this driving image pair (driving image 1, driving image 2) is the captured time of driving image 1, and the captured time of driving image 2 is after that of driving image 1), this driving image pair (driving image 2, driving image 3) can be considered the second driving image pair.

[0125] Furthermore, the first angular velocity of the vehicle at the first moment can be determined based on the lane lines respectively included in the first driving sub-image and the second driving sub-image in the first driving image pair. That is to say, the angular velocity of the vehicle at a certain moment (the radian traveled in unit time, such as the radian traveled per second) can be determined by the two front and rear driving images captured by the camera device. The specific method may be: the first offset angle of the vehicle at the first shooting moment can be determined based on the lane lines included in the first driving sub-image; wherein the first offset angle may be the offset angle between the driving direction of the vehicle at the first shooting moment and the lane line; then, the second offset angle of the vehicle at the second shooting moment can be determined based on the lane lines included in the second driving sub-image; wherein the second offset angle is the offset angle between the driving direction of the vehicle at the second shooting moment and the lane line; according to the first offset angle, the second offset angle, the first shooting moment and the second shooting moment, the first angular velocity of the vehicle at the first moment can be determined. Among them, for the specific implementation method of determining the first offset angle of the vehicle at the first shooting moment based on the lane lines included in the first driving sub-image, please refer to the subsequent Figure 6 For the specific implementation of determining the second offset angle corresponding to the vehicle at the second shooting time based on the lane line included in the second driving sub-image, please refer to the description of determining the first offset angle.

[0126] Furthermore, taking the first and second shooting moments as adjacent moments as an example, a specific method for determining the first angular velocity of the vehicle at the first moment based on the first offset angle, the second offset angle, the first shooting moment, and the second shooting moment may include: determining the absolute value of the angular difference between the first offset angle and the second offset angle; then obtaining the intermediate shooting moment between the first and second shooting moments, and determining the intermediate shooting moment as the first moment; then determining the shooting interval between the first and second shooting moments, and determining the first angular velocity of the vehicle at the first moment based on the absolute value of the angular difference and the shooting interval. For example, the absolute value of the angular difference may be divided by the shooting interval, and the resulting value may be used as the first angular velocity of the vehicle at the first moment.

[0127] Of course, for determining the first moment, other moments between the first shooting moment and the second shooting moment (such as the moment at the 1 / 3 position) can also be used as the first moment. The determination of the first moment can be determined according to actual scene requirements.

[0128] Step S502: Determine the second angular velocity of the vehicle at the second moment based on the lane lines respectively included in the third driving sub-image and the fourth driving sub-image; the second moment is determined based on the third shooting moment and the fourth shooting moment, the third shooting moment is the moment when the camera device shoots the third driving sub-image, and the fourth shooting moment is the moment when the camera device shoots the fourth driving sub-image.

[0129] Specifically, the second angular velocity is determined in the same manner as the first angular velocity. Please refer to the description of determining the first angular velocity in step S501 above, which will not be repeated here.

[0130] Step S503 : determining a first angular velocity change curve corresponding to the vehicle according to the first angular velocity, the second angular velocity, the first moment, and the second moment.

[0131] Specifically, the specific method for determining the first angular velocity change curve corresponding to the vehicle based on the first angular velocity, the second angular velocity, the first moment and the second moment may be as follows: the initial angular velocity change curve corresponding to the vehicle may be determined based on the first moment, the first angular velocity, the second moment and the second angular velocity; then, the mean moment between the first moment and the second moment may be determined, and the mean angular velocity between the first angular velocity and the second angular velocity may be determined as the angular velocity corresponding to the vehicle at the mean moment; then, the first angular velocity change curve corresponding to the vehicle may be determined based on the first moment, the first angular velocity, the second moment, the second angular velocity, the mean moment and the mean angular velocity.

[0132] It should be understood that determining the first angular velocity corresponding to the vehicle at the first moment and the second angular velocity corresponding to the second moment can be understood as determining the angular velocity of the vehicle at each moment. At this point, a coordinate system can be established to plot an angular velocity change curve. The resulting angular velocity change curve can be referred to as an initial angular velocity change curve. This initial angular velocity change curve can then serve as the final first angular velocity change curve.

[0133] In one feasible embodiment, to make the angular velocity variation curve smoother, a mean smoothing method can be used to smooth the angular velocity variation curve. Specifically, the mean time between two consecutive moments (e.g., the first moment and the second moment) can be determined, and the angular velocity values ​​corresponding to these two moments can be averaged to obtain a mean angular velocity. This mean angular velocity can then be used as the angular velocity value corresponding to the mean time. Thus, the angular velocity values ​​corresponding to multiple mean times can be obtained. Using the angular velocities corresponding to the first moment, the second moment, and the mean time, a final, smoother first angular velocity variation curve can be redrawn.

[0134] For easier understanding, please refer to Figure 6 , Figure 6 This is a schematic diagram of a curve smoothing process provided by an embodiment of the present application. Figure 6 As shown, curve 60b (which may be the initial angular velocity change curve) can be drawn in the coordinate system based on the angular velocity 0 at the 0th second, the angular velocity 0.1 at the 10th second, the angular velocity 0.2 at the 20th second, the angular velocity 0.15 at the 30th second, the angular velocity 0.1 at the 40th second, and the angular velocity 0.15 at the 50th second. Furthermore, based on the angular velocities at the 0th and 10th seconds, the angular velocity at the 5th second (i.e., the average moment between the 0th and 10th seconds, (10+0) / 2=5) can be determined to be 0.05 (i.e., the average between 0.1 and 0, (0.1+0) / 2=0.05). Based on the angular velocities at the 10th and 20th seconds, the angular velocity at the 15th second (i.e., the average moment between the 10th and 20th seconds, (10+20) / 2=15) can be determined (i.e., the average of the absolute values ​​of the differences between 0.1 and 0.2, (0.2+0.1) / 2=0.15). Similarly, the angular velocity at the 25th second, the angular velocity at the 35th second, and the angular velocity at the 45th second can be determined to be 0.175, 0.125, and 0.125.

[0135] Furthermore, based on the angular velocities corresponding to the 0th, 5th, 10th, 15th, 20th, 25th, 30th, 35th, 40th, 45th, and 50th seconds, a curve 60b is redrawn. Compared to the curve 60a, the curve 60b is smoother because it has more coordinate points. It should be noted that Figure 6 The various numerical values ​​and curves provided in the corresponding embodiments are examples provided for ease of understanding and have no actual reference significance.

[0136] In an embodiment of the present application, a camera device can capture an image of a vehicle traveling on a road surface, wherein the image includes lane markings. Subsequently, a first angular velocity change curve of the vehicle in a first time interval can be determined based on the lane markings in the image. A sensor in the vehicle can also detect a set of angular velocities of the vehicle, and based on this set of angular velocities, a second angular velocity change curve of the vehicle in a second time interval can be determined. The curve correlation degree (i.e., the degree of curve correlation) between the first angular velocity change curve and the second angular velocity change curve can be calculated to perform time calibration between the camera device and the sensor. As can be seen, the present application does not require additional equipment or a special external environment when performing time calibration between the camera device and the sensor, only lane markings are required, which is highly flexible and greatly reduces calibration costs. At the same time, by determining the curve correlation degree between the angular velocity change curve corresponding to the camera device and the angular velocity change curve corresponding to the sensor to perform time calibration, time calibration can be performed when the angular velocity captured by the camera device and the angular velocity detected by the sensor are highly correlated (i.e., the angular velocities are highly correlated), resulting in a highly accurate time calibration result. In summary, this application can improve calibration flexibility and accuracy and reduce calibration costs in scenarios where time calibration of camera devices and sensors is performed.

[0137] Further, see Figure 7 , Figure 7 This is a schematic diagram of a process for determining the offset angle of a vehicle provided by an embodiment of the present application. The process may correspond to the above Figure 5 In the corresponding embodiment, the process of determining the first offset angle of the vehicle corresponding to the first shooting moment is performed based on the lane line included in the first driving sub-image. Figure 7 The process may include at least the following steps S601 to S603:

[0138] Step S601 : Identify lane lines included in the first driving sub-image, and determine pixel coordinates of pixel points corresponding to the lane lines in an image coordinate system; the image coordinate system is the coordinate system corresponding to the first driving sub-image.

[0139] Specifically, the terminal device may have an image recognition function. Specifically, a pre-trained image recognition module may be deployed in the terminal device, and the lane lines in the driving image may be recognized through the image recognition module.

[0140] In practical applications, a large number of training images can be collected in advance. For example, these can include images with or without various lane markings. Lane markings can be applied to each training image, specifically by marking the positions of lane markings within the training image. These markings serve as the true lane label for the training image. This allows the deep learning network to be trained with the true lane label to yield a trained deep learning network (i.e., the aforementioned image recognition module).

[0141] It is understood that when a driving image is input into the image recognition module, the image recognition module can extract the features of each pixel in the driving image and determine the pixel points belonging to the lane line based on the pixel features. For example, a pixel point can be identified as belonging to a certain lane line in the driving image. Furthermore, if the driving image includes a dashed lane line, the image recognition module can also determine the pixel points corresponding to the blank positions in the dashed lane line as pixels belonging to the lane line based on the extracted pixel features. The blank positions can refer to the positions in the dashed lane line where the lane line is not drawn (i.e., the positions in the dashed lane line where the line is not drawn).

[0142] In practical applications, each pixel in a driving image can have pixel coordinates. An image coordinate system can be established, using any vertex of the image as the origin and the two intersecting image edges of that vertex as coordinate axes. Each pixel has a set of coordinate values ​​(e.g., (x, y)). The coordinate values ​​of a pixel in the image coordinate system are called pixel coordinates. The image recognition module can output the pixel coordinates of each pixel on each lane line.

[0143] In practical applications, the above-mentioned deep learning network can refer to a lane detection network (such as a LaneNet network). After training, the LaneNet network can accurately identify the pixels belonging to the same lane line and the pixel coordinates of each pixel.

[0144] Step S602 : performing coordinate conversion on the pixel coordinates according to the initial pitch angle value to obtain the spatial position coordinates of the pixel point in the world coordinate system.

[0145] Specifically, the world coordinate system can refer to the coordinate system of the vehicle. This world coordinate system can be a spatial coordinate system, and the origin of this world coordinate system can be the vertical projection position of the camera device on the road surface (which can be understood as the position directly below). When the camera device is mounted on the vehicle, there is actually a certain height (a certain distance, which can be understood as the vertical distance, i.e., the straight-line distance between the camera device and the road surface) between the camera device and the road surface (horizontal ground). Therefore, when converting pixel coordinates in the driving image to spatial coordinates in the world coordinate system, an initial pitch angle value (pitch angle parameter) of the camera device can be set, so that the spatial coordinates in the world coordinate system include the initial pitch angle parameter. Based on this, the spatial coordinates corresponding to each pixel can be determined based on the initial pitch angle value and the camera height (vertical distance). The specific method can be: obtaining the vertical distance corresponding to the camera device; where the vertical distance refers to the straight-line distance between the camera device and the road surface; determining the rotation matrix corresponding to the camera device based on the initial pitch angle value; and determining the spatial coordinates of the pixel point in the world coordinate system based on the rotation matrix, the vertical distance, and the pixel coordinates.

[0146] For ease of understanding, please refer to formula (1). The spatial position coordinates of the pixel point can be determined according to formula (1):

[0147]

[0148] Among them, P in formula (1) w It can be used to represent the pixel coordinates in the image coordinate system; h can be used to represent the vertical distance between the camera device and the horizontal ground; R in formula (1) can be used to represent the rotation matrix of the camera device, which can be calculated and determined by the camera extrinsic parameters of the camera device (such as pitch angle, camera installation height, etc.); K can be used to represent the camera intrinsic parameters of the camera device (such as focal length, distortion, etc.); x and y in formula (1) can form the spatial position coordinates.

[0149] Step S603 , performing straight line fitting on the spatial position coordinates to obtain a fitting line corresponding to the pixel point, and determining a first offset angle corresponding to the vehicle at the first shooting moment based on the fitting line.

[0150] Specifically, after determining the spatial coordinates corresponding to each pixel point on each lane line, these spatial coordinates can be fitted with a straight line to obtain a fitted straight line corresponding to each lane line. Based on the one or more fitted straight lines corresponding to each driving image (such as the first driving sub-image), the corresponding offset angle of the vehicle at each capture time (such as the first capture time) can be determined. Taking the first driving sub-image as an example, assuming that the first driving sub-image includes N lane lines, there will also be N fitted straight lines corresponding to the first driving sub-image. A specific method for determining the first offset angle corresponding to the vehicle at the first shooting moment based on N fitted straight lines may be as follows: the straight line slope corresponding to each of the N fitted straight lines may be obtained; then, the N straight line slopes may be sorted in order of size to obtain a straight line slope sequence; then, the first straight line slope and the second straight line slope may be obtained in sequence in the straight line slope sequence, and the fitted straight line corresponding to the first straight line slope may be determined as the first target fitted straight line, and the fitted straight line corresponding to the second straight line slope may be determined as the second target fitted straight line; based on the first target fitted straight line and the second target fitted straight line, the first offset angle corresponding to the vehicle at the first shooting moment may be determined.

[0151] Specifically, a specific implementation method for determining the first offset angle corresponding to the vehicle at the first shooting moment based on the first target fitting straight line and the second target fitting straight line may be: the vertical projection ground position of the camera device on the horizontal ground of the road may be determined, and the vertical projection ground position may be determined as the world coordinate origin of the world coordinate system; then, the intersection between the first target fitting straight line and the second target fitting straight line, as well as the distance between the intersection and the world coordinate origin may be determined; the straight line relationship between the first target fitting straight line and the second target fitting straight line is determined based on the distance; and the first offset angle corresponding to the vehicle at the first shooting moment is determined based on the straight line relationship.

[0152] Specifically, a specific implementation method for determining the first offset angle corresponding to the vehicle at the first shooting moment based on the straight line relationship may be: if the straight line relationship is a parallel relationship, the initial pitch angle value can be determined to be the correct pitch angle value, and the angle between the driving direction of the vehicle at the first shooting moment and the first target fitting straight line can be determined as the first offset angle; if the straight line relationship is a non-parallel relationship, the initial pitch angle value can be adjusted, and the pixel coordinates can be converted according to the adjusted pitch angle value to obtain the updated spatial position coordinates of the pixel point in the world coordinate system, and the updated spatial position coordinates can be fitted with a straight line to obtain the updated fitting straight line corresponding to the pixel point, and the first offset angle corresponding to the vehicle at the first shooting moment is determined based on the updated fitting straight line.

[0153] For easier understanding, please refer to Figure 8a , Figure 8aThis is a schematic diagram of a scenario for determining an offset angle provided by an embodiment of the present application. Figure 8a As shown, driving image 700a may be the first driving sub-image, and driving image 700a includes multiple lane lines. For driving image 700a, driving image 700a may be input into an image recognition module (e.g., a trained Lannet network), which may identify the pixels and pixel coordinates of each lane line. Subsequently, these pixels may be projected onto the horizontal ground (i.e., the road surface) according to the camera intrinsic parameters, camera extrinsic parameters, and installation angle (i.e., converting the pixel coordinates into control position coordinates). For ease of presentation, this application may calculate a top view after projection, as shown in top view image 700b. This top view includes pixels corresponding to multiple lane lines, and the arrow direction shown in image 700b may be the actual forward direction of the lane line.

[0154] Furthermore, multiple fitted lines can be obtained by fitting the spatial position coordinates of the pixels on the same lane line. For example, in image 700c, the fitted lines may include fitted line 700a, fitted line 700b, fitted line 700c, fitted line 700d, and fitted line 700e.

[0155] Further, please see Figure 8b , Figure 8b This is a schematic diagram of a scenario for determining an offset angle provided by an embodiment of the present application. Figure 8b As shown, in image 700c, the slope of each fitting straight line can be determined (for example, the straight line equation of each fitting straight line can be determined by the spatial position coordinates of the pixel points, and then the slope thereof can be determined based on the straight line equation). Among these slopes, the maximum slope and the second largest slope (only smaller than the maximum slope but larger than all other slopes) can be selected, and the fitting straight lines corresponding to the maximum slope and the second largest slope can be used as the first target fitting straight line and the second target fitting straight line.

[0156] For example, Figure 8bAs shown, the slope of fitted line 70d is the largest and the slope of fitted line 70c is the second largest. Fitted lines 70c and 70d can be selected as the first target fitted line and the second target fitted line, respectively. Furthermore, the intersection of fitted lines 70c and 70d can be determined, and the distance between the intersection and the world coordinate origin in the world coordinate system can be determined and recorded as distance 1. If the intersection does not exist, it can be recorded as 0 or a null value, and distance 1 is also null (or a negative value), which proves that fitted lines 70c and 70d are parallel. In actual three-dimensional space, fitted lines 70c and 70d should be parallel, indicating that the pitch angle value (i.e., the initial pitch angle value) of the camera device set above is correct, and the spatial position coordinates obtained therefrom are also correct values, thus obtaining a correct fitted line.

[0157] If distance 1 is a positive value, it proves that the intersection point exists and that the fitted lines 70c and 70d intersect. However, in actual three-dimensional space, the fitted lines 70c and 70d should be parallel. This proves that the pitch angle value of the camera device (i.e., the initial pitch angle value) is incorrect, and the spatial position coordinates obtained from it are also incorrect. In this case, the initial pitch angle value can be adjusted to obtain an adjusted pitch angle value. Subsequently, the pixel coordinates of the pixel point can be converted again using the adjusted pitch angle value to obtain new spatial position coordinates. From these new spatial position coordinates, new fitted lines 70c and 70d can also be obtained. It should be understood that the significance of adjusting the pitch angle value is to make the fitted lines 70c and 70d continuously approach a parallel relationship. At this point, the intersection point between the new fitted lines 70c and 70d can be determined again, and the distance between the new intersection point and the world coordinate origin can be obtained, which can be recorded as distance 2. If the intersection point does not exist, the new intersection point can be recorded as 0 or a null value (or positive infinity). Distance 2 is also null (or negative, positive infinity), proving that new fitted lines 70c and 70d are parallel. In actual three-dimensional space, fitted lines 70c and 70d should be parallel, indicating that the pitch angle value (i.e., the initial pitch angle value) of the camera device set above is correct, and the spatial position coordinates obtained from it are also correct, resulting in a correct fitted line. If distance 2 is a positive value, this indicates that new fitted lines 70c and 70d intersect. The pitch angle value can be adjusted again, and coordinate transformation can be performed again to obtain new spatial position coordinates. Linear fitting can then be performed again to obtain new lines 70c and 70d, until the distance determines that the final fitted lines 70c and 70d are parallel. At this point, the correct pitch angle value of the camera device is obtained.

[0158] After the fitting straight line 70c is parallel to the fitting straight line 70d, as shown in FIG. Figure 8b As shown, any one of the fitting lines 70c and 70d can be selected to determine the offset angle (which can be understood as the yaw angle of the vehicle compared to the lane line). Figure 8b As shown, the fitting straight line 70c can be selected to determine the offset angle of the vehicle, the driving direction of the vehicle at this time (such as the arrow direction in image 700c) can be obtained, and the angle value of the angle M between the driving direction and the fitting straight line 70c can be determined. The angle of the angle M can be the yaw angle (i.e., the offset angle) of the vehicle when the driving image 700a is taken.

[0159] It should be understood that in the embodiments of the present application, by acquiring driving images captured by a camera device, the vehicle's yaw angle and angular velocity can be calculated using the lane markings in the driving images. Furthermore, based on the correlation between the angular velocity of the camera device and the angular velocity of the inertial sensor, time calibration can be performed between the camera device and the inertial sensor. This eliminates the need for additional hardware and does not require high camera imaging quality; accurate time calibration can be performed simply by the presence of lane markings, providing greater flexibility.

[0160] Further, see Figure 9 , Figure 9 This is a structural diagram of a data processing device provided in an embodiment of the present application. The data processing device may be a computer program (including program code) running on a computer device, for example, the data processing device is an application software; the data processing device may be used to execute Figure 3 As shown in the method. Figure 9 As shown, the data processing device 1 may include: an image acquisition module 11 , a curve determination module 12 and a time calibration module 13 .

[0161] The image acquisition module 11 is used to acquire a driving image associated with the vehicle; the driving image is captured by a camera device configured on the vehicle; the driving image includes lane lines on the ground of the road on which the vehicle is traveling;

[0162] a curve determination module 12, configured to determine a first angular velocity change curve of the vehicle within a first time interval based on lane lines included in the driving image;

[0163] The curve determination module 12 is further configured to determine a second angular velocity change curve of the vehicle in the second time interval based on the set of angular velocities of the vehicle detected by the sensor in the second time interval;

[0164] The time calibration module 13 is used to perform time calibration on the camera device and the sensor according to the curve correlation between the first angular velocity change curve of the vehicle in the first time interval and the second angular velocity change curve of the vehicle in the second time interval.

[0165] The specific implementation of the image acquisition module 11, the curve determination module 12 and the time calibration module 13 can be found in the above Figure 2 The description of steps S101 to S104 in the corresponding embodiment will not be repeated here.

[0166] In one embodiment, the time calibration module 13 may include: a correlation matching unit 131 , a time updating unit 132 , and a time calibration unit 133 .

[0167] a correlation matching unit 131 for determining a curve correlation between a first angular velocity change curve in a first time interval and a second angular velocity change curve in a second time interval, and matching the curve correlation with a correlation threshold;

[0168] a time updating unit 132 configured to update the second time interval used for mapping the second angular velocity change curve if the curve correlation is less than a correlation threshold, to obtain an updated time interval for remapping the second angular velocity change curve; the duration of the updated time interval being equal to the duration of the second time interval;

[0169] A time calibration unit 133 is configured to perform time calibration on the camera device and the sensor based on the second angular velocity change curve of the vehicle in the update time interval and the first angular velocity change curve of the vehicle in the first time interval;

[0170] The time calibration unit 133 is further configured to use the time matching result as the time calibration result between the camera device and the sensor if the curve correlation degree is greater than the correlation degree threshold.

[0171] The specific implementation of the correlation matching unit 131, the time updating unit 132 and the time calibration unit 133 can be found in the above Figure 2 The description of step S104 in the corresponding embodiment will not be repeated here.

[0172] In one embodiment, the second angular velocity change curve is in a curvilinear coordinate system, and the curvilinear coordinate system includes a time axis;

[0173] The time updating unit 132 may include: a curve translation subunit 1321 , a timestamp obtaining subunit 1322 , and an interval determining subunit 1323 .

[0174] The curve translation subunit 1321 is used to obtain a translation amount and translate the second angular velocity change curve in the second time interval in the curve coordinate system according to the translation amount and the target translation direction; the target translation direction is parallel to the axis direction of the time axis;

[0175] The timestamp acquisition subunit 1322 is configured to acquire the starting position and the ending position of the second angular velocity variation curve after translation in the curve coordinate system, and acquire the starting timestamp corresponding to the starting position and the ending timestamp corresponding to the ending position of the curve on the time axis;

[0176] The interval determination subunit 1323 is configured to determine a time interval consisting of a start timestamp and an end timestamp as an update time interval.

[0177] The specific implementation of the curve translation subunit 1321, the timestamp acquisition subunit 1322 and the interval determination subunit 1323 can be found in the above Figure 2 The description of step S104 in the corresponding embodiment will not be repeated here.

[0178] In one embodiment, the time calibration unit 133 is further configured to determine an update curve correlation between the second angular velocity change curve in the update time interval and the first angular velocity change curve in the second time interval, and match the update curve correlation with a correlation threshold.

[0179] The time calibration unit 133 is further specifically configured to determine a time difference between the update time interval and the second time interval if the update curve correlation is greater than a correlation threshold, and determine the time difference as a time calibration result between the camera device and the sensor.

[0180] In one embodiment, the driving image includes a first driving image pair and a second driving image pair; the first driving image pair includes a first driving sub-image and a second driving sub-image; the second driving image pair includes a third driving sub-image and a fourth driving sub-image;

[0181] The curve determination module 12 may include an angular velocity determination unit 121 and a curve determination unit 122 .

[0182] An angular velocity determination unit 121 is configured to determine a first angular velocity of the vehicle at a first moment based on lane lines included in the first and second driving sub-images, respectively. The first moment is determined based on a first capturing moment and a second capturing moment, where the first capturing moment is when the camera device captures the first driving sub-image, and the second capturing moment is when the camera device captures the second driving sub-image.

[0183] The angular velocity determination unit 121 is further configured to determine a second angular velocity of the vehicle corresponding to a second moment based on the lane lines respectively included in the third and fourth driving sub-images; the second moment being determined based on a third capturing moment and a fourth capturing moment, where the third capturing moment is the moment when the camera device captured the third driving sub-image, and the fourth capturing moment is the moment when the camera device captured the fourth driving sub-image.

[0184] The curve determining unit 122 is configured to determine a first angular velocity change curve corresponding to the vehicle according to the first angular velocity, the second angular velocity, the first moment, and the second moment.

[0185] The specific implementation of the angular velocity determination unit 121 and the curve determination unit 122 can be found in the above Figure 2 The description of step S102 in the corresponding embodiment will not be repeated here.

[0186] In one embodiment, the angular velocity determination unit 121 may include: an offset determination subunit 1211 and an angular velocity determination subunit 1212 .

[0187] The offset determination subunit 1211 is configured to determine a first offset angle of the vehicle corresponding to the first capture time based on the lane line included in the first driving sub-image; the first offset angle is the offset angle between the vehicle's driving direction at the first capture time and the lane line;

[0188] The offset determination subunit 1211 is further configured to determine a second offset angle of the vehicle corresponding to the second shooting moment based on the lane line included in the second driving sub-image; the second offset angle is the offset angle between the driving direction of the vehicle at the second shooting moment and the lane line;

[0189] The angular velocity determination subunit 1212 is configured to determine a first angular velocity of the vehicle at the first moment according to the first offset angle, the second offset angle, the first shooting moment, and the second shooting moment.

[0190] The specific implementation of the offset determination subunit 1211 and the angular velocity determination subunit 1212 can be found in the above Figure 2 The description of step S102 in the corresponding embodiment will not be repeated here.

[0191] In one embodiment, the offset determination subunit 1211 is further configured to identify lane lines included in the first driving sub-image and determine pixel coordinates of pixel points corresponding to the lane lines in an image coordinate system; the image coordinate system is a coordinate system corresponding to the first driving sub-image;

[0192] The offset determination subunit 1211 is further specifically configured to perform coordinate conversion on the pixel coordinates according to the initial pitch angle value to obtain the spatial position coordinates of the pixel point in the world coordinate system;

[0193] The offset determination subunit 1211 is further specifically configured to perform straight line fitting on the spatial position coordinates to obtain a fitting straight line corresponding to the pixel point, and determine a first offset angle corresponding to the vehicle at the first shooting moment based on the fitting straight line.

[0194] In one embodiment, the offset determination subunit 1211 is further configured to obtain a vertical distance corresponding to the camera device; the vertical distance refers to a straight-line distance between the camera device and the road surface;

[0195] The offset determination subunit 1211 is further specifically used to determine the rotation matrix corresponding to the camera device according to the initial pitch angle value, and determine the spatial position coordinates of the pixel point in the world coordinate system according to the rotation matrix, vertical distance and pixel coordinates.

[0196] In one embodiment, the lane lines included in the first driving sub-image include N lane lines, and the first driving sub-image corresponds to N fitted straight lines;

[0197] The offset determination subunit 1211 is further specifically configured to obtain the slope of each of the N fitted straight lines.

[0198] The offset determination subunit 1211 is further specifically configured to sort the N straight line slopes in order of magnitude to obtain a straight line slope sequence;

[0199] The offset determination subunit 1211 is further specifically configured to sequentially obtain a first straight line slope and a second straight line slope in the straight line slope sequence, determine a fitting straight line corresponding to the first straight line slope as a first target fitting straight line, and determine a fitting straight line corresponding to the second straight line slope as a second target fitting straight line;

[0200] The offset determination subunit 1211 is further specifically configured to determine a first offset angle corresponding to the vehicle at the first shooting moment according to the first target fitting straight line and the second target fitting straight line.

[0201] In one embodiment, the offset determination subunit 1211 is further specifically configured to determine a vertically projected ground position of the camera device on the horizontal ground of the road, and determine the vertically projected ground position as the world coordinate origin of the world coordinate system;

[0202] The offset determination subunit 1211 is further specifically configured to determine the intersection point between the first target fitting line and the second target fitting line, and the distance between the intersection point and the world coordinate origin;

[0203] The offset determination subunit 1211 is further specifically configured to determine a linear relationship between the first target fitting line and the second target fitting line according to the distance;

[0204] The offset determination subunit 1211 is further specifically configured to determine a first offset angle corresponding to the vehicle at the first shooting moment according to the straight line relationship.

[0205] In one embodiment, the offset determination subunit 1211 is further configured to determine, if the straight line relationship is a parallel relationship, an initial pitch angle value as a correct pitch angle value, and determine the angle between the driving direction of the vehicle at the first shooting moment and the first target fitting line as a first offset angle;

[0206] The offset determination subunit 1211 is also specifically used to adjust the initial pitch angle value if the straight line relationship is a non-parallel relationship, convert the pixel coordinates according to the adjusted pitch angle value, obtain the updated spatial position coordinates of the pixel point in the world coordinate system, perform straight line fitting on the updated spatial position coordinates, obtain the updated fitting straight line corresponding to the pixel point, and determine the first offset angle corresponding to the vehicle at the first shooting moment based on the updated fitting straight line.

[0207] In one embodiment, the first shooting moment and the second shooting moment are adjacent shooting moments;

[0208] The angular velocity determination subunit 1212 is further specifically configured to determine an absolute value of an angle difference between the first offset angle and the second offset angle;

[0209] The angular velocity determination subunit 1212 is further specifically configured to obtain an intermediate shooting moment between the first shooting moment and the second shooting moment, and determine the intermediate shooting moment as the first moment;

[0210] The angular velocity determination subunit 1212 is further specifically configured to determine a shooting interval between the first shooting moment and the second shooting moment, and determine a first angular velocity of the vehicle corresponding to the first moment according to the absolute value of the angle difference and the shooting interval.

[0211] In one embodiment, the curve determining unit 122 may include: an initial curve determining subunit 1221 , a mean data determining subunit 1222 , and a curve determining subunit 1223 .

[0212] The initial curve determining subunit 1221 is configured to determine an initial angular velocity change curve corresponding to the vehicle according to the first moment, the first angular velocity, the second moment, and the second angular velocity;

[0213] The mean data determining subunit 1222 is configured to determine a mean time between the first time and the second time, and determine the mean angular velocity between the first angular velocity and the second angular velocity as the angular velocity of the vehicle at the mean time;

[0214] The curve determination subunit 1223 is configured to determine a first angular velocity change curve corresponding to the vehicle according to the first moment, the first angular velocity, the second moment, the second angular velocity, the average moment, and the average angular velocity.

[0215] The specific implementation of the initial curve determination subunit 1221, the mean data determination subunit 1222 and the curve determination subunit 1223 can be found in the above Figure 2 The description of step S102 in the corresponding embodiment will not be repeated here.

[0216] In an embodiment of the present application, a camera device can capture an image of a vehicle traveling on a road surface, wherein the image includes lane markings. Subsequently, a first angular velocity change curve of the vehicle in a first time interval can be determined based on the lane markings in the image. A sensor in the vehicle can also detect a set of angular velocities of the vehicle, and based on this set of angular velocities, a second angular velocity change curve of the vehicle in a second time interval can be determined. The curve correlation degree (i.e., the degree of curve correlation) between the first angular velocity change curve and the second angular velocity change curve can be calculated to perform time calibration between the camera device and the sensor. As can be seen, the present application does not require additional equipment or a special external environment when performing time calibration between the camera device and the sensor, only lane markings are required, which is highly flexible and greatly reduces calibration costs. At the same time, by determining the curve correlation degree between the angular velocity change curve corresponding to the camera device and the angular velocity change curve corresponding to the sensor to perform time calibration, time calibration can be performed when the angular velocity captured by the camera device and the angular velocity detected by the sensor are highly correlated (i.e., the angular velocities are highly correlated), resulting in a highly accurate time calibration result. In summary, this application can improve calibration flexibility and accuracy and reduce calibration costs in scenarios where time calibration of camera devices and sensors is performed.

[0217] Further, see Figure 10 , Figure 10 This is a schematic diagram of the structure of a computer device provided in an embodiment of the present application. Figure 10 As shown above Figure 9 The device 1 in the corresponding embodiment can be applied to the above-mentioned computer device 8000, which may include: a processor 8001, a network interface 8004 and a memory 8005. In addition, the above-mentioned computer device 8000 also includes: a user interface 8003, and at least one communication bus 8002. The communication bus 8002 is used to realize the connection and communication between these components. The user interface 8003 may include a display screen (Display), a keyboard (Keyboard), and the user interface 8003 may optionally include a standard wired interface and a wireless interface. The network interface 8004 may optionally include a standard wired interface and a wireless interface (such as a WI-FI interface). The memory 8005 may be a high-speed RAM memory, or a non-volatile memory (non-volatile memory), such as at least one disk memory. The memory 8005 may optionally also be at least one storage device located away from the aforementioned processor 8001. As Figure 10As shown, the memory 8005 as a computer-readable storage medium may include an operating system, a network communication module, a user interface module, and a device control application.

[0218] exist Figure 10 In the computer device 8000 shown, the network interface 8004 can provide network communication functions; the user interface 8003 is mainly used to provide an interface for user input; and the processor 8001 can be used to call the device control application stored in the memory 8005 to achieve:

[0219] Acquire a driving image associated with a vehicle; the driving image is captured by a camera device configured on the vehicle; the driving image includes lane lines on the ground of the road on which the vehicle is traveling;

[0220] determining a first angular velocity change curve of the vehicle within a first time interval based on lane lines included in the driving image;

[0221] determining a second angular velocity change curve of the vehicle in the second time interval based on the angular velocity set of the vehicle detected by the sensor in the second time interval;

[0222] The camera device and the sensor are time-calibrated according to a curve correlation between the first angular velocity change curve of the vehicle in the first time interval and the second angular velocity change curve of the vehicle in the second time interval.

[0223] It should be understood that the computer device 8000 described in the embodiment of the present application can execute the above Figures 2 to 7 The description of the data processing method in the corresponding embodiment can also be performed as described above. Figure 9 The description of the data processing device 1 in the corresponding embodiment will not be repeated here. In addition, the description of the beneficial effects of adopting the same method will not be repeated here either.

[0224] In addition, it should be noted that: the embodiment of the present application also provides a computer-readable storage medium, and the computer-readable storage medium stores a computer program executed by the computer device 1000 for data processing mentioned above, and the computer program includes program instructions. When the processor executes the program instructions, the computer program can execute the above-mentioned data processing. Figures 2 to 7 The description of the above-mentioned data processing method in the corresponding embodiment will therefore not be repeated here. In addition, the description of the beneficial effects of adopting the same method will not be repeated. For technical details not disclosed in the computer-readable storage medium embodiment involved in this application, please refer to the description of the method embodiment of this application.

[0225] The computer-readable storage medium may be the data processing device provided in any of the aforementioned embodiments or the internal storage unit of the computer device, such as the hard disk or memory of the computer device. The computer-readable storage medium may also be an external storage device of the computer device, such as a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, etc. equipped on the computer device. Furthermore, the computer-readable storage medium may also include both the internal storage unit of the computer device and an external storage device. The computer-readable storage medium is used to store the computer program and other programs and data required by the computer device. The computer-readable storage medium may also be used to temporarily store data that has been output or is to be output.

[0226] In one aspect of the present application, a computer program product or computer program is provided. The computer program product or computer program includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the method provided in one aspect of the embodiments of the present application.

[0227] The terms "first", "second", etc. in the description, claims, and drawings of the embodiments of the present application are used to distinguish different objects, rather than to describe a specific order. In addition, the terms "comprising" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, apparatus, product, or device comprising a series of steps or units is not limited to the listed steps or modules, but may optionally include steps or modules not listed, or may optionally include other step units inherent to these processes, methods, apparatuses, products, or devices.

[0228] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in terms of function in the above description. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0229] The methods and related devices provided by the embodiments of the present application are described with reference to the method flow charts and / or structural diagrams provided by the embodiments of the present application. Specifically, each process and / or block in the method flow charts and / or structural diagrams, as well as the combination of processes and / or blocks in the flow charts and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the steps in the process. Figure 1 Schematic diagram of one or more processes and / or structures Figure 1 These computer program instructions can also be stored in a computer-readable memory that can guide a computer or other programmable data processing device to work in a specific way, so that the instructions stored in the computer-readable memory produce a product including the instruction device, which implements the function specified in the process. Figure 1 Schematic diagram of one or more processes and / or structures Figure 1 These computer program instructions can also be loaded onto a computer or other programmable data processing device, so that a series of operation steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process in the process. Figure 1 The flow or flows and / or structures illustrate the steps of the functions specified in one block or multiple blocks.

[0230] The above disclosure is only a preferred embodiment of the present application, and certainly cannot be used to limit the scope of rights of the present application. Therefore, equivalent changes made according to the claims of the present application are still within the scope covered by the present application.

Claims

1. A data processing method, characterized in that: include: Acquiring a driving image associated with a vehicle; the driving image is captured by a camera device configured on the vehicle; The driving image includes lane lines on the road surface on which the vehicle is traveling, and at least two lane lines on the road surface in the driving image are in a parallel relationship; determining a first angular velocity change curve of the vehicle within a first time interval based on lane lines included in the driving image; capturing multiple driving images front and back within the first time interval, and determining angular velocities at each moment within the first time interval based on two temporally adjacent driving images captured front and back, the first angular velocity change curve within the first time interval being obtained based on the angular velocities at each moment within the first time interval, the angular velocity at the first moment within the first time interval being obtained based on an offset angle determined based on lane lines included in a driving image in a first driving image pair corresponding to the first moment; the offset angle being an offset angle between a driving direction of the vehicle and a lane line; determining a second angular velocity change curve of the vehicle in the second time interval based on the angular velocity set of the vehicle detected by the sensor in the second time interval; performing time calibration on the camera device and the sensor according to a curve correlation between the first angular velocity change curve of the vehicle in the first time interval and the second angular velocity change curve of the vehicle in the second time interval; Among them, the first offset angle of the vehicle is determined according to the lane line included in the first driving sub-image of the first driving image pair corresponding to the first moment in the first time interval, including: identifying the lane line included in the first driving sub-image, and determining the pixel coordinates of the pixel point corresponding to the lane line in the image coordinate system; the image coordinate system is the coordinate system corresponding to the first driving sub-image; the pixel coordinates are converted according to the initial pitch angle value to obtain the spatial position coordinates of the pixel point in the world coordinate system; the spatial position coordinates are fitted with a straight line to obtain a fitting straight line corresponding to the pixel point, and the first offset angle of the vehicle corresponding to the first shooting moment is determined according to the fitting straight line.

2. The method according to claim 1, characterized in that The performing time calibration on the camera device and the sensor according to a curve correlation between the first angular velocity change curve of the vehicle in the first time interval and the second angular velocity change curve of the vehicle in the second time interval includes: determining a curve correlation between the first angular velocity change curve in the first time interval and the second angular velocity change curve in the second time interval, and matching the curve correlation with a correlation threshold; If the curve correlation is less than the correlation threshold, the second time interval used for mapping the second angular velocity change curve is time-updated to obtain an updated time interval for remapping the second angular velocity change curve, and the camera device and the sensor are time-calibrated according to the second angular velocity change curve of the vehicle in the updated time interval and the first angular velocity change curve of the vehicle in the first time interval; the duration of the updated time interval is equal to the duration of the second time interval; If the curve correlation degree is greater than the correlation degree threshold, the time matching result is used as the time calibration result between the camera device and the sensor.

3. The method according to claim 2, characterized in that The second angular velocity change curve is in a curvilinear coordinate system, and the curvilinear coordinate system includes a time axis; The updating of the second time interval for mapping the second angular velocity change curve to obtain an updated time interval for remapping the second angular velocity change curve includes: Obtaining a translation amount, and translating the second angular velocity change curve in the second time interval in the curvilinear coordinate system according to the translation amount and a target translation direction; the target translation direction is parallel to the axis direction of the time axis; Acquire a curve starting position and a curve ending position of the second angular velocity change curve after translation in the curve coordinate system, and acquire a start timestamp corresponding to the curve starting position and an end timestamp corresponding to the curve ending position on the time axis; A time interval formed by the start timestamp and the end timestamp is determined as the update time interval.

4. The method according to claim 2, characterized in that The performing time calibration on the camera device and the sensor according to the second angular velocity change curve of the vehicle in the update time interval and the first angular velocity change curve of the vehicle in the first time interval includes: determining an update curve correlation between the second angular velocity change curve within the update time interval and the first angular velocity change curve within the first time interval, and matching the update curve correlation with the correlation threshold; If the update curve correlation is greater than the correlation threshold, a time difference between the update time interval and the second time interval is determined, and the time difference is determined as a time calibration result between the camera device and the sensor.

5. The method according to claim 1, wherein The driving image includes a first driving image pair and a second driving image pair; the first driving image pair includes a first driving sub-image and a second driving sub-image; the second driving image pair includes a third driving sub-image and a fourth driving sub-image; The determining, based on the lane lines included in the driving image, a first angular velocity change curve of the vehicle in a first time interval, includes: determining, based on lane lines respectively included in the first driving sub-image and the second driving sub-image, a first angular velocity of the vehicle corresponding to a first moment; the first moment being determined based on a first capturing moment and a second capturing moment, the first capturing moment being the moment when the camera device captured the first driving sub-image, and the second capturing moment being the moment when the camera device captured the second driving sub-image; determining, based on lane lines respectively included in the third driving sub-image and the fourth driving sub-image, a second angular velocity of the vehicle corresponding to a second moment; the second moment being determined based on a third capturing moment and a fourth capturing moment, the third capturing moment being the moment when the camera device captured the third driving sub-image, and the fourth capturing moment being the moment when the camera device captured the fourth driving sub-image; A first angular velocity change curve corresponding to the vehicle is determined according to the first angular velocity, the second angular velocity, the first moment, and the second moment.

6. The method according to claim 5, characterized in that The determining, based on lane lines respectively included in the first driving sub-image and the second driving sub-image, of a first angular velocity of the vehicle at a first moment includes: determining, based on the lane line included in the first driving sub-image, a first offset angle corresponding to the vehicle at the first shooting moment; the first offset angle being the offset angle between the driving direction of the vehicle at the first shooting moment and the lane line; determining, based on the lane line included in the second driving sub-image, a second offset angle corresponding to the vehicle at the second shooting moment; the second offset angle being an offset angle between the driving direction of the vehicle at the second shooting moment and the lane line; A first angular velocity of the vehicle at a first moment is determined according to the first offset angle, the second offset angle, the first shooting time, and the second shooting time.

7. The method according to claim 6, characterized in that The step of performing coordinate conversion on the pixel coordinates according to the initial pitch angle value to obtain the spatial position coordinates of the pixel point in the world coordinate system includes: Obtaining a vertical distance corresponding to the camera device; the vertical distance refers to a straight-line distance between the camera device and the road surface; The rotation matrix corresponding to the camera device is determined according to the initial pitch angle value, and the spatial position coordinates of the pixel point in the world coordinate system are determined according to the rotation matrix, the vertical distance and the pixel coordinates.

8. The method according to claim 6, characterized in that The lane lines included in the first driving sub-image include N lane lines, and the first driving sub-image corresponds to N fitted straight lines; The determining, based on the fitted straight line, a first offset angle corresponding to the vehicle at the first shooting moment, includes: Obtaining the slope of each of the N fitted straight lines; Sort the N straight line slopes in order of size to obtain a straight line slope sequence; Sequentially acquiring a first straight line slope and a second straight line slope from the straight line slope sequence, determining a fitting straight line corresponding to the first straight line slope as a first target fitting straight line, and determining a fitting straight line corresponding to the second straight line slope as a second target fitting straight line; A first offset angle corresponding to the vehicle at the first shooting moment is determined according to the first target fitting straight line and the second target fitting straight line.

9. The method according to claim 8, wherein determining the first offset angle corresponding to the vehicle at the first shooting moment based on the first target fitting line and the second target fitting line comprises: Determine a vertical projection ground position of the camera device on the horizontal ground of the road, and determine the vertical projection ground position as the world coordinate origin of the world coordinate system; Determine an intersection point between the first target fitting straight line and the second target fitting straight line, and a distance between the intersection point and the world coordinate origin; determining a linear relationship between the first target fitting straight line and the second target fitting straight line according to the distance; A first offset angle corresponding to the vehicle at the first shooting moment is determined according to the straight line relationship.

10. The method according to claim 9, characterized in that The determining, based on the straight line relationship, a first offset angle corresponding to the vehicle at the first shooting moment includes: If the straight line relationship is a parallel relationship, determining the initial pitch angle value as a correct pitch angle value, and determining the angle between the driving direction of the vehicle at the first shooting moment and the first target fitting line as the first offset angle; If the straight line relationship is a non-parallel relationship, the initial pitch angle value is adjusted, and the pixel coordinates are converted according to the adjusted pitch angle value to obtain the updated spatial position coordinates of the pixel point in the world coordinate system. The updated spatial position coordinates are linearly fitted to obtain the updated fitting straight line corresponding to the pixel point, and the first offset angle corresponding to the vehicle at the first shooting moment is determined based on the updated fitting straight line.

11. The method according to claim 6, characterized in that The first shooting moment and the second shooting moment are adjacent shooting moments; The determining, based on the first offset angle, the second offset angle, the first shooting time, and the second shooting time, of a first angular velocity of the vehicle at a first moment includes: determining an absolute value of an angle difference between the first offset angle and the second offset angle; Obtaining an intermediate shooting moment between the first shooting moment and the second shooting moment, and determining the intermediate shooting moment as the first moment; A shooting interval duration between the first shooting moment and the second shooting moment is determined, and a first angular velocity corresponding to the vehicle at the first moment is determined according to the absolute value of the angle difference and the shooting interval duration.

12. The method according to claim 5, characterized in that The determining, according to the first angular velocity, the second angular velocity, the first moment, and the second moment, of a first angular velocity change curve corresponding to the vehicle includes: determining an initial angular velocity change curve corresponding to the vehicle according to the first moment, the first angular velocity, the second moment, and the second angular velocity; determining a mean time between the first time and the second time, and determining the mean angular velocity between the first angular velocity and the second angular velocity as the angular velocity of the vehicle corresponding to the mean time; A first angular velocity change curve corresponding to the vehicle is determined according to the first moment, the first angular velocity, the second moment, the second angular velocity, the average moment, and the average angular velocity.

13. A computer device, characterized in that: include: processor, memory, and network interface; The processor is connected to the memory and the network interface, wherein the network interface is used to provide a network communication function, the memory is used to store program code, and the processor is used to call the program code so that the computer device executes the method according to any one of claims 1 to 12.

14. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, and the computer program is suitable for being loaded by a processor and executing the method according to any one of claims 1 to 12.

15. A computer program product, characterized in that The computer program product includes computer instructions, which are stored in a computer-readable storage medium. The computer instructions are suitable for being read and executed by a processor, so as to enable a computer device having the processor to perform the method according to any one of claims 1 to 12.

Images