Vehicle state recognition method, electronic device, and computer-readable storage medium

Abstract

The application discloses a vehicle state recognition method, an electronic device and a computer readable storage medium. The method comprises the following steps: acquiring a first video frame sequence, wherein the first video frame sequence comprises a plurality of target video frames containing a target vehicle; determining whether reference time information exists based on posture information of the target vehicle in the first video frame sequence, wherein the reference time information is time information in which the target vehicle is suspected to be in a first target state; in response to the existence of the reference time information, searching for a second video frame sequence related to the reference time information from the first video frame sequence; and determining whether the target vehicle is in the first target state based on state reference information of the target vehicle in the second video frame sequence. In this way, the accuracy of the state determination result of the target vehicle in the first target state can be improved.

Description

Technical Field

[0001] This application relates to the field of video data processing, and in particular to a vehicle status recognition method, electronic device, and computer-readable storage medium. Background Technology

[0002] In traffic monitoring, it's often necessary to use video frame sequences to identify vehicle states and determine their behavior, thereby improving traffic safety. For example, if a vehicle needs to stop before turning, the video frame sequence can be used to identify its state and determine if it stopped before turning. Similarly, in a rear-end collision, the video frame sequences of the two vehicles prior to the collision can be used to identify their states and determine which vehicle caused the collision. Furthermore, video sequences of vehicles in no-parking zones can be used to determine if they are illegally parked.

[0003] However, current vehicle state recognition methods do not provide accurate vehicle state information. Summary of the Invention

[0004] This application provides a vehicle state recognition method, an electronic device, and a computer-readable storage medium, which can solve the problem that the accuracy of the vehicle state obtained by the current vehicle state recognition method is not high.

[0005] To address the aforementioned technical problems, this application provides a vehicle state recognition method. The method includes: acquiring a first video frame sequence, wherein the first video frame sequence includes a plurality of target video frames containing a target vehicle; determining, based on the pose information of the target vehicle in the first video frame sequence, whether there is reference time information indicating that the target vehicle is suspected of being in a first target state; responding to the existence of reference time information, using the reference time information to find a second video frame sequence related to the reference time information from the first video frame sequence; and determining, based on the state reference information of the target vehicle in the second video frame sequence, whether the target vehicle is in the first target state.

[0006] To solve the above-mentioned technical problems, another technical solution adopted in this application is: to provide an electronic device, which includes a processor and a memory connected to the processor, wherein the memory stores program instructions; the processor is used to execute the program instructions stored in the memory to implement the above-mentioned method.

[0007] To solve the above-mentioned technical problems, another technical solution adopted in this application is to provide a computer-readable storage medium storing program instructions that, when executed, can implement the above-mentioned method.

[0008] By employing the above method, this application does not directly determine whether the target vehicle is in the first target state based on the target vehicle's attitude information in the first video frame sequence. Instead, it determines whether reference time information exists based on the target vehicle's attitude information in the first video frame sequence. The reference time information is the time information at which the target vehicle is suspected of being in the first target state. The existence of reference time information means that the target vehicle is suspected of being in the first target state. Therefore, when reference time information exists, a second video frame sequence related to the reference time information is found from the second video frame sequence. Based on the target vehicle's state reference information in the second video frame sequence, it is determined whether the target vehicle is actually in the first target state. Therefore, this application can provide accurate results for determining the state of the target vehicle. Attached Figure Description

[0009] Figure 1 This is a flowchart illustrating an embodiment of the vehicle state recognition method of this application;

[0010] Figure 2 yes Figure 1 A detailed flowchart of the S11 process;

[0011] Figure 3 This is a flowchart illustrating another embodiment of the vehicle state recognition method of this application;

[0012] Figure 4 This is a schematic diagram of the head module;

[0013] Figure 5 This is a schematic diagram of the key points of a large truck;

[0014] Figure 6 yes Figure 3 A detailed flowchart of the S22 process;

[0015] Figure 7 This is a schematic diagram showing the correspondence between the world coordinate system, camera coordinate system, and image coordinate system;

[0016] Figure 8 This is a detailed flowchart of S14;

[0017] Figure 9 This is a flowchart illustrating yet another embodiment of the vehicle state recognition method of this application;

[0018] Figure 10 This is a flowchart illustrating another embodiment of the vehicle state recognition method of this application;

[0019] Figure 11 This is a flowchart illustrating a specific example of the vehicle state recognition method of this application;

[0020] Figure 12 This is a schematic diagram of the structure of an embodiment of the electronic device of this application;

[0021] Figure 13 This is a schematic diagram of the structure of an embodiment of the computer-readable storage medium of this application. Detailed Implementation

[0022] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of the embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of this application.

[0023] The terms "first," "second," and "third" used in this application are for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Therefore, a feature defined as "first," "second," or "third" may explicitly or implicitly include at least one of that feature. In the description of this application, "multiple" means at least two, such as two, three, etc., unless otherwise explicitly specified.

[0024] In this document, the term "embodiment" means that a particular feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of this application. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a separate or alternative embodiment mutually exclusive with other embodiments. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described herein may be combined with other embodiments without conflict.

[0025] The following details the application scenarios of several vehicle state recognition methods:

[0026] Application Scenario 1: Determining whether a vehicle stops before turning. Acquire the vehicle's video frame sequence; based on the video frame sequence, determine whether the vehicle was initially stationary and then changed to a turning state; if so, the vehicle stops before turning, and the behavior is normal; otherwise, the vehicle does not stop before turning, and the behavior is abnormal.

[0027] Application Scenario 2: Determining the responsible vehicle in a rear-end collision. After a rear-end collision, acquire the video frame sequence of the two vehicles involved. Based on the video frame sequence, determine whether the two vehicles were stationary or moving before the accident. If the vehicle in front was stationary and the vehicle behind was moving, and the scene is a no-parking zone, then the vehicle in front is determined to be the responsible vehicle.

[0028] Application Scenario 3: Determination of Illegal Parking. Acquire video frame sequences of vehicles in the illegal parking area. Based on the video frame sequence, determine whether the vehicle is in motion or stationary. If in motion, determine that the vehicle is moving and no illegal parking has occurred; if stationary, determine that the vehicle is parked and an illegal parking has occurred.

[0029] The vehicle status recognition method provided in this application is described below:

[0030] Figure 1 This is a flowchart illustrating an embodiment of the vehicle state recognition method of this application. It should be noted that if substantially the same result is achieved, this embodiment is not necessarily identical. Figure 1 The sequence of processes shown is limited.

[0031] like Figure 1 As shown, this embodiment may include:

[0032] S11: Obtain the first video frame sequence.

[0033] The first video frame sequence includes several target video frames containing the target vehicle.

[0034] The vehicle state recognition method provided in this application can be applied to any type of vehicle, such as trucks, cars, motorcycles, etc. Each target video frame in the first video frame sequence corresponds to a shooting time. Cameras are deployed in the state monitoring area to capture images of vehicles within the area, obtaining the original video frame sequence of the state monitoring area. Vehicle tracking is performed on the original video frame sequence to obtain the trajectory information of each vehicle. At least one vehicle is designated as the target vehicle, and based on the trajectory information of the target vehicle, target video frames containing the target vehicle are selected from the original video frame sequence to obtain the first video frame sequence.

[0035] The status monitoring area depends on the application scenario. For example, in the aforementioned application scenario 1, the status monitoring area can be the area of ​​electronic police or checkpoints; in the aforementioned application scenario 2, the status monitoring area can be the area of ​​a rear-end collision; and in the aforementioned application scenario 3, the status monitoring area can be the area of ​​illegal parking.

[0036] Vehicle trajectory information can include vehicle identification information, vehicle category information, vehicle location information, the video frame number to which the vehicle's location information belongs, timestamp information, etc. The vehicle's location information can include the first location information of the vehicle region in the original video frame, and can also include the second location information of key points of the vehicle in the original video frame. The vehicle region can be a connected component, and the first location information can be a mask of the connected component. Alternatively, the vehicle region can be a bounding box, and the first location information is (the length, width, and coordinates of the bounding box). The second location information can be the coordinates of the key points.

[0037] S12: Based on the attitude information of the target vehicle in the first video frame sequence, determine whether there is reference time information indicating that the target vehicle is suspected to be in the first target state.

[0038] Attitude information can include at least one of rotational attitude information and translational attitude information. Rotational attitude information can include at least one of the target vehicle's pitch, yaw, and roll angles. An XYZ coordinate system can be established centered on the target vehicle, where the X, Y, and Z axes are perpendicular to each other. The positive X-axis direction is from the front to the rear of the vehicle, the positive Y-axis direction is from the left to the right side of the vehicle, and the positive Z-axis direction is from the bottom to the top of the vehicle. The X, Y, and Z axes represent the target vehicle's straight-ahead orientation (i.e., orientation in a straight-ahead state). The pitch angle is the angle by which the target vehicle's actual orientation rotates relative to the X-axis direction, the yaw angle is the angle by which the target vehicle's actual orientation rotates relative to the Y-axis direction, and the roll angle is the angle by which the target vehicle's actual orientation rotates relative to the Z-axis direction. Translational attitude information can include the target vehicle's translational displacement, keypoint translational displacement, etc.

[0039] The target vehicle's state can include a stationary state and motion-related states. Motion-related states can include turning, lane changing, acceleration, deceleration, straight-line driving, and overtaking states. The target vehicle's attitude information differs to some extent in different states. For example, a stationary state has little or no change in translational / rotational attitude information; straight-line driving, acceleration, and deceleration states have significant changes in translational attitude information but little or no change in rotational attitude information; and turning, lane changing, and overtaking states have significant changes in rotational attitude information. The first target state can be one of the aforementioned target vehicle states. Therefore, based on the changes in attitude information, the time information at which the target vehicle is suspected to be in the first target state can be preliminarily determined, i.e., reference time information.

[0040] The reference time information can be a time period or a single moment. It can be the start time, end time, midpoint, or the entire duration of the target vehicle's suspected first target state.

[0041] In some embodiments, a target video frame can be determined from the first video frame sequence, and reference time information can be determined based on the shooting time of the determined target video frame. For example, if the reference time information is a start time, it can start from the first target video frame in the first video frame sequence, sequentially traversing each target video frame in the first video frame sequence, and determining whether the change in the target vehicle's attitude information in the target video frame and the predetermined number of target video frames thereafter meets the attitude change condition. If the attitude change condition is met, the shooting time of the target video frame is used as the reference time information. Alternatively, if the reference time information is an end time, it can start from the last target video frame in the first video frame sequence, sequentially traversing each target video frame in the first video frame sequence, and determining whether the target vehicle's attitude information in the target video frame and the predetermined number of target video frames before it meets the attitude change condition. If the attitude change condition is met, the shooting time of the target video frame is used as the reference time information. For a description of the attitude change condition, please refer to the relevant explanation in S121.

[0042] In some embodiments, a video frame sequence can be determined from a first video frame sequence, and reference time information can be determined based on the capture time of at least one target video frame in the determined video frame sequence. In this case, refer to... Figure 2 S12 may include the following sub-steps:

[0043] S121: Based on the changes in the attitude information of the target vehicle in the first video frame sequence, determine whether a third video frame sequence exists.

[0044] Among them, the change in attitude information corresponding to the third video frame sequence satisfies the attitude change condition.

[0045] The attitude change condition can be determined based on the actual attitude change attributes of the first target state. For example, if the first target state is a state with large attitude information changes, the attitude change condition could be that the attitude information of the target vehicle in each target video frame of the third video frame sequence increases sequentially and is greater than an attitude threshold; or it could be that the attitude information in each target video frame of the third video frame sequence decreases sequentially and is greater than an attitude threshold; or it could be that the difference between different target video frames in the third video frame sequence is greater than an attitude difference threshold. Alternatively, if the first target state is a state with no attitude information changes or small attitude information changes, the attitude change condition could be that the difference between the attitude information of the target vehicle in different target video frames of the third video frame sequence is less than an attitude difference threshold.

[0046] Taking the first target state as a state with large changes in attitude information, and the attitude change condition as the attitude information of the target vehicle in each target video frame in the third video frame sequence increasing sequentially and exceeding the attitude threshold as an example, the attitude information of the target vehicle in each target video frame in the first video frame sequence can be compared sequentially; in response to the sequential increase of the attitude information of the target vehicle in a consecutive preset number of target video frames, and all exceeding the attitude threshold, it is determined that a third video frame sequence exists, and the preset number of target video frames constitute the third video frame sequence.

[0047] If a third video frame sequence exists, execute S122; if a third video frame sequence does not exist, execute S123.

[0048] S122: Reference time information is confirmed to exist.

[0049] The reference time information is determined based on the capture time of at least one target video frame in the third video frame sequence.

[0050] The reference time information may be the shooting time of a target video frame in the third video frame sequence, such as the shooting time of the first target video frame or the shooting time of the last target video frame; or the reference time information may be the shooting period of the third video frame sequence; or the reference time information may include the shooting period of the third video frame sequence and the extended period before / after it.

[0051] S123: It has been determined that there is no reference time information and the target vehicle is not in the first target state.

[0052] If reference time information exists, execute S13 to S14; if reference time information does not exist, execute S15.

[0053] S13: Find the second video frame sequence that is related to the reference time information from the first video frame sequence.

[0054] In some embodiments, the second video frame sequence consists of target video frames in the first video frame sequence whose time difference between the shooting time and the reference time information is less than a time difference threshold.

[0055] For example, the time difference threshold is three consecutive shooting moments. The reference time information is the start moment; the first video frame sequence includes 10 video frames, and the reference time information is the shooting moment of the second video frame. The second video frame sequence consists of the second to fifth video frames. Alternatively, the reference time information is the end moment; the first video frame sequence includes 10 video frames, and the reference time information is the shooting moment of the fifth video frame. The second video frame sequence consists of the second to fifth video frames. Yet another example is the middle moment; the first video frame sequence includes 10 video frames, and the reference time information is the shooting moment of the third video frame. The second video frame sequence consists of the second to fifth video frames.

[0056] In some embodiments, the second video frame sequence consists of several target video frames corresponding to the reference time information in the first video frame sequence. For example, the reference time information is the entire time period, the first video frame sequence includes 10 video frames, the reference time information corresponds to the 2nd to 5th video frames, and the second video frame sequence consists of the 2nd to 5th video frames.

[0057] S14: Based on the state reference information of the target vehicle in the second video frame sequence, determine whether the target vehicle is in the first target state.

[0058] State reference information may include at least one sub-state reference information, such as attitude information and velocity information. Velocity information may include vehicle speed, vehicle acceleration, etc.

[0059] If the state reference information includes only attitude information, or includes both attitude and velocity information, compared to S12, it can be understood that, based on determining that the target vehicle is suspected to be in the first target state, S14 uses a more refined and complex algorithm to determine whether the target vehicle is actually in the first target state. If the state reference information includes only velocity information, compared to S12, it can be understood that, based on determining that the target vehicle is suspected to be in the first target state, S14 uses an information dimension different from attitude information to determine whether the target vehicle is actually in the first target state.

[0060] State reference information can be determined based on the relevant position information of the target vehicle. For example, regarding vehicle speed, the relevant position information of the target vehicle in each target video frame can be determined based on the target vehicle's trajectory information. Based on the relevant position information of the target vehicle in each target video frame, the displacement of the target vehicle between different target video frames can be obtained. Then, combined with the shooting time difference of different target video frames, the vehicle speed can be calculated.

[0061] In some embodiments, the first target state is an acceleration state, and whether the target vehicle is in the first target state can be determined based on the change in the vehicle speed of the target vehicle.

[0062] In some embodiments, the first target state is a steering state, and whether the target vehicle is in the first target state can be determined based on the changes in the yaw angle, pitch angle, roll angle, and vehicle speed of the target vehicle.

[0063] S15: Determined that the target vehicle is not in the first target state.

[0064] Understandably, because the differences in the target vehicle's attitude information are relatively small across different states, directly determining whether the target vehicle is in the first target state based on its attitude information in the first video frame sequence would lead to inaccurate determinations. For example, if the first target state is a turning state, the attitude information differences between the lane-changing and overtaking states are small. Directly determining whether the target vehicle is in a turning state based on its attitude information would result in inaccurate determinations due to interference from the lane-changing and overtaking states.

[0065] In this embodiment, instead of directly determining whether the target vehicle is in the first target state based on the attitude information of the target vehicle in the first video frame sequence, this application determines whether reference time information exists based on the attitude information of the target vehicle in the first video frame sequence. The reference time information is the time information at which the target vehicle is suspected to be in the first target state; the existence of reference time information means that the target vehicle is suspected to be in the first target state. Therefore, when reference time information exists, a second video frame sequence related to the reference time information is found from the second video frame sequence. Based on the state reference information of the target vehicle in the second video frame sequence, it is determined whether the target vehicle is indeed in the first target state. Therefore, this application can provide accurate results for determining the state of the target vehicle.

[0066] Figure 3 This is a flowchart illustrating another embodiment of the vehicle state recognition method of this application. It should be noted that if substantially the same result is achieved, this embodiment is not necessarily identical. Figure 3 The illustrated process sequence is limited. This embodiment is a further extension of S11. Figure 3 As shown, this embodiment may include:

[0067] S21: Perform vehicle detection on each original video frame in the original video frame sequence to obtain the relevant location information of each vehicle in each original video frame.

[0068] This can be achieved using any neural network model capable of vehicle detection. For example, a trained neural network model first detects the first location information of the vehicle region, and then performs keypoint detection on the vehicle region based on the first location information to obtain the second location information of the keypoints. Alternatively, the neural network model first detects the center point of the vehicle, and then offsets the center point to obtain the first location information of the vehicle region and the second location information of the keypoints.

[0069] Taking the Anchorfree-based CenterNet neural network model as an example, it consists of a Backbone module and a Head module. The Backbone module can map the original video frames to a high-dimensional feature space to obtain feature vectors. The Head module can predict the feature vectors to obtain the center point of the vehicle, and then offset the center point of the vehicle to obtain the first position information of the vehicle region and the second position information of the key points.

[0070] Figure 4 This is a diagram of the head module, as shown below. Figure 4 As shown, the Head module includes six branches during training: a center point heatmap branch, which outputs a center point heatmap to predict the vehicle's center point; a center point offset branch, which corrects the error introduced when mapping the center point from the center point heatmap back to the original video frame; a vehicle WH branch, which predicts the length and width of the vehicle region; a keypoint (Kp) offset branch, which predicts the offset distance of each keypoint relative to the vehicle's center point; a keypoint heatmap branch, which obtains the keypoint heatmap; and a keypoint offset branch, which corrects the error introduced when mapping the keypoint from the keypoint heatmap back to the original video frame.

[0071] The original video frames contain annotations indicating the vehicle's location. A loss function can be calculated based on the difference between the annotations and the detection results, and CenterNet can be optimized based on this loss function. Combined with... Figure 5 Provide explanations for the annotation information of key points. Figure 5 The key points of the truck shown include the center point of the left front wheel (A1), the center point of the right front wheel (A2), the center point of the left rear wheel (A3), the center point of the right rear wheel (A4), the left front vertex of the roof (B1), the right front vertex of the roof (B2), the left rear vertex of the roof (B3), the right rear vertex of the roof (B4), the center point of the left front headlight (C1), the center point of the right front headlight (C2), the center point of the left taillight (C3), the center point of the right taillight (C4), the left vertex of the lower boundary of the front of the vehicle (D1), the right vertex of the lower boundary of the front of the vehicle (D2), the left vertex of the lower boundary of the rear of the vehicle (D3), the right vertex of the lower boundary of the rear of the vehicle (D4), the upper left corner of the front window (E1), the upper right corner of the front window (E2), the lower left corner of the front window (E3), the lower right corner of the front window (E4), the center point of the left rearview mirror (F1), and the center point of the right rearview mirror (F2). It should be noted that the left and right mentioned in this application are based on the direction the vehicle driver is facing, and all key points are visible.

[0072] During training, the CenterNet loss is obtained by weighting the losses of the six branches:

[0073] Loss=λ hm ·L hm +λ off ·L off +λ wh ·L wh +λ kp ·L kp +λ hm_kp ·L hm_kp +λ kp_off ·L kp_off ;

[0074] Among them, L hm L off L wh L kp L hm_kp and L kp_off Let λ represent the losses for the center point heatmap branch, center point offset branch, vehicle WH branch, keypoint (Kp) offset branch, keypoint heatmap branch, and keypoint offset branch, respectively. hm , λ off , λ wh , λ kp , λ hm_kp , λ kp_off These represent the weights of different branches.

[0075] S22: Associate the relevant location information of each vehicle to obtain the trajectory information of each vehicle.

[0076] In some embodiments, the relevant location information of each vehicle can be associated based on the first location information to obtain the trajectory information of each vehicle. In this case, the matching degree of the first location information of each vehicle can be obtained, and the relevant location information of each vehicle can be associated based on the matching degree of the first location information. For example, if the matching degree of the first location information of vehicle A and the first location information of vehicle B is greater than the matching degree threshold, it is considered that the relevant location information of vehicle A matches the relevant location information of vehicle B.

[0077] In some embodiments, the relevant location information of each vehicle can be associated based on first location information and second location information to obtain the trajectory information of each vehicle. In this case, the matching degree of the first location information and the matching degree of the second location information of each vehicle can be obtained, and the relevant location information of each vehicle can be associated based on the matching degree of the first location information and the matching degree of the second location information. It is understood that associating the relevant location information of each vehicle based on the first location information and the second location information utilizes more location information than associating the relevant location information of each vehicle based solely on the first location information, can cope with more complex situations, and can reduce errors caused by the occlusion of the target vehicle to a certain extent. Specifically, refer to the following... Figure 6 S22 may include the following sub-steps:

[0078] S221: Sequentially take each original video frame as a video frame to be associated, take the vehicle in the video frame to be associated as the vehicle to be associated, and match the first position information of the vehicle to be associated with the trajectory information of each existing vehicle.

[0079] Each first location information in the trajectory information of an existing vehicle can be matched with the first location information of the vehicle to be associated, or only the latest first location information in the trajectory information of an existing vehicle can be matched with the first location information of the vehicle to be associated. The trajectory information of the existing vehicle with a matching degree higher than the matching degree threshold is used as the trajectory information of the existing vehicle to be matched.

[0080] S222: Determine whether there is existing vehicle trajectory information that matches the first location information of the vehicle to be associated.

[0081] If trajectory information of an existing vehicle that matches the first location information of the vehicle to be associated exists, execute S223; otherwise execute 227.

[0082] S223: Determine whether the number of existing vehicle trajectory information matches is one or more.

[0083] If the number of matching existing vehicle trajectory information is one, execute S224; if the number of matching existing vehicle trajectory information is multiple, execute S225 to S226.

[0084] S224: Update the trajectory information of existing vehicles using the relevant location information of the vehicles to be associated.

[0085] S225: Obtain the second distance between the second location information in the trajectory information of each existing vehicle and the second location information of the vehicle to be associated.

[0086] The distance between a key point and an existing vehicle and the vehicle to be associated can be calculated as the second distance. Alternatively, the distance between the second location information of multiple key points and the vehicle to be associated can be calculated, and the average of the distances between the multiple key points can be used as the second distance.

[0087] S226: Using the relevant location information of the vehicle to be associated, update the trajectory information of existing vehicles that meet the distance requirements for the second distance.

[0088] Distance requirements can include the second maximum distance, etc.

[0089] S227: Establish new vehicle trajectory information based on the first location information of the vehicle to be associated.

[0090] The following example illustrates S221 to S227:

[0091] The video frame to be associated is the kth original video frame, and the set of relevant location information of the N trucks to be associated in it is B. K ={b1 K ,···,b N K}, b n K ={b 11 K b 12 K}(n∈[1,N]), b 11 K This represents the first location information (location box) of the nth truck to be associated, b 12 K This represents the second location information of the nth truck to be associated. The set of trajectory information of the M existing trucks is T = {T1, ..., T2}. M}

[0092] 1) Calculate b n1 K The intersection-over-union ratio between the location bounding boxes of the existing truck trajectory information in T and T.

[0093] 2) If there exists a relationship with b n1 K Matching (greater than the intersection-union ratio threshold Ω) V If the existing truck trajectory information is available, proceed to step 3); otherwise, proceed to step 7).

[0094] 3) Determine whether there is one or multiple existing truck trajectory information for matching. If there is only one, proceed to 4); otherwise, proceed to 5) to 6).

[0095] 4) Place b n KThis is the latest relevant location information among the existing truck trajectory information for matching.

[0096] 5) Calculate b n2 K The second distance between the trajectory information of each existing truck matched in T and the second location information.

[0097] 6) Take the trajectory information of the existing truck with the smallest second distance that matches the existing truck trajectory information that meets the distance requirement, and set b n K This is the latest relevant location information among the existing truck trajectory information that meets the distance requirements.

[0098] 7) Establish a new vehicle trajectory information T M+1 , will b n K As T M+1 The latest relevant location information in the middle.

[0099] S23: Select at least one vehicle as the target vehicle, and use the trajectory information of the target vehicle to select several original video frames as target video frames to obtain the first video frame sequence.

[0100] The trajectory information of the target vehicle can be used to find the relevant position information of the target vehicle in each of the target video frames, and the relevant position information of the target vehicle can be used to determine the position information and / or status reference information of the target vehicle.

[0101] Furthermore, the attitude information of the target vehicle in S12 above can be obtained through at least the following two methods:

[0102] Method 1: The position of the target vehicle in the first target video frame of the first video sequence can be used as the standard position. The pose information of the target vehicle in each non-first target video frame relative to the standard position can be calculated as the pose information of the target vehicle in each non-first target video frame.

[0103] Method 2: Based on the two-dimensional and three-dimensional position information corresponding to the preset position of the target vehicle in the first video frame sequence, the transformation relationship parameter between the two-dimensional and three-dimensional positions can be obtained; the attitude information can then be obtained using the transformation relationship parameter. The second position information corresponding to the preset position can be obtained from the trajectory information of the target vehicle.

[0104] The preset position can be any position of the target vehicle, such as the center of the target vehicle, key points of the target vehicle, etc. The two-dimensional position information corresponding to the preset position can be two-dimensional coordinates in the image coordinate system. The three-dimensional position information corresponding to the preset position can be regarded as the standard three-dimensional position information corresponding to the preset position when the target vehicle is in a straight-line orientation. It can be based on the true scale of the vehicle to calibrate the three-dimensional position information corresponding to the preset position. The three-dimensional position information can be three-dimensional coordinates in the world coordinate system, or it can be three-dimensional coordinates in the image coordinate system. The transformation relationship parameter can be at least one of rotation matrix and translation matrix. For each target video frame included in the first video frame sequence, the transformation relationship parameter between the two-dimensional position information and the three-dimensional position information corresponding to the preset position of the target vehicle can be calculated; based on the transformation relationship parameter, the attitude information of the target vehicle can be obtained.

[0105] See also Figure 7 Let's illustrate method two with an example. Figure 7 This diagram illustrates the correspondence between the world coordinate system, camera coordinate system, and image coordinate system. Here, c represents the optical center, O represents the origin of the camera coordinate system, OC represents the camera's focal length, P represents the point corresponding to O in the image coordinate system, and P' represents the point corresponding to O in the world coordinate system.

[0106] The attitude information is the yaw angle, the preset position is the key point of the target vehicle, the three-dimensional position information corresponding to the preset position is the three-dimensional coordinates (U, W, V) in the world coordinate system, and the two-dimensional position information corresponding to the preset position is the two-dimensional coordinates (x, y) in the image coordinate system.

[0107] The following transformation relationship exists between (U, W, V) and the three-dimensional coordinates (X, Y, Z) of the preset position in the camera coordinate system:

[0108]

[0109] Where R represents the rotation matrix of the world coordinate system relative to the camera coordinate system, and t represents the translation parameter of the world coordinate system relative to the camera coordinate system.

[0110] Furthermore, according to the camera imaging principle, (X, Y, Z) and (x, y) have the following transformation relationship:

[0111]

[0112] Where s represents the scaling factor, (f x f y () represents the focal length of the camera, and (Cx,Cy) represents the optical center of the camera.

[0113] Furthermore, (U, W, V) and (x, y) have the following transformation relationship:

[0114]

[0115] Therefore, the rotation matrix R can be obtained by solving the above equation (for example, by using the DLT direct linear transformation method), and then the yaw angle θ can be determined based on R.

[0116] Understandably, compared to method one, obtaining the target vehicle's attitude information through method two takes into account the three-dimensional position information corresponding to the target vehicle's preset position, thus avoiding errors introduced by treating the vehicle as a point mass.

[0117] Furthermore, the above-mentioned S14 can be implemented in at least the following two ways:

[0118] Method 3: Logical judgment rules can be set to determine whether the state reference information in the first target state conforms to the rules. The state reference information of the target vehicle in the second video frame sequence is then analyzed to determine if it conforms to the logical judgment rules. If it does, the target vehicle is determined to be in the first target state. For example, if the first target state is an acceleration state, the state reference information includes vehicle speed, and the logical judgment rules include the vehicle speed increasing sequentially in the second video frame sequence, etc.

[0119] Method 4: A classification model can be used to classify and determine the state reference information of the target vehicle in the second video frame sequence, obtaining a first state determination result representing whether the target vehicle is in the first target state. The state reference information may include at least one sub-state reference information, and the classification model can be any well-trained neural network model with classification capabilities, such as Long Short-Term Memory (LSTM) network or Transformer model.

[0120] See also Figure 8 Taking an LSTM classification model as an example, S14 can include the following sub-steps:

[0121] S31: Obtain the encoding features of each sub-state reference information corresponding to the current target video frame in the second video frame sequence.

[0122] S32: Fuse the encoded features of the state information corresponding to the current target video frame to obtain the fused features of the current target video frame.

[0123] S33: Based on the first state determination result of the previous target video frame and the fusion features of the current target video frame, determine the first state determination result of the current target video frame.

[0124] Repeat the above steps until each target video frame in the second video frame sequence has been traversed, and take the first state determination result of the last target video frame in the second video sequence as the first state determination result of the target vehicle.

[0125] The following example will be used to explain S31 to S33 in detail:

[0126] The state reference information includes yaw angle, vehicle speed, pitch angle, and roll angle. The yaw angle, vehicle speed, pitch angle, and roll angle of the target vehicle in the current target video frame are mapped into fixed-length encoded features. The mapping results of yaw angle, vehicle speed, pitch angle, and roll angle are weighted and fused to obtain the fused features of the current target video frame. Based on the first state determination result of the previous target video frame and the fused features of the current target video frame, the first state determination result of the target vehicle is obtained.

[0127] Understandably, compared to logical judgment rules, classification models offer several advantages. First, they provide a more flexible approach to determining state status, allowing for the expansion of sub-state reference information within the state reference information during the classification process, based on factors such as the type of the first target state and the required classification accuracy. Second, classification models are trained using batches of training samples, which can cover various scenarios where vehicles are in or not in the first target state. This results in high robustness, adapting to different vehicle states and effectively modeling the state reference information of the target vehicle. Consequently, the accuracy of state classification results is improved.

[0128] Figure 9 This is a flowchart illustrating another embodiment of the vehicle state recognition method of this application. It should be noted that if substantially the same result is achieved, this embodiment is not necessarily identical. Figure 9 The illustrated process sequence is limited. This embodiment is a further extension of the above embodiment to determine whether the target vehicle is in a second target state based on a first video frame sequence. Figure 9 As shown, this embodiment may include:

[0129] S41: Obtain the displacement of adjacent target video frames in the first video frame sequence with respect to the target vehicle.

[0130] In some embodiments, any location point of the target vehicle can be selected, and the distance between adjacent target video frames with respect to the selected location point can be calculated as the displacement of the target vehicle.

[0131] In some embodiments, a first distance may be obtained between at least one location point in adjacent target video frames; the displacement of the target vehicle is obtained by weighting the first distance between the at least one location point. The at least one location point includes at least one key point of the target vehicle and / or the center of the target vehicle.

[0132] Understandably, compared to calculating the displacement of a target vehicle based solely on its center, combining the target vehicle's center with key points makes the displacement of the target vehicle more accurate.

[0133] S42: Based on the displacement of the target vehicle, determine whether the target vehicle is in the second target state.

[0134] The second target state can be a state related to velocity information. For example, a stationary state (velocity is 0, acceleration is 0), an accelerating state (velocity gradually increases, acceleration is greater than 0), a decelerating state (velocity gradually decreases, acceleration is less than 0), and so on.

[0135] Furthermore, based on any of the above embodiments, the second target state and the first target state can be different states associated with vehicle behavior, and under the normal behavior of the target vehicle, the time the target vehicle is in the first target state and the time it is in the second target state have a sequence and a time difference. For example, in the application scenario of "stopping before turning", the normal behavior of the target vehicle is "stopping before turning", the second target state is a stationary state, the first target state is a turning state, and the time the target vehicle is in the second target state precedes the time it is in the first target state and the time difference between the time the target vehicle is in the second target state and the time it is in the second target state is less than a time difference threshold.

[0136] Based on this, it can be determined whether the target vehicle exhibits abnormal behavior based on the first state determination result, which characterizes whether the target vehicle is in a first target state, and the second state determination result, which characterizes whether the target vehicle is in a second target state. In some embodiments, this can be achieved in the following manner:

[0137] Figure 10 This is a flowchart illustrating another embodiment of the vehicle state recognition method of this application. It should be noted that if substantially the same result is achieved, this embodiment is not necessarily identical. Figure 10 The illustrated process sequence is limited. This embodiment is a further extension of the above embodiment to determine whether the target vehicle exhibits abnormal behavior. Figure 10 As shown, this embodiment may include:

[0138] S51: Determine whether the result of the first state determination indicates that the target vehicle is in the first target state, and whether the result of the second state determination indicates whether the target vehicle is in the second target state.

[0139] If the first state determination result indicates that the target vehicle is not in the first target state, and the second state determination result indicates that the target vehicle is not in the second target state, execute S52; if the first state determination result indicates that the target vehicle is in the first target state, and the second state determination result indicates that the target vehicle is not in the second target state, execute S53; if the first state determination result indicates that the target vehicle is in the first target state, and the second state determination result indicates that the target vehicle is in the second target state, execute S54.

[0140] S52: Determine that the target vehicle does not exhibit any abnormal behavior.

[0141] S53: The target vehicle has been found to be exhibiting abnormal behavior.

[0142] S54: Based on the sequence and time difference between the first and second state times of the target vehicle, determine whether the target vehicle exhibits abnormal behavior.

[0143] The first state time is the time the target vehicle is in the first target state, and the second state time is the time the target vehicle is in the second target state.

[0144] The following combination Figure 11 This paper provides a detailed explanation of the vehicle state recognition method provided in this application, taking into account the application scenario of determining the pre-turning stopping behavior of large trucks:

[0145] 1) Perform vehicle tracking on the original video frame sequence to obtain the trajectory information of the truck and the first video frame sequence of the truck. The first video frame sequence includes several target video frames containing the truck. The trajectory information of the truck includes the first position information of the truck in the vehicle area of ​​each target video frame and the second position information (two-dimensional position information) of key points.

[0146] 2) Obtain the yaw angle of the truck in each target video frame based on the two-dimensional position information and the three-dimensional position information corresponding to the key points.

[0147] 3) Determine whether the difference in yaw angle between the truck and the target video frame is greater than the yaw angle threshold; if it is greater, determine that there is reference time information indicating that the truck is suspected of being in a turning state; if it is not greater, determine that the truck is not in a turning state.

[0148] 4) Find the second video frame sequence that is related to the reference time information from the first video frame sequence.

[0149] 5) Using the classification model LSTM, the steering state is classified based on the yaw angle, vehicle speed, pitch angle and roll angle of the target vehicle in the second video frame sequence, and the first state determination result representing that the truck is in a steering state is obtained.

[0150] 6) Based on the first and second position information, obtain the displacement of the truck in the preceding and following video frames.

[0151] 7) Determine if the displacement of the truck is less than the displacement threshold; if it is less, determine that the truck is stationary; if it is not less, determine that the truck is not stationary.

[0152] 8) Determine whether the second state time of the truck when it is stationary is earlier than the first state time when it is turning, and whether the time difference between the second state time and the first state time is less than the time difference threshold; if not, determine that the target vehicle did not stop before turning and there is abnormal behavior; if so, determine that the target vehicle stopped before turning and there is no abnormal behavior.

[0153] Figure 12 This is a schematic diagram of the structure of an embodiment of the electronic device of this application. Figure 12 As shown, the electronic device includes a processor 21 and a memory 22 coupled to the processor 21.

[0154] The memory 22 stores program instructions for implementing the methods of any of the above embodiments; the processor 21 executes the program instructions stored in the memory 22 to implement the steps of the above method embodiments. The processor 21 may also be referred to as a CPU (Central Processing Unit). The processor 21 may be an integrated circuit chip with signal processing capabilities. The processor 21 may also be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. The general-purpose processor may be a microprocessor or any conventional processor.

[0155] Figure 13Figure 13 is a schematic diagram of the structure of an embodiment of the computer-readable storage medium of this application. As shown in Figure 13, the computer-readable storage medium 30 of this application embodiment stores program instructions 31, which, when executed, implement the methods provided in the above embodiments of this application. The program instructions 31 can form a program file and be stored in the computer-readable storage medium 30 in the form of a software product, so that a computer device (which may be a personal computer, server, or network device, etc.) or processor can execute all or part of the steps of the methods of various embodiments of this application. The aforementioned computer-readable storage medium 30 includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks, or terminal devices such as computers, servers, mobile phones, and tablets.

[0156] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces, or indirect coupling or communication connection between apparatuses or units, and may be electrical, mechanical, or other forms.

[0157] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated units described above can be implemented in hardware or as software functional units. The above are merely embodiments of this application and do not limit the patent scope of this application. Any equivalent structural or procedural transformations made based on the description and drawings of this application, or direct or indirect applications in other related technical fields, are similarly included within the patent protection scope of this application.

Claims

1. A vehicle state recognition method, characterized in that, include: Obtain a first video frame sequence, wherein the first video frame sequence includes a plurality of target video frames containing the target vehicle; Based on the attitude information of the target vehicle in the first video frame sequence, it is determined whether there is reference time information, which is the time information when the target vehicle is suspected to be in the first target state; In response to the existence of the reference time information, a second video frame sequence associated with the reference time information is retrieved from the first video frame sequence; Based on the state reference information of the target vehicle in the second video frame sequence, determine whether the target vehicle is in the first target state; The step of determining whether reference time information exists based on the attitude information of the target vehicle in the first video frame sequence includes: Based on whether the change in the attitude information of the target vehicle in the first video frame sequence is greater than the attitude threshold, it is determined whether a third video frame sequence exists. In response to the change in the attitude information of the target vehicle in the first video frame sequence being greater than the attitude threshold, it is determined that the third video frame sequence exists, wherein the third video frame sequence corresponds to the change in the attitude information of the target vehicle being greater than the attitude threshold. The reference time information is determined based on the capture time of at least one target video frame in the third video frame sequence.

2. The method according to claim 1, characterized in that, The step of determining whether a third video frame sequence exists based on whether the change in the target vehicle's attitude information in the first video frame sequence is greater than an attitude threshold includes: The attitude information of the target vehicle in each target video frame of the first video frame sequence is compared sequentially. In response to the fact that the attitude information of the target vehicle in a series of predetermined number of target video frames increases sequentially and is greater than the attitude threshold, it is determined that the third video frame sequence exists, and the predetermined number of target video frames constitute the third video frame sequence.

3. The method according to claim 1, characterized in that, Before determining whether reference time information exists based on the attitude information of the target vehicle in the first video frame sequence, the method further includes: Based on the two-dimensional position information and the three-dimensional position information corresponding to the preset position of the target vehicle in the first video frame sequence, the conversion relationship parameter between the preset position and the three-dimensional position is obtained. The attitude information is obtained using the transformation relationship parameters.

4. The method according to claim 3, characterized in that, The preset location is a key point of the target vehicle; And / or, the transformation relationship parameter is a rotation matrix, and the attitude information is the yaw angle of the target vehicle.

5. The method according to claim 1, characterized in that, The second video frame sequence consists of the target video frames in the first video frame sequence whose time difference between the shooting time and the reference time information is less than a time difference threshold; And / or, the state reference information includes at least one sub-state reference information among the attitude information and speed information of the target vehicle.

6. The method according to claim 1, characterized in that, Determining whether the target vehicle is in the first target state based on the state reference information of the target vehicle in the second video frame sequence includes: The state reference information of the target vehicle in the second video frame sequence is classified and judged using a classification model to obtain a first state determination result characterizing whether the target vehicle is in the first target state.

7. The method according to claim 6, characterized in that, The state reference information includes at least one sub-state reference information. The step of classifying and determining the state reference information of the target vehicle in the second video frame sequence using a classification model to obtain a first state determination result characterizing whether the target vehicle is in the first target state includes: Obtain the encoding features of each state information corresponding to the current target video frame in the second video frame sequence; The fused features of the current target video frame are obtained by fusing the encoding features of each of the sub-state reference information corresponding to the current target video frame; Based on the first state determination result of the previous target video frame and the fusion features of the current target video frame, the first state determination result of the current target video frame is determined; Repeat the above steps until each target video frame in the second video frame sequence has been traversed, and take the first state determination result of the last target video frame in the second video frame sequence as the first state determination result of the target vehicle.

8. The method according to claim 1, characterized in that, The method further includes: Obtain the displacement of adjacent target video frames in the first video frame sequence with respect to the target vehicle; Based on the displacement of the target vehicle, it is determined whether the target vehicle is in a second target state, which is a state related to speed information.

9. The method according to claim 8, characterized in that, The step of obtaining the displacement of adjacent target video frames in the first video frame sequence with respect to the target vehicle includes: Obtain a first distance between at least one location point of the adjacent target video frames; wherein the at least one location point includes at least one key point of the target vehicle and / or the center of the target vehicle; The displacement of the target vehicle is obtained by weighting the first distance between the at least one location point.

10. The method according to claim 8, characterized in that, After determining whether the target vehicle is in the first target state based on the state reference information of the target vehicle in the second video frame sequence, and after determining whether the target vehicle is in the second target state based on the displacement of the target vehicle, the method further includes: Based on the first state determination result characterizing whether the target vehicle is in the first target state and the second state determination result characterizing whether the target vehicle is in the second target state, it is determined whether the target vehicle has abnormal behavior.

11. The method according to claim 10, characterized in that, The determination of whether the target vehicle exhibits abnormal behavior based on a first state determination result characterizing whether the target vehicle is in the first target state and a second state determination result characterizing whether the target vehicle is in the second target state includes: If the first state determination result indicates that the target vehicle is in the first target state, and the second state determination result indicates that the target vehicle is not in the second target state, then it is determined that the target vehicle has abnormal behavior. If the first state determination result indicates that the target vehicle is in the first target state, and the second state determination result indicates that the target vehicle is in the second target state, then based on the order and time difference between the first state time and the second state time of the target vehicle, it is determined whether the target vehicle has abnormal behavior, wherein the first state time is the time when the target vehicle is in the first target state, and the second state time is the time when the target vehicle is in the second target state.

12. The method according to claim 1, characterized in that, The step of obtaining the first video frame sequence includes: Vehicle detection is performed on each original video frame in the original video frame sequence to obtain the relevant location information of each vehicle in each original video frame. The relevant location information of the vehicle includes the first location information of the vehicle area corresponding to the vehicle in the original video frame and the second location information of the key point. The relevant location information of each vehicle is correlated to obtain the trajectory information of each vehicle; At least one of the vehicles is designated as the target vehicle, and several original video frames are selected as the target video frames using the trajectory information of the target vehicle to obtain the first video frame sequence. The trajectory information of the target vehicle is used to find the relevant position information of the target vehicle in each of the target video frames, and the relevant position information of the target vehicle is used to determine the position information and / or status reference information of the target vehicle.

13. The method according to claim 12, characterized in that, The step of associating the relevant location information of each vehicle to obtain the trajectory information of each vehicle includes: Each of the original video frames is sequentially used as a video frame to be associated, and the vehicle in the video frame to be associated is used as the vehicle to be associated. The first location information of the vehicle to be associated is matched with the trajectory information of each existing vehicle. In response to the existence of trajectory information of an existing vehicle that matches the first location information of the vehicle to be associated, determine whether the number of matching trajectory information of the existing vehicle is one or more. In response to the fact that the number of existing vehicle trajectory information that is matched is one, the trajectory information of the existing vehicle is updated using the relevant location information of the vehicle to be associated. In response to the fact that there are multiple trajectory information of the existing vehicles initially matched, a second distance is obtained between the second location information of each matched existing vehicle trajectory information and the second location information of the vehicle to be associated; using the relevant location information of the vehicle to be associated, the trajectory information of the existing vehicles whose second distance meets the distance requirement is updated.

14. An electronic device, characterized in that, Includes a processor and a memory connected to the processor, wherein, The memory stores program instructions; The processor is used to execute the program instructions stored in the memory to implement the method of any one of claims 1-13.

15. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores program instructions that can be executed by a processor to implement the method as described in any one of claims 1-13.

Images