Scene recognition method and device, equipment, storage medium and product

Abstract

The invention discloses a scene recognition method and device, equipment, a storage medium and a product, and relates to the technical field of computers, and the scene recognition method comprises the steps: obtaining scene data; traversing all frames in the scene data, and finding out a VRU object in a preset traversing monitoring range from the current frame until traversing is completed; and if it is continuously detected that the VRU object has a traversing behavior in multiple frames of the monitoring time interval with the current frame as the center, identifying the current scene data as that the VRU object traverses the scene. According to the method and the device, the VRU with the traversing behavior is identified through the historical moment data frame and the future moment data frame, so that the VRU traversing scene in the data fragment is identified under the condition of no high-precision map information; the VRU scene meeting the effective frame number range is identified and recorded, so that the accurate identification of the VRU crossing scene under the condition of no high-precision map information can be realized.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to scene recognition methods, devices, equipment, storage media, and products. Background Technology

[0002] Testing vehicles on public roads carries high costs and significant accident risks. Therefore, simulation testing, as a safe, efficient, and cost-effective verification method, has attracted considerable attention from researchers. Among the key aspects of simulation testing, identifying the testing scenario is crucial for data acquisition.

[0003] Current methods for identifying VRU (Vulnerable Road Users) crossing scenarios mainly rely on combining high-precision map information and assisted positioning technology. By identifying the intersection of the VRU's trajectory and the vehicle's lane trajectory in the high-precision map information, the VRU crossing scenario can be identified. However, the accuracy of identifying VRU crossing scenarios is low in the absence of high-precision maps. Summary of the Invention

[0004] The main purpose of this application is to provide a scene recognition method, apparatus, device, storage medium and product, which aims to solve the technical problem of low recognition accuracy of VRU traversing scenes when high-precision map information is lacking.

[0005] To achieve the above objectives, this application proposes a scene recognition method, which includes:

[0006] Acquire scene data;

[0007] Traverse all frames in the scene data, find the VRU objects within the preset cross-monitoring range in the current frame, and continue traversing until the traversal is complete;

[0008] If the VRU object is continuously detected to be traversing across multiple frames within the monitoring time interval centered on the current frame, then the current scene data is identified as the VRU object traversing the scene.

[0009] In one embodiment, the step of traversing all frames in the scene data and finding VRU objects within a preset cross-monitoring range in the current frame until the traversal is complete includes:

[0010] Iterate through the frames in the scene data and take the traversed frame as the current frame;

[0011] If the monitoring time interval centered on the current frame does not exceed the scene data, extract the location information of all VRU objects from the current frame;

[0012] Based on the location information, the relative position of each VRU object with respect to the vehicle coordinates is calculated, and VRU objects within the preset cross-traverse monitoring range are identified.

[0013] In one embodiment, the step of calculating the relative position of each VRU object with respect to the vehicle coordinates based on the location information, and finding the VRU objects within a preset cross-traverse monitoring range, includes:

[0014] Obtain the vehicle's coordinates and heading angle;

[0015] Based on the location information of all VRU objects, the vehicle coordinates, and the vehicle orientation angle, calculate the relative position of each VRU object relative to the vehicle.

[0016] Based on the relative position, locate the VRU object that is within the preset cross-monitoring range.

[0017] In one embodiment, the step of identifying the current scene data as a VRU object traversing the scene if the VRU object is continuously detected to be traversing the scene across multiple frames within the monitoring time interval centered on the current frame includes:

[0018] Based on the current frame and the monitoring time interval, determine the historical frame from which the traversal begins and the future frame from which the traversal ends;

[0019] Traverse the historical frames preceding the current frame, calculate the relative position change of the VRU object, and determine the number of valid historical frames traversed by the VRU object.

[0020] Traverse the future frames following the current frame, calculate the relative position change of the VRU object, and determine the number of effective frames the VRU object will traverse in the future.

[0021] Based on the historical number of valid cross-traversal frames and the future number of valid cross-traversal frames, the scene data in which the VRU object exhibits valid cross-traversal behavior within the monitoring time interval is identified as a VRU object cross-traversal scene.

[0022] In one embodiment, the step of traversing historical frames prior to the current frame, calculating the relative position change of the VRU object, and determining the number of valid historical frames traversed by the VRU object includes:

[0023] Traverse the historical frames to the current frame and calculate the relative position of the VRU object and the vehicle.

[0024] Based on the changes in the relative position of the VRU object and the vehicle in each frame, update the historical valid movement trajectory of the VRU object during its traversal movement within the traversal monitoring range;

[0025] Based on the historical valid action trajectory, determine the number of historical valid frames traversed by the VRU object.

[0026] In one embodiment, the step of traversing future frames after the current frame, calculating the relative position change of the VRU object, and determining the number of effective frames the VRU object will traverse in the future includes:

[0027] Traverse from the current frame to the future frame to calculate the relative position of the VRU object and the vehicle.

[0028] Based on the changes in the relative position of the VRU object and the vehicle in each frame, update the future effective movement trajectory of the VRU object during its traversal movement within the traversal monitoring range;

[0029] Based on the future valid action trajectory, determine the number of future valid frames traversed by the VRU object.

[0030] In one embodiment, the step of identifying the current scene data as a VRU object traversing the scene if the VRU object is continuously detected to be traversing the scene across multiple frames within the monitoring time interval centered on the current frame includes:

[0031] Query the frame information list corresponding to the VRU object, read the frame index of the historical traversal event record of the VRU object, and determine the time interval with the frame index of the current frame;

[0032] If the time interval of the frame index is less than the preset repetition determination threshold, then the traversal event of the VRU object of the current frame is determined to be a repetition determination event.

[0033] If the time interval of the frame index is not less than the preset repetition determination threshold, then the traversal event of the VRU object of the current frame is determined as a new event, and the frame information list of the current frame to the VRU object is updated.

[0034] Furthermore, to achieve the above objectives, this application also proposes a scene recognition device, which includes:

[0035] The acquisition module is used to acquire scene data;

[0036] The search module is used to traverse all frames in the scene data, find the VRU objects that are within the preset cross-monitoring range in the current frame, until the traversal is complete;

[0037] The identification module is used to identify the current scene data as a VRU object crossing the scene if the VRU object is continuously detected to be crossing the scene in multiple frames within the monitoring time interval centered on the current frame.

[0038] In addition, to achieve the above objectives, this application also proposes a scene recognition device, the device comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, the computer program being configured to implement the steps of the scene recognition method as described above.

[0039] In addition, to achieve the above objectives, this application also proposes a storage medium, which is a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, it implements the steps of the scene recognition method described above.

[0040] In addition, to achieve the above objectives, this application also provides a computer program product, which includes a computer program that, when executed by a processor, implements the steps of the scene recognition method described above.

[0041] One or more technical solutions proposed in this application have at least the following technical effects:

[0042] In related technologies, the intersection of VRU trajectories and vehicle trajectories is determined from data segments by combining high-precision map information and other auxiliary positioning techniques to identify VRU crossing scenarios in the vicinity. However, the accuracy of VRU crossing scenario identification is low when high-precision map information is lacking. Compared with this approach, this application identifies VRUs exhibiting crossing behavior by using historical and future time data frames. This enables the identification of VRU crossing scenarios in data segments without high-precision map information. By identifying and recording VRU scenarios that meet the effective frame count range, accurate identification of VRU crossing scenarios can be achieved even without high-precision map information. Attached Figure Description

[0043] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0044] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0045] Figure 1 This is a flowchart illustrating an embodiment of the scene recognition method of this application.

[0046] Figure 2 This is a flowchart illustrating Embodiment 2 of the scene recognition method of this application;

[0047] Figure 3This is a flowchart illustrating Embodiment 3 of the scene recognition method of this application;

[0048] Figure 4 This is a schematic diagram of the module structure of the scene recognition device according to an embodiment of this application;

[0049] Figure 5 This is a schematic diagram of the device structure of the hardware operating environment involved in the scene recognition method in the embodiments of this application.

[0050] The purpose, features, and advantages of this application will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0051] It should be understood that the specific embodiments described herein are merely illustrative of the technical solutions of this application and are not intended to limit this application.

[0052] To better understand the technical solution of this application, a detailed description will be provided below in conjunction with the accompanying drawings and specific implementation methods.

[0053] The main solution in this application's embodiments is:

[0054] Acquire scene data;

[0055] Traverse all frames in the scene data, find the VRU objects within the preset cross-monitoring range in the current frame, and continue traversing until the traversal is complete;

[0056] If the VRU object is continuously detected to be traversing across multiple frames within the monitoring time interval centered on the current frame, then the current scene data is identified as the VRU object traversing the scene.

[0057] In this embodiment, the application uses a scene recognition device as the execution subject. For ease of description, it will be referred to as "device" in the following detailed description.

[0058] Existing technologies mainly rely on combining high-precision map information and assisted positioning technology to identify VRU crossing scenarios by recognizing the intersection of the VRU's trajectory and the vehicle lane trajectory in the high-precision map information. However, the accuracy of recognizing VRU crossing scenarios is low in the absence of high-precision maps.

[0059] This application provides a solution that identifies VRUs exhibiting cross-traversal behavior by using historical and future data frames. This enables the identification of VRU cross-traversal scenarios in data segments without high-precision map information. By identifying and recording VRU scenarios that meet the effective frame count range, accurate identification of VRU cross-traversal scenarios can be achieved even without high-precision map information.

[0060] Based on this, embodiments of this application provide a scene recognition method, referring to... Figure 1 , Figure 1 This is a flowchart illustrating the first embodiment of the scene recognition method of this application.

[0061] In this embodiment, the scene recognition method includes steps S10 to S30:

[0062] Step S10: Obtain scene data;

[0063] It should be noted that the scene data refers to a series of continuous environmental data collected from the vehicle's sensors (such as cameras, radar, lidar, etc.). These data fragments contain real-time information about the vehicle's surrounding environment, which is used to support subsequent scene recognition and decision-making.

[0064] Step S20: Traverse all frames in the scene data, find the VRU objects within the preset cross-monitoring range in the current frame, until the traversal is complete;

[0065] It should be noted that the frame refers to a video or image sequence, with each frame representing a single instant. The current frame is the frame currently being processed or analyzed during the traversal process; it represents the instant the traversal reaches, used for information extraction and further calculations. VRU stands for "Vulnerable Road User," referring to vulnerable users in road traffic, including pedestrians, cyclists, motorcyclists, and other non-motorized traffic participants. The preset traversal monitoring range is a pre-defined area used to monitor whether VRU objects cross the road in or near the vehicle's travel path.

[0066] For example, the vehicle's sensors begin to operate, collecting data about the surrounding environment to form scene data. The device begins to examine these data segments frame by frame. For each frame, the device marks it as the "current frame." In each frame, the device uses a preset algorithm to identify and locate VRU objects (such as pedestrians, cyclists, etc.). The device checks whether these VRU objects are within a preset cross-traffic detection range. This range is preset by the device based on safety standards and vehicle driving characteristics. The device continues this process until all frames have been traversed, ensuring that no possible cross-traffic behavior is missed. Once the device detects a VRU object in the current frame and confirms that it is within the cross-traffic detection range, the device marks this frame and takes appropriate safety measures, such as slowing down, stopping, or issuing a warning.

[0067] In one feasible implementation, refer to Figure 2 Step S20 may include steps A1 to A3:

[0068] Step A1: Traverse the frames in the scene data and take the traversed frame as the current frame.

[0069] It should be noted that the current frame refers to the frame being processed or analyzed during the traversal process; it is the image at the moment it is traversed, used to extract information and perform further calculations.

[0070] Step A2: If the monitoring time interval centered on the current frame does not exceed the scene data, extract the location information of all VRU objects from the current frame;

[0071] It should be noted that the monitoring time interval is a time-defined interval within which the device monitors the behavior of the VRU object. This interval extends forward and backward from the current frame to capture the dynamic behavior of the VRU object. VRU is an abbreviation for "Vulnerable Road User," referring to vulnerable users in road traffic, including pedestrians, cyclists, motorcyclists, and other non-motorized traffic participants. The location information is the specific location data of the VRU object in the scene, which can be extracted from the current frame for subsequent analysis and calculation.

[0072] Step A3: Based on the location information, calculate the relative position of each VRU object with respect to the vehicle coordinates, and find the VRU objects within the preset cross-traverse monitoring range.

[0073] It should be noted that the vehicle coordinates refer to the autonomous vehicle's own coordinate system, used to determine the vehicle's position in space. The position information of all other objects is determined relative to this coordinate system. The relative position refers to the position of each VRU object relative to the autonomous vehicle. By calculating the relative position of the VRU object's position information with the vehicle coordinates, the autonomous driving system can determine whether the VRU object is on or near the vehicle's driving path. The preset traverse monitoring range is an area pre-defined by the autonomous driving system to monitor whether a VRU object crosses the road on or near the vehicle's driving path. This range can be adjusted according to factors such as vehicle speed and road conditions.

[0074] In one feasible approach, step A3 includes:

[0075] Obtain the vehicle's coordinates and heading angle;

[0076] Based on the location information of all VRU objects, the vehicle coordinates, and the vehicle orientation angle, calculate the relative position of each VRU object relative to the vehicle.

[0077] Based on the relative position, locate the VRU object that is within the preset cross-monitoring range.

[0078] It should be noted that the vehicle coordinates refer to the vehicle's own coordinates, used to determine the vehicle's position in space. The position information is the specific position data of the VRU object in the scene, which can be extracted from the current frame for subsequent analysis and calculation. The relative position is the position of each VRU object relative to the vehicle.

[0079] For example, the position information of the vehicle in this frame is obtained, including the X coordinate (ego_x), Y coordinate (ego_y), and orientation angle (ego_theta);

[0080] If the coordinates of the VRU object are P(global_x, global_y), then the formulas for calculating local_x and local_y are as follows:

[0081] x_diff = global_x - ego_x

[0082] y_diff = global_y - ego_y

[0083] angle_cos=math.cos(ego_theta)

[0084] angle_sin=math.sin(ego_theta)

[0085] local_x=x_diff*angle_cos-y_diff*angle_sin

[0086] local_y=x_diff*angle_sin+y_diff*angle_cos

[0087] Where x_diff is the translation of point P in the x-direction relative to the vehicle, specifically calculated as the difference between the global x-coordinate of point P and the global x-coordinate of the vehicle; y_diff is the translation of point P in the y-direction relative to the vehicle, specifically calculated as the difference between the global y-coordinate of point P and the global y-coordinate of the vehicle; angle_cos refers to the cosine of the vehicle's orientation angle; angle_sin refers to the sine of the vehicle's orientation angle; local_x refers to the x-coordinate of point P in the local coordinate system, specifically calculated by multiplying the x-direction translation by the cosine and the y-direction translation by the sine, and then subtracting them; local_y refers to the y-coordinate of point P in the local coordinate system, specifically calculated by multiplying the x-direction translation by the sine and the y-direction translation by the cosine, and then adding them together.

[0088] Based on the relative position, it is determined that the vertical coordinate local_x>0 and the horizontal coordinate local_y<1 must be satisfied before it can be used as a candidate VRU traversal target.

[0089] Step S30: If the VRU object is continuously detected to be traversing across multiple frames in the monitoring time interval centered on the current frame, then the current scene data is identified as the VRU object traversing the scene.

[0090] It should be noted that the monitoring time interval is a time-defined interval within which the device monitors the behavior of the VRU object. This interval extends forward and backward from the current frame to capture the dynamic behavior of the VRU object.

[0091] In one feasible implementation, refer to Figure 3 Step S30 may include steps B1 to B4:

[0092] Step B1: Based on the current frame and the monitoring time interval, determine the historical frame from which the traversal begins and the future frame from which the traversal ends.

[0093] It should be noted that the historical frame from which the traversal begins is the earliest historical frame traced back from the current frame within the monitoring time interval, serving as the starting point for traversing historical frames. The future frame from which the traversal ends is the latest future frame extending forward from the current frame within the monitoring time interval, serving as the ending point for traversing future frames.

[0094] Step B2: Traverse through the historical frames preceding the current frame, calculate the relative position change of the VRU object, and determine the number of valid historical frames traversed by the VRU object.

[0095] It should be noted that the relative position change refers to the change in the position of the VRU object between consecutive frames. By calculating these changes, the device can determine the direction and speed of movement of the VRU object.

[0096] In one feasible approach to the facts, step B2 includes:

[0097] Traverse the historical frames to the current frame and calculate the relative position of the VRU object and the vehicle.

[0098] Based on the changes in the relative position of the VRU object and the vehicle in each frame, update the historical valid movement trajectory of the VRU object during its traversal movement within the traversal monitoring range;

[0099] Based on the historical valid action trajectory, determine the number of historical valid frames traversed by the VRU object.

[0100] For example, count_front represents the number of frames within a specified frame range that satisfy VRU traversal behavior in the past.

[0101] Calculate the starting frame index p of the historical frames considered, where p = frame_idx - duration, where frame_idx is the frame index of the current frame and duration is the length of the time window for viewing historical moments;

[0102] Initialize the leftward movement trend counter c_left (default 0, leftward movement trend) and the rightward movement trend counter c_right (default 0, rightward movement trend), and initialize the coordinate position of the object VRU relative to the host vehicle (in the host vehicle coordinate system) diff_x_temp (default 0) and diff_y_temp (default 0).

[0103] Traverse the frame data within the specified range (from the starting frame index p of the historical frames to the current frame frame_idx), and for the VRU object in each frame:

[0104] For each frame, the device calculates the speed ego_v of the host vehicle and uses ego_v to determine a longitudinal threshold threshold_x, which is used to determine whether the VRU object is considered to be crossing a lane. Among them, threshold_x = max(4 * ego_v, 30)

[0105] Calculate the position change temp of the object VRU relative to the host vehicle. temp includes the longitudinal displacement temp_x and the lateral displacement temp_y of the object from the host vehicle. If the object VRU is not in the current historical frame, then temp = False.

[0106] If temp is False, skip the current historical frame and update the longitudinal displacement temp_x and the lateral displacement temp_y of the host vehicle.

[0107] If diff_x_temp is 0, indicating that the frame considered is the starting frame index p of the historical frames, then directly update diff_x_temp and diff_y_temp, where diff_x_temp = temp_x and diff_y_temp = temp_y.

[0108] Update the c_left or c_right counter according to the value of temp. If the longitudinal position temp_x of the object VRU is less than threshold_x and the lateral position temp_y has increased compared to the previous record, that is, 0 < temp_x < threshold_x and diff_y_temp < temp_y, then c_left is incremented by 1.

[0109] If the vertical position temp_x of the object VRU is less than threshold_x and the horizontal position temp_y decreases compared to the previous record, i.e., 0 < temp_x < threshold_x and diff_y_temp > temp_y, then c_right is incremented by 1. At the same time, update diff_x_temp and diff_y_temp to the values of the current frame, i.e., diff_x_temp = temp_x, diff_y_temp = temp_y.

[0110] Compare the values of c_left and c_right, and use the larger one as the number of valid crossing frames. If c_left > c_right, consider the c_left quantity, i.e., count_front = c_left; if c_left < c_right, consider the c_right quantity, i.e., count_front = c_right; if they are equal, return None or it can be considered that there is no determined direction, i.e., count_front = None.

[0111] Step B3, traverse within the future frames after the current frame, calculate the relative position change of the VRU object, and determine the number of future valid crossing frames of the VRU object; [[ID=https: / / www.example.com]]

[0112] In a feasible factual manner, step B3 includes:

[0113] Traverse from the current frame to the future frame, and calculate the relative position of the VRU object with respect to the ego vehicle;

[0114] According to the change in the relative position of the VRU object with respect to the ego vehicle in each frame, update the future valid action trajectory of the VRU object for crossing within the crossing monitoring range;

[0115] According to the future valid action trajectory, determine the number of future valid crossing frames of the VRU object.

[0116] Exemplarily, count_back represents the number of frames that satisfy the VRU crossing behavior in the future within a specified frame range; determining the future frame range is first to determine a future time range, starting from the current frame, and this range is determined by the duration parameter and is used to calculate the end index q of the future frame. The initialized variables are that the algorithm initializes two counters c_left and c_right to record the number of times the VRU crosses to the left and right respectively. At the same time, initialize the coordinate positions diff_x_temp and diff_y_temp of the VRU with respect to the ego vehicle. The algorithm for traversing frame data starts from the current frame frame_idx and traverses until the end index q of the future frame.

[0117] The threshold calculation is performed for each frame by calculating the vehicle's speed ego_v and using it to determine a longitudinal threshold threshold_x. This threshold is used to determine whether the VRU is considered to be crossing a lane. The position change calculation involves calculating the VRU's position change temp relative to the vehicle, including longitudinal displacement temp_x and lateral displacement temp_y. If the VRU is not in the current frame, temp is set to False. The position change update is performed if temp is not False; the device updates the c_left or c_right counters based on the value of temp. If the VRU moves to the left (lateral position increases), c_left increases; if the VRU moves to the right (lateral position decreases), c_right increases. Simultaneously, diff_x_temp and diff_y_temp are updated to the values ​​of the current frame. The number of valid cross-lane frames is determined by comparing the values ​​of c_left and c_right, and taking the larger one as the number of valid cross-lane frames count_back. If c_left and c_right are equal, it is considered that there is no definite direction, and count_back is None.

[0118] If count_front (historical traversal frame count) or count_back (future traversal frame count) is None, it means that the movement direction of the VRU object is not clear, so this VRU object is skipped and no further analysis is performed.

[0119] Step B4: Based on the number of historical valid cross-traversal frames and the number of future valid cross-traversal frames, identify the scene data where the VRU object exhibits valid cross-traversal behavior within the monitoring time interval as a VRU object cross-traversal scene.

[0120] For example, if the number of valid frames in both the front and back time windows is greater than 6, i.e., count_front>6 and count_back>6, then it is considered a valid VRU traversal event and the event is recorded.

[0121] In one feasible implementation, the step of identifying the current scene data as a VRU object traversing the scene if the VRU object is continuously detected to be traversing the scene across multiple frames within the monitoring time interval centered on the current frame includes:

[0122] Query the frame information list corresponding to the VRU object, read the frame index of the historical traversal event record of the VRU object, and determine the time interval with the frame index of the current frame;

[0123] If the time interval of the frame index is less than the preset repetition determination threshold, then the traversal event of the VRU object of the current frame is confirmed as a repetition determination event.

[0124] If the time interval of the frame index is not less than the preset repetition determination threshold, then the traversal event of the VRU object of the current frame is confirmed as a new event, and the frame information list of the current frame to the VRU object is updated.

[0125] For example, if the VRU object is not in the vru_traverse field, it indicates that the VRU object is the first VRU traversal behavior in this data segment. The current frame_idx is recorded in the specific valid information dictionary vru_traverse of the VRU traversal behavior, where key is the VRU object index, valu is a list containing the current frame index, and frame_idx is a list, such as [frame_idx].

[0126] If the VRU object is already in the `vru_traverse` field, further determination is made. The specific frame information list `frame_list` for the VRU object in the valid information dictionary `vru_traverse` is retrieved. The last element `frame_end` of the `frame_list` corresponding to the VRU object is then taken. `frame_end` represents the frame index where the object was most recently determined to be a VRU traversal. If the current frame index `frame_idx - frame_end` < 20, then the object is determined to be a duplicate VRU traversal in the current frame, and this object is skipped. Otherwise, the current frame `frame_idx` is added to the end of the `frame_list` corresponding to the VRU object in the valid information dictionary `vru_traverse`.

[0127] For example, to help understand the implementation process of the scene recognition method obtained by combining this embodiment with the above embodiment one, specifically:

[0128] First, it is necessary to determine the total number of frames in the data segment, i.e., the frame length frame_num. This is achieved by calculating the number of frames contained in the data segment.

[0129] Next, we define a list, VRU_type_list, to store the object attribute categories belonging to VRUs. This list includes all VRU types that we want to identify, such as pedestrians, two-wheeled vehicles, and three-wheeled vehicles.

[0130] To determine whether the traversal behavior of a VRU object is valid, a time window length (duration) needs to be set. This time window will be used during subsequent frame data traversal to determine which frames are within the valid time window.

[0131] Define a dictionary vru_traverse to record the specific valid information of VRU crossing behavior. This dictionary will use the object index obj_id of the VRU as the key and a list containing the current frame index frame_idx as the value.

[0132] Traverse each frame of data data_item, whose frame index is frame_idx. For each frame of data, perform the following operations:

[0133] First, check whether the frame is within a valid time window. If frame_idx is greater than or equal to duration and frame_idx + duration is less than frame_num, then the frame is within a valid time window; otherwise, the frame is within an invalid time window. Only when the frame is within a valid time window can further judgment be made, otherwise skip this frame.

[0134] Obtain the obstacle obstacles information in this frame. This obstacle information will be used for subsequent VRU recognition and crossing behavior analysis.

[0135] For each obstacle in obstacles (object index is obj_id, object specific information is obj_info), perform the following operations:

[0136] If the object index is equal to 0 (indicating that the object is the ego vehicle) or the object does not belong to the predefined VRU category (i.e., the object category is not in VRU_type_list), then skip this obstacle.

[0137] Check whether the speed obj_v of the object VRU is less than 0.1. If it is less, then consider this VRU as stationary and also skip this obstacle.

[0138] Calculate the relative position (local_x, local_y) of the object VRU with respect to the vehicle in the ego vehicle coordinate system. First, obtain the position information of the ego vehicle in this frame, including the X coordinate ego_x, the Y coordinate ego_y, and the orientation angle ego_theta. Then, calculate local_x and local_y based on the coordinates (global_x, global_y) of the object obj_id.

[0139] Judge whether the object VRU is in front of the vehicle and close to the lane centerline. First, the conditions local_x > 0 and -1.1 < local_y < 1 need to be met to be a potential candidate for VRU crossing target. If the conditions are met, then further check the number of valid frames count_front and count_back of this VRU in the front and back time windows.

[0140] If local_x > 0 and -1.1 < local_y < 1, further check whether the number of valid frames count_front and count_back of the VRU exists in the front and back time windows.

[0141] Calculate count_front, which represents the number of frames that satisfy the VRU crossing behavior in the historical time within the specified frame range. This involves calculating the starting frame index p of the historical frames, initializing the counters c_left and c_right, traversing the frame data within the specified range, calculating the position change temp of the object VRU relative to the ego vehicle, and updating the c_left or c_right counter according to the value of temp.

[0142] Calculate count_back, which represents the number of frames that satisfy the VRU crossing behavior in the future time within the specified frame range. This is similar to the historical frame traversal but in the opposite direction, from the current frame frame_idx to the end frame index q of the future frames.

[0143] If count_front is None, or count_back is None, skip this object as the movement direction of this object is not clear enough. If the number of valid frames in both the front and back time windows is greater than 6, that is, count_front > 6 and count_back > 6, then consider this as a valid VRU crossing event and record this event.

[0144] To avoid duplicate recording of the VRU crossing behavior, if the object obj_id is not in the vru_traverse field, record the current frame frame_idx into vru_traverse. If obj_id is already in the vru_traverse field, further judge. Take the specific frame information list frame_list of the object obj_id in the valid information dictionary vru_traverse, and take the last element frame_end of the frame information list frame_list corresponding to the object obj_id. frame_end represents the frame index when the object was recently determined to have a VRU crossing behavior. If the current frame index frame_idx - frame_end < 20, it is determined that the object has a duplicate determination of the VRU crossing behavior in the current frame, and skip this object. Otherwise, add the current frame frame_idx to the tail of the frame information list frame_list corresponding to the object obj_id in vru_traverse.

[0145] After completing the above process, the resulting valid information dictionary `vru_traverse` contains the objects in the current data segment that exhibit VRU traversal behavior, along with their corresponding frame indices. This dictionary can be used for further analysis and processing to improve the vehicle's ability to identify and respond to VRU traversal scenarios during vehicle operation.

[0146] It should be noted that the above examples are only for understanding this application and do not constitute a limitation on the scene recognition method of this application. Any simple transformations based on this technical concept are within the protection scope of this application.

[0147] This application also provides a scene recognition device; please refer to... Figure 4 The scene recognition device includes:

[0148] Module 10 is used to acquire scene data;

[0149] The search module 20 is used to traverse all frames in the scene data, find the VRU objects in the current frame that are within the preset cross-monitoring range, until the traversal is complete.

[0150] The identification module 30 is used to identify the current scene data as a VRU object crossing the scene if the VRU object is continuously detected to have traversing behavior in multiple frames within the monitoring time interval centered on the current frame.

[0151] And / or, the lookup module 20 includes:

[0152] The first traversal module is used to traverse the frames in the scene data and take the traversed frame as the current frame.

[0153] The first extraction module is used to extract the location information of all VRU objects from the current frame if the monitoring time interval centered on the current frame does not exceed the scene data.

[0154] The first calculation module is used to calculate the relative position of each VRU object with respect to the vehicle coordinates based on the location information, and to find the VRU objects within the preset cross-traverse monitoring range.

[0155] And / or, the first computing module includes:

[0156] The first acquisition module is used to acquire the vehicle's coordinates and orientation angle.

[0157] The first calculation module is used to calculate the relative position of each VRU object relative to the vehicle based on the position information of all VRU objects, the vehicle coordinates, and the vehicle orientation angle.

[0158] The first search module is used to find VRU objects within a preset cross-monitoring range based on the relative position.

[0159] And / or, the identification module 30 includes:

[0160] The first determining module is used to determine the historical frame from which the traversal begins and the future frame from which the traversal ends, based on the current frame and the monitoring time interval.

[0161] The second traversal module is used to traverse within the historical frames before the current frame, calculate the relative position change of the VRU object, and determine the number of historical valid frames traversed by the VRU object.

[0162] The third traversal module is used to traverse within future frames after the current frame, calculate the relative position change of the VRU object, and determine the number of effective frames that the VRU object will traverse in the future.

[0163] The first judgment module is used to determine, based on the number of historical effective traversal frames and the number of future effective traversal frames, the scene data in which the VRU object exhibits effective traversal behavior during the monitoring time interval is identified as a VRU object traversal scene.

[0164] And / or, the second traversal module includes:

[0165] The fourth traversal module is used to traverse the historical frames to the current frame and calculate the relative position of the VRU object and the vehicle.

[0166] The first update module is used to update the historical valid movement trajectory of the VRU object during its traversal movement within the traversal monitoring range based on the changes in the relative position of the VRU object and the vehicle in each frame.

[0167] The second determining module is used to determine the number of historical traversal valid frames of the VRU object based on the historical valid action trajectory.

[0168] And / or, the third traversal module includes:

[0169] The fifth traversal module is used to traverse from the current frame to the future frame and calculate the relative position of the VRU object and the vehicle.

[0170] The second update module is used to update the future effective movement trajectory of the VRU object when it crosses the monitoring range, based on the change in the relative position of the VRU object and the vehicle in each frame.

[0171] The third determining module is used to determine the number of future valid traversal frames of the VRU object based on the future valid action trajectory.

[0172] And / or, the scene recognition device includes:

[0173] The first reading module is used to query the frame information list corresponding to the VRU object, read the frame index of the historical traversal event record of the VRU object, and determine the time interval with the frame index of the current frame.

[0174] The fourth determination module is used to determine the cross-traversal event of the VRU object of the current frame as a duplicate event if the time interval of the frame index is less than the preset duplicate determination threshold.

[0175] The fifth determining module is used to determine the traversal event of the VRU object of the current frame as a new event if the time interval of the frame index is not less than the preset repetition determination threshold, and update the frame information list of the current frame to the VRU object.

[0176] The scene recognition device provided in this application, employing the scene recognition method in the above embodiments, can solve the technical problem of low recognition accuracy of VRU traversing scenes when high-precision map information is lacking. Compared with the prior art, the beneficial effects of the scene recognition device provided in this application are the same as those of the scene recognition method provided in the above embodiments, and other technical features in the scene recognition device are the same as those disclosed in the methods of the above embodiments, and will not be repeated here.

[0177] This application provides a scene recognition device, which includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the scene recognition method in the above embodiment 1.

[0178] The following is for reference. Figure 5 The diagram illustrates a structural schematic suitable for implementing the scene recognition device in the embodiments of this application. The scene recognition device in the embodiments of this application may include, but is not limited to, mobile terminals such as mobile phones, tablets, laptops, digital broadcast receivers, PDAs (Personal Digital Assistants), PMPs (Portable Media Players), in-vehicle terminals (e.g., in-vehicle navigation terminals), and fixed terminals such as digital TVs and desktop computers. Figure 5 The scene recognition device shown is merely an example and should not impose any limitations on the functionality and scope of use of the embodiments of this application.

[0179] like Figure 5As shown, the scene recognition device may include a processing unit 1001 (e.g., a central processing unit, a graphics processing unit, etc.), which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 1002 or a program loaded from a storage device 1003 into a random access memory (RAM) 1004. The RAM 1004 also stores various programs and data required for the operation of the scene recognition device. The processing unit 1001, ROM 1002, and RAM 1004 are interconnected via a bus 1005. An input / output (I / O) interface 1006 is also connected to the bus. Typically, the following systems can be connected to the I / O interface 1006: input devices 1007 including, for example, a touchscreen, touchpad, keyboard, mouse, image sensor, microphone, accelerometer, gyroscope, etc.; output devices 1008 including, for example, a liquid crystal display (LCD), speaker, vibrator, etc.; storage devices 1003 including, for example, magnetic tape, hard disk, etc.; and communication devices 1009. Communication device 1009 allows the scene recognition device to communicate wirelessly or wiredly with other devices to exchange data. While the figures show scene recognition devices with various systems, it should be understood that implementing or having all of the systems shown is not required. More or fewer systems may be implemented alternatively.

[0180] Specifically, according to the embodiments disclosed in this application, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments disclosed in this application include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication device, or installed from storage device 1003, or installed from ROM 1002. When the computer program is executed by processing device 1001, it performs the functions defined in the methods of the embodiments disclosed in this application.

[0181] The scene recognition device provided in this application, employing the scene recognition method described in the above embodiments, can solve the technical problem of low recognition accuracy for VRU traversing scenes when high-precision map information is lacking. Compared with the prior art, the beneficial effects of the scene recognition device provided in this application are the same as those of the scene recognition method provided in the above embodiments, and other technical features of this scene recognition device are the same as those disclosed in the previous embodiment method, and will not be repeated here.

[0182] It should be understood that the various parts disclosed in this application can be implemented using hardware, software, firmware, or a combination thereof. In the description of the above embodiments, specific features, structures, materials, or characteristics can be combined in any suitable manner in one or more embodiments or examples.

[0183] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

[0184] This application provides a computer-readable storage medium having computer-readable program instructions (i.e., a computer program) stored thereon, the computer-readable program instructions being used to execute the scene recognition method in the above embodiments.

[0185] The computer-readable storage medium provided in this application may be, for example, a USB flash drive, but is not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to: electrical connections having one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this embodiment, the computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, system, or device. The program code contained on the computer-readable storage medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (Radio Frequency), etc., or any suitable combination thereof.

[0186] The aforementioned computer-readable storage medium may be included in the scene recognition device; or it may exist independently and not be assembled into the scene recognition device.

[0187] The aforementioned computer-readable storage medium carries one or more programs. When these programs are executed by the scene recognition device, the scene recognition device: acquires scene data; traverses all frames in the scene data, searches for VRU objects traversing a preset monitoring range in the current frame, until traversal is complete; if, in multiple frames within a monitoring time interval centered on the current frame, the VRU objects are continuously detected to be traversing the scene, then the current scene data is identified as a VRU object traversing the scene. Computer program code for performing the operations of this application can be written in one or more programming languages ​​or a combination thereof. These programming languages ​​include object-oriented programming languages—such as Java, Smalltalk, and C++—and conventional procedural programming languages—such as the "C" language or similar programming languages. The program code can be executed entirely on a user's computer, partially on a user's computer, as a standalone software package, partially on a user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or it can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0188] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0189] The modules described in the embodiments of this application can be implemented in software or hardware. The names of the modules do not necessarily limit the functionality of the unit itself.

[0190] The readable storage medium provided in this application is a computer-readable storage medium that stores computer-readable program instructions (i.e., a computer program) for executing the above-described scene recognition method. This solves the technical problem of low accuracy in recognizing VRU traversing scenes when high-precision map information is lacking. Compared with the prior art, the beneficial effects of the computer-readable storage medium provided in this application are the same as those of the scene recognition method provided in the above embodiments, and will not be elaborated upon here.

[0191] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the scene recognition method described above.

[0192] The computer program product provided in this application can solve the technical problem of low accuracy in recognizing VRU traversing scenes when high-precision map information is lacking. Compared with the prior art, the beneficial effects of the computer program product provided in this application are the same as those of the scene recognition method provided in the above embodiments, and will not be repeated here.

[0193] All acquisition of signals, information, or actions in this application are carried out in compliance with the relevant data protection laws and policies of the country where the application is located, and with the authorization of the relevant device owner.

[0194] The above description is only a part of the embodiments of this application and does not limit the patent scope of this application. All equivalent structural transformations made under the technical concept of this application and using the contents of the specification and drawings of this application, or direct / indirect applications in other related technical fields, are included in the patent protection scope of this application.

Claims

1. A scene recognition method, characterized in that, The method includes: Acquire scene data; Traverse all frames in the scene data, find the VRU objects within the preset cross-monitoring range in the current frame, and continue traversing until the traversal is complete; If the VRU object is continuously detected to be traversing across multiple frames within the monitoring time interval centered on the current frame, then the current scene data is identified as the VRU object traversing the scene.

2. The method as described in claim 1, characterized in that, The step of traversing all frames in the scene data and finding VRU objects within the preset cross-monitoring range in the current frame until the traversal is complete includes: Iterate through all frames in the scene data and take the traversed frame as the current frame; If the monitoring time interval centered on the current frame does not exceed the scene data, extract the location information of all VRU objects from the current frame; Based on the location information, the relative position of each VRU object with respect to the vehicle coordinates is calculated, and VRU objects within the preset cross-traverse monitoring range are identified.

3. The method as described in claim 2, characterized in that, The step of calculating the relative position of each VRU object with respect to the vehicle coordinates based on the location information, and finding the VRU objects within the preset cross-traverse monitoring range, includes: Obtain the vehicle's coordinates and heading angle; Based on the location information of all VRU objects, the vehicle coordinates, and the vehicle orientation angle, calculate the relative position of each VRU object relative to the vehicle. Based on the relative position, locate the VRU object that is within the preset cross-monitoring range.

4. The method as described in claim 1, characterized in that, The step of identifying the current scene data as a VRU object traversing the scene if the VRU object is continuously detected to be traversing the scene across multiple frames within the monitoring time interval centered on the current frame includes: Based on the current frame and the monitoring time interval, determine the historical frame from which the traversal begins and the future frame from which the traversal ends; Traverse the historical frames preceding the current frame, calculate the relative position change of the VRU object, and determine the number of valid historical frames traversed by the VRU object. Traverse the future frames following the current frame, calculate the relative position change of the VRU object, and determine the number of effective frames the VRU object will traverse in the future. Based on the historical number of valid cross-traversal frames and the future number of valid cross-traversal frames, the scene data in which the VRU object exhibits valid cross-traversal behavior within the monitoring time interval is identified as a VRU object cross-traversal scene.

5. The method as described in claim 4, characterized in that, The step of traversing historical frames prior to the current frame, calculating the relative position change of the VRU object, and determining the number of valid historical frames traversed by the VRU object includes: Traverse the historical frames to the current frame and calculate the relative position of the VRU object and the vehicle. Based on the changes in the relative position of the VRU object and the vehicle in each frame, update the historical valid movement trajectory of the VRU object during its traversal movement within the traversal monitoring range; Based on the historical valid action trajectory, determine the number of historical valid frames traversed by the VRU object.

6. The method as described in claim 4, characterized in that, The step of traversing future frames after the current frame, calculating the relative position change of the VRU object, and determining the number of valid frames the VRU object will traverse in the future includes: Traverse from the current frame to the future frame to calculate the relative position of the VRU object and the vehicle. Based on the changes in the relative position of the VRU object and the vehicle in each frame, update the future effective movement trajectory of the VRU object during its traversal movement within the traversal monitoring range; Based on the future valid action trajectory, determine the number of future valid frames traversed by the VRU object.

7. The method as described in claim 1, characterized in that, The step of identifying the current scene data as a VRU object traversing the scene if the VRU object is continuously detected to be traversing the scene across multiple frames within the monitoring time interval centered on the current frame includes the following: Query the frame information list corresponding to the VRU object, read the frame index of the historical traversal event record of the VRU object, and determine the time interval with the frame index of the current frame; If the time interval of the frame index is less than the preset repetition determination threshold, then the traversal event of the VRU object of the current frame is determined to be a repetition determination event. If the time interval of the frame index is not less than the preset repetition determination threshold, then the traversal event of the VRU object of the current frame is determined as a new event, and the frame information list of the current frame to the VRU object is updated.

8. A scene recognition device, characterized in that, The device includes: The acquisition module is used to acquire scene data; The search module is used to traverse all frames in the scene data, find the VRU objects that are within the preset cross-monitoring range in the current frame, until the traversal is complete; The identification module is used to identify the current scene data as a VRU object crossing the scene if the VRU object is continuously detected to be crossing the scene in multiple frames within the monitoring time interval centered on the current frame.

9. A scene recognition device, characterized in that, The device includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, the computer program being configured to implement the steps of the scene recognition method as described in any one of claims 1 to 7.

10. A storage medium, characterized in that, The storage medium is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by a processor, it implements the steps of the scene recognition method as described in any one of claims 1 to 7.

11. A computer program product, characterized in that, The computer program product includes a computer program that, when executed by a processor, implements the steps of the scene recognition method as described in any one of claims 1 to 7.

Images