Driving environment sensing method and device and electronic equipment
By matching and fusion of candidate perception boxes in the current environmental image with historical perception boxes in the autonomous driving system, the problem of instability in the generation of perception boxes is solved, the stability and accuracy of perception boxes are improved, and the decision quality of the autonomous driving system is improved.
Patent Information
- Application Number
- CN202311386079.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2023-10-24
- Publication Date
- 2025-04-25
AI Technical Summary
In an autonomous driving environment, due to factors such as mutual occlusion of objects and light changes, the generation of perception boxes in the prior art is unstable, which affects subsequent operation decisions.
By obtaining the candidate perception box of the target object from the current environment image and matching it with the perception box in the historical perception box set, the perception box to be fused is obtained using the overlap degree, and fuses to update the historical perception box set.
It improves the stability and accuracy of the perceptual box, reduces the generation of instability problems, and thus improves the decision-making quality of the autonomous driving system.
Smart Images
Figure CN119888635A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of assisted driving technology, and in particular to a driving environment perception method, device and electronic equipment. Background Art
[0002] In the autonomous driving function, it is necessary to first use environmental perception technology to identify the specific location of each target object in the environment. The existing method is mainly to use on-board sensors to collect environmental images, and then use recognition models to identify target objects in the environment, and use perception frames to mark their specific locations in the environmental images. After that, the specific location information of this target object can be combined to complete subsequent operational decisions. However, in the actual driving environment, due to interference factors such as mutual occlusion between objects and changes in lighting, the perception frame will often have a series of unstable generation problems such as position, orientation, size, and perception frame splitting, which will inevitably affect subsequent operational decisions. Therefore, how to obtain the perception frame corresponding to the obstacle and avoid the problem of unstable perception frame generation has become an urgent problem to be solved. Summary of the invention
[0003] The embodiments of the present application provide a driving environment perception method, device and electronic device for solving the problem of unstable perception frame generation.
[0004] In a first aspect, an embodiment of the present application provides a driving environment perception method, the method comprising:
[0005] Obtain at least one candidate perception box of the target object from the current environment image;
[0006] Matching each candidate perception frame with a perception frame in a set of historical perception frames, and obtaining at least one perception frame to be fused using a degree of overlap between the candidate perception frame and the perception frame in the set of historical perception frames;
[0007] For each perception frame to be fused, taking the perception frame in the historical perception frame set that has the largest overlap with the perception frame to be fused as the matching perception frame corresponding to the perception frame to be fused;
[0008] For each matching perception frame, the matching perception frame is fused with the corresponding perception frame to be fused to obtain the updated historical perception frame set.
[0009] As an optional implementation of the embodiment of the present application, the step of acquiring at least one candidate perception box of the target object from the current environment image includes:
[0010] Identifying at least one observation perception frame from the current environment image;
[0011] An observation perception frame whose ratio of length to width falls within a preset range is selected as at least one candidate perception frame of the target object.
[0012] As an optional implementation of the embodiment of the present application, matching each candidate perception frame with a perception frame in a set of historical perception frames, and obtaining at least one perception frame to be fused by using the overlap between the candidate perception frame and the perception frame in the set of historical perception frames, includes performing the following steps for each candidate perception frame:
[0013] Obtaining the overlap between the candidate perception frame and each perception frame in the historical perception frame set;
[0014] According to the overlap between the candidate perception frame and each perception frame in the set of historical perception frames, a maximum overlap between the candidate perception frame and each perception frame in the set of historical perception frames is obtained, where the maximum overlap between the candidate perception frame and each perception frame in the set of historical perception frames is the maximum value of the overlap between the candidate perception frame and each perception frame in the set of historical perception frames;
[0015] When the maximum overlap of the candidate perception frames is greater than the threshold overlap, the candidate perception frame is determined as the perception frame to be fused.
[0016] As an optional implementation of the embodiment of the present application, for each matching perception frame, fusing the matching perception frame with the corresponding perception frame to be fused includes performing the following steps for each matching perception frame:
[0017] If the matched perception frame includes multiple perception frames to be fused, the multiple perception frames to be fused corresponding to the matched perception frame are merged to obtain a merged perception frame, and weighted fusion is performed on the matched perception frame and the merged perception frame;
[0018] If the matching perception frame corresponds to one perception frame to be fused, weighted fusion is performed on the matching perception frame and the corresponding perception frame to be fused.
[0019] As an optional implementation of the embodiment of the present application, after obtaining the updated historical perception frame set, the method further includes:
[0020] For each perception frame in the updated historical perception frame set, if the perception frame is used as a corresponding matching perception frame by the to-be-fused perception frames in the environment images for more than a first preset number of consecutive times, the perception frame is marked as a stable tracking perception frame; if the to-be-fused perception frames in the environment images for more than a second preset number of consecutive times do not use the perception frame as a corresponding matching perception frame, the perception frame is deleted from the historical perception frame set;
[0021] All of the stable tracking perception frames in the updated historical perception frame set are displayed.
[0022] As an optional implementation of the embodiment of the present application, before displaying all the stable tracking perception frames in the updated historical perception frame set, the method further includes:
[0023] horizontally merging two stable tracking perception frames that meet the first merging condition;
[0024] Among them, the first merging condition includes: the direction angle is less than the threshold angle, the longitudinal distance and the lateral distance are both less than half the width of the larger stable tracking perception frame; the direction of the stable tracking perception frame is the length direction of the stable tracking perception frame.
[0025] As an optional implementation of the embodiment of the present application, before displaying all the stable tracking perception frames in the updated historical perception frame set, the method further includes:
[0026] Merging two stable tracking perception frames that meet the second merging condition vertically;
[0027] Among them, the second merging condition includes: the direction angle is less than the threshold angle, the longitudinal distance is less than one-third of the length of the larger stable tracking perception frame, and the lateral distance is less than twice the width of the larger stable tracking perception frame; the direction of the stable tracking perception frame is the length direction of the stable tracking perception frame.
[0028] As an optional implementation of the embodiment of the present application, after obtaining the updated historical perception frame set, the method further includes:
[0029] If the distance between two perception frames in the updated historical perception frame set is less than the distance threshold and the angle between them belongs to the preset angle range, one of the two perception frames is deleted from the updated historical perception frame set.
[0030] In a second aspect, an embodiment of the present application provides a driving environment perception device, comprising:
[0031] An acquisition module, used to acquire at least one candidate perception box of the target object from the current environment image;
[0032] A matching module, used to match each candidate perception frame with a perception frame in a set of historical perception frames, and obtain at least one perception frame to be fused by using the overlap between the candidate perception frame and the perception frame in the set of historical perception frames;
[0033] The fusion module is used to, for each perception frame to be fused, use the perception frame with the largest overlap with the perception frame to be fused in the historical perception frame set as the matching perception frame corresponding to the perception frame to be fused, and for each matching perception frame, fuse the matching perception frame with the corresponding perception frame to be fused to obtain the updated historical perception frame set.
[0034] As an optional implementation of the embodiment of the present application, the acquisition module is specifically used to identify at least one observation perception frame from the current environment image; and select an observation perception frame whose length and width ratio falls within a preset range as at least one candidate perception frame of the target object.
[0035] As an optional implementation of an embodiment of the present application, the matching module is specifically used to perform the following steps for each candidate perceptual frame: obtaining the degree of overlap between the candidate perceptual frame and each perceptual frame in the set of historical perceptual frames; obtaining the maximum overlap of the candidate perceptual frame based on the degree of overlap between the candidate perceptual frame and each perceptual frame in the set of historical perceptual frames, the maximum overlap of the candidate perceptual frame being the maximum value of the degree of overlap between the candidate perceptual frame and each perceptual frame in the set of historical perceptual frames; when the maximum overlap of the candidate perceptual frame is greater than a threshold overlap, determining the candidate perceptual frame as a perceptual frame to be fused.
[0036] As an optional implementation of an embodiment of the present application, the matching module is specifically used to perform the following steps for each matching perception frame: if the matching perception frame corresponds to multiple perception frames to be fused, the multiple perception frames to be fused corresponding to the matching perception frame are merged to obtain a merged perception frame, and the matching perception frame and the merged perception frame are weightedly fused; if the matching perception frame corresponds to one perception frame to be fused, the matching perception frame and the corresponding perception frame to be fused are weightedly fused.
[0037] As an optional implementation of the embodiment of the present application, the fusion module is also used to, after obtaining the updated historical perception frame set, for each perception frame in the updated historical perception frame set, if the perception frame is used as a corresponding matching perception frame by the to-be-fused perception frames in the environmental images for more than a first preset number of consecutive times, then the perception frame is marked as a stable tracking perception frame; if the to-be-fused perception frames in the environmental images for more than a second preset number of consecutive times do not use the perception frame as a corresponding matching perception frame, then the perception frame is deleted from the historical perception frame set; and all the stable tracking perception frames in the updated historical perception frame set are displayed.
[0038] As an optional implementation of the embodiment of the present application, the fusion module is further used to horizontally merge the stable tracking perception frames that meet the first merging condition before displaying all the stable tracking perception frames in the updated historical perception frame set;
[0039] Among them, the first merging condition includes: the direction angle is less than the threshold angle, the longitudinal distance and the lateral distance are both less than half the width of the larger stable tracking perception frame; the direction of the stable tracking perception frame is the length direction of the stable tracking perception frame.
[0040] As an optional implementation of the embodiment of the present application, the fusion module is further used to vertically merge the stable tracking perception frames that meet the second merging condition before displaying all the stable tracking perception frames in the updated historical perception frame set;
[0041] Among them, the second merging condition includes: the direction angle is less than the threshold angle, the longitudinal distance is less than one-third of the length of the larger stable tracking perception frame, and the lateral distance is less than twice the width of the larger stable tracking perception frame; the direction of the stable tracking perception frame is the length direction of the stable tracking perception frame.
[0042] As an optional implementation of the embodiment of the present application, the fusion module is also used to, after obtaining the updated historical perception frame set, if the distance between two perception frames in the updated historical perception frame set is less than a distance threshold and the angle belongs to a preset angle range, delete one of the two perception frames from the updated historical perception frame set.
[0043] In a third aspect, an embodiment of the present application provides an electronic device, comprising: a processor, a memory, and a computer program stored in the memory and executable on the processor, wherein the computer program, when executed by the processor, implements a driving environment perception method as described in any one of the above items.
[0044] The driving environment perception method provided in the embodiment of the present application first obtains at least one candidate perception frame of the target object from the current environment image; then matches each candidate perception frame with the perception frame in the historical perception frame set, and obtains at least one perception frame to be fused by using the overlap between the candidate perception frame and the perception frame in the historical perception frame set; then, for each perception frame to be fused, the perception frame in the historical perception frame set with the largest overlap with the perception frame to be fused is used as the matching perception frame corresponding to the perception frame to be fused; finally, for each matching perception frame, the matching perception frame is fused with the corresponding perception frame to be fused to obtain the updated historical perception frame set. Since the driving environment perception method provided in the embodiment of the present application obtains at least one perception frame to be fused by using the overlap between the selected perception frame and the perception frame in the historical perception frame set, and the matching perception frame and the corresponding perception frame to be fused can be fused to update the perception frame in the historical perception frame set, the embodiment of the present application can solve the problem of unstable perception frame generation. BRIEF DESCRIPTION OF THE DRAWINGS
[0045] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.
[0046] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.
[0047] Figure 1 It is a flowchart of an embodiment of the driving environment perception method provided by the embodiment of the present application;
[0048] Figure 2 is a flowchart of another embodiment of the driving environment perception method provided in an embodiment of the present application;
[0049] Figure 3 is a simplified structural diagram of any two stable tracking perception frames that meet the first merging condition in the driving environment perception method provided in an embodiment of the present application;
[0050] Figure 4 is a simplified structural diagram of any two stable tracking perception frames that meet the second merging condition in the driving environment perception method provided in an embodiment of the present application;
[0051] Figure 5 It is a simplified structural diagram of the third tracked perception frame and the fourth tracked perception frame that meet the filtering condition in the driving environment perception method provided in an embodiment of the present application;
[0052] Figure 6 A schematic diagram of the structure of a driving environment perception device provided in an embodiment of the present application;
[0053] Figure 7 It is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION
[0054] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.
[0055] In the following description, for the purpose of explanation rather than limitation, specific details such as specific system structures, interfaces, and technologies are provided to facilitate a thorough understanding of the present application.
[0056] The terms "first", "second", and "third" in the embodiments of the present application are only used for descriptive purposes, and cannot be understood as indicating or implying relative importance or implicitly indicating the number of indicated technical features. Thus, the features defined as "first", "second", and "third" can expressly or implicitly include at least one of the features. In the description of the embodiments of the present application, the meaning of "multiple" is at least two, such as two, three, etc., unless otherwise clearly and specifically defined. In the embodiments of the present application, all directional indications (such as up, down, left, right, front, back...) are only used to explain the relative position relationship, movement, etc. between the components under a certain specific posture (as shown in the accompanying drawings). If the specific posture changes, the directional indication also changes accordingly. The terms "including" and "having" in the embodiments of the present application and any of their variations are intended to cover non-exclusive inclusions. For example, a process, method, system, product, or device that includes a series of steps or modules is not limited to the listed steps or modules, but optionally also includes steps or modules that are not listed, or optionally also includes other steps or components inherent to these processes, methods, products, or devices.
[0057] Reference to "embodiments" herein means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase in various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein may be combined with other embodiments.
[0058] The present application is described in detail below with reference to the accompanying drawings and embodiments.
[0059] The present application provides an embodiment of a driving environment perception method. Figure 1 As shown, the driving environment perception method includes the following steps:
[0060] S110. Obtain at least one candidate perception box of the target object from the current environment image.
[0061] Specifically, the driving environment is the driving environment of the vehicle, and an image acquisition device is installed on the vehicle to acquire an environmental image of the driving environment. The image acquisition device may be a vehicle-mounted camera, a driving recorder, or other device, and the image acquisition device may continuously acquire an environmental image of the driving environment of the vehicle while the vehicle is driving.
[0062] Exemplarily, the target object may be a ground sign, specifically a sidewalk, a parking space, etc.
[0063] S120: Match each candidate perception frame with a perception frame in the set of historical perception frames, and obtain at least one perception frame to be fused by using the overlap between the candidate perception frame and the perception frame in the set of historical perception frames.
[0064] In some embodiments, the overlap between the candidate perceptual frame and the perceptual frame in the historical perceptual frame set may be the intersection over union (IOU) of the candidate perceptual frame and the perceptual frame in the historical perceptual frame set. That is, denoting a candidate perceptual frame as A and denoting a perceptual frame in the historical perceptual frame set as B, the overlap between the candidate perceptual frame A and the perceptual frame B in the historical perceptual frame set is (A∩B) / (A∪B).
[0065] In some embodiments, the overlap between the candidate perceptual frame and the perceptual frame in the historical perceptual frame set may be the ratio of the intersection of the candidate perceptual frame and the perceptual frame in the historical perceptual frame set to the perceptual frame in the historical perceptual frame set (Intersection Over Frame, IOF). That is, denoting a candidate perceptual frame as A and denoting a perceptual frame in the historical perceptual frame set as B, the overlap between the candidate perceptual frame A and the perceptual frame B in the historical perceptual frame set is (A∩B) / B.
[0066] In some embodiments, the overlap between the candidate perceptual frame and the perceptual frame in the historical perceptual frame set may be the ratio of the intersection of the candidate perceptual frame and the perceptual frame in the historical perceptual frame set to the perceptual frame with the smaller area. That is, denoting a candidate perceptual frame as A and denoting a perceptual frame in the historical perceptual frame set as B, the overlap between the candidate perceptual frame A and the perceptual frame B in the historical perceptual frame set is (A∩B) / min(A,B).
[0067] In some embodiments, the overlap between the candidate perceptual frame and the perceptual frame in the historical perceptual frame set may be the ratio of the intersection of the candidate perceptual frame and the perceptual frame in the historical perceptual frame set to the perceptual frame with the larger area. That is, denoting a candidate perceptual frame as A and denoting a perceptual frame in the historical perceptual frame set as B, the overlap between the candidate perceptual frame A and the perceptual frame B in the historical perceptual frame set is (A∩B) / max(A,B).
[0068] S130 . For each perception frame to be fused, take a perception frame in the set of historical perception frames that has the greatest overlap with the perception frame to be fused as a matching perception frame corresponding to the perception frame to be fused.
[0069] For example: the historical perception frame set includes 4 perception frames 1, 2, 3 and 4, and the overlap degrees of the perception frame A to be fused with the perception frame 1, 2, 3 and 4 are a, b, c and d respectively, and c>a>d>b, then the matching perception frame corresponding to the perception frame A to be fused is perception frame 3.
[0070] S140 . For each matching perception frame, fuse the matching perception frame with the corresponding perception frame to be fused to obtain the updated historical perception frame set.
[0071] Since the above embodiment will fuse the matching perception frame with the corresponding perception frame to be fused for each matching perception frame to obtain the updated historical perception frame set, the updated historical perception frame set can be used to obtain the perception frame to be fused for the next collected environmental image (the next frame of environmental image), and the updated historical perception frame set can be updated again according to the perception frame to be fused in the next collected environmental image. Therefore, the above embodiment can continuously update the historical perception frame set based on the environmental image, improve the accuracy of the perception frame in the historical perception frame set, and further improve the stability of the perception frame to be fused in the environmental image determined based on the historical perception frame set.
[0072] In some embodiments, after obtaining the updated historical perception frame set, all the perception frames of the historical perception frame set may be displayed, or the display may be performed after executing the following preferred steps.
[0073] The driving environment perception method provided in the embodiment of the present application first obtains at least one candidate perception frame of the target object from the current environment image; then matches each candidate perception frame with the perception frame in the historical perception frame set, and obtains at least one perception frame to be fused by using the overlap between the candidate perception frame and the perception frame in the historical perception frame set; then, for each perception frame to be fused, the perception frame in the historical perception frame set with the largest overlap with the perception frame to be fused is used as the matching perception frame corresponding to the perception frame to be fused; finally, for each matching perception frame, the matching perception frame is fused with the corresponding perception frame to be fused to obtain the updated historical perception frame set. Since the driving environment perception method provided in the embodiment of the present application obtains at least one perception frame to be fused by using the overlap between the selected perception frame and the perception frame in the historical perception frame set, and the matching perception frame and the corresponding perception frame to be fused can be fused to update the perception frame in the historical perception frame set, the embodiment of the present application can solve the problem of unstable perception frame generation.
[0074] As an extension and refinement of the above embodiment, the present application embodiment provides another embodiment of the driving environment perception method, referring to Figure 2 As shown, the driving environment perception method of this embodiment includes the following steps:
[0075] S201. Identify at least one observation perception box from the current environment image.
[0076] In step S201, the environmental image of the driving environment may be acquired periodically; for example, the period may be 1 second, and the environmental image acquired at the current moment is the current environmental image.
[0077] In some embodiments, the current environment image can be input into a target object detection model, and at least one candidate perception frame of the target object output by the target object detection model can be obtained. The target object detection model is a model obtained by training a machine learning model based on sample data, and the sample data includes multiple sample target objects and perception frames corresponding to the multiple targets. For example, if the target object is a sidewalk, then multiple sample sidewalks and the perception frames corresponding to the multiple sample sidewalks can be input into the machine learning model so that the machine learning model learns the characteristics of the sidewalk, and after the machine learning model obtains the sidewalk model, the current environment image is input into the sidewalk model to obtain at least one candidate perception frame of the sidewalk output by the sidewalk detection model.
[0078] S202: Select an observation perception frame whose length-to-width ratio falls within a preset range as at least one candidate perception frame of the target object.
[0079] In some embodiments, the above step S202 includes: for each observation perception frame, obtaining the ratio of the length and width of the observation perception frame, and determining whether the ratio of the length and width of the observation perception frame is within the preset range; if the ratio of the length and width of the observation perception frame is within the preset range, determining the observation perception frame as the candidate perception frame; if the ratio of the length and width of the observation perception frame is not within the preset range, discarding the observation perception frame.
[0080] Exemplarily, the preset range may be [2, 8], that is, an observed perception frame whose length-to-width ratio is greater than or equal to 2 and less than or equal to 8 is selected as a candidate perception frame.
[0081] By selecting an observed perception frame whose length-to-width ratio falls within a preset range as a candidate perception frame, it is possible to further avoid erroneous recognition when identifying the candidate perception frame.
[0082] It should be noted that steps S201 and S202 are specific implementation steps of obtaining at least one candidate perception box of the target object from the environment image in S110 of the above embodiment.
[0083] After obtaining at least one candidate perception box of the target object through steps S201 and S202, the embodiment of the present application continues to perform the following steps S203 to S205 for each candidate perception box:
[0084] S203: Obtain the degree of overlap between the candidate perception frame and each perception frame in the historical perception frame set.
[0085] S204: Obtain a maximum degree of overlap of the candidate perception frame according to the degree of overlap of the candidate perception frame with each perception frame in the set of historical perception frames.
[0086] The maximum overlap of the candidate perception frame is the maximum value of the overlap between the candidate perception frame and each perception frame in the historical perception frame set.
[0087] For example: the historical perception frame set includes 4 perception frames 1, 2, 3 and 4, and the overlaps of the perception frame A to be fused with the perception frame 1, 2, 3 and 4 are a, b, c and d respectively, and c>a>d>b, then the matching perception frame corresponding to the perception frame A to be fused is perception frame 3, and the maximum overlap of the perception frame A to be fused is c.
[0088] S205: When the maximum overlap of the candidate perception frames is greater than a threshold overlap, the candidate perception frame is determined as a perception frame to be fused.
[0089] Exemplarily, the threshold overlap may be 0.4. That is, if the maximum overlap of the candidate perception frames is greater than 0.4, the candidate perception frame is determined as the perception frame to be fused, and if the maximum overlap of the candidate perception frame is less than or equal to 0.4, the candidate perception frame is discarded.
[0090] It should be noted that steps S203 to S205 are specific implementation steps of matching each candidate perception frame with a perception frame in a set of historical perception frames in S120 of the above embodiment, and obtaining at least one perception frame to be fused by using the overlap between the candidate perception frame and the perception frame in the set of historical perception frames.
[0091] After obtaining at least one perception frame to be fused through steps S203 to S205, the embodiment of the present application continues to perform the following step S206 for each perception frame to be fused:
[0092] S206: taking the perception frame in the historical perception frame set that has the greatest overlap with the perception frame to be fused as the matching perception frame corresponding to the perception frame to be fused.
[0093] As described in the above example, the historical perception frame set includes 4 perception frames 1, 2, 3 and 4, and the overlap degrees of the perception frame A to be fused with the perception frame 1, 2, 3 and 4 are a, b, c and d respectively, and c>a>d>b, then the matching perception frame corresponding to the perception frame A to be fused is perception frame 3.
[0094] In the above step S206, the perception frame with the largest overlap with the perception frame to be fused in the historical perception frame set is used as the matching perception frame corresponding to the perception frame to be fused, and at least one matching perception frame can be obtained. The driving environment perception method provided in the embodiment of the present application further includes executing the following steps S207 and S209 for each matching perception frame:
[0095] If the matching perception frame corresponds to multiple perception frames to be fused, the following steps S207 and S208 are performed:
[0096] S207: Merge multiple perception frames to be fused corresponding to the matching perception frame to obtain a merged perception frame.
[0097] In some embodiments, multiple to-be-fused perceptual frames corresponding to the matching perceptual frame are merged to obtain a merged perceptual frame, including: obtaining a union of multiple to-be-fused perceptual frames corresponding to the matching perceptual frame, and using the union of multiple to-be-fused perceptual frames corresponding to the matching perceptual frame as the merged perceptual frame.
[0098] For example, if the matching perception frames corresponding to the perception frame 1 to be fused, the perception frame 2 to be fused, and the perception frame 3 to be fused are the same perception frame, the perception frames can be merged into the union of the perception frame 1 to be fused, the perception frame 2 to be fused, and the perception frame 3 to be fused.
[0099] S208: Perform weighted fusion on the matching perception box and the merged perception box.
[0100] In some embodiments, weighted fusion of the matching perceptual frame and the merged perceptual frame includes: weighted fusion of directions, positions, and sizes of the matching perceptual frame and the merged perceptual frame according to weight values of the matching perceptual frame and the merged perceptual frame.
[0101] For example, if the length of the merged perceptual frame is a, the length of the matching perceptual frame is b, the weight value of the merged perceptual frame is 0.1, and the weight value of the matching perceptual frame is 0.9, then the length of the perceptual frame obtained by weighted fusion of the matching perceptual frame and the merged perceptual frame is 0.1a+0.9b.
[0102] If the perception frame to be fused corresponding to the matching perception frame includes one, the following step S209 is performed:
[0103] S209: Perform weighted fusion on the matching perception frame and the corresponding perception frame to be fused.
[0104] In some embodiments, weighted fusion is performed on the matching perception frame and the corresponding perception frame to be fused, including: weighted fusion of the direction, position, and size of the matching perception frame and the corresponding perception frame to be fused according to the weight values of the matching perception frame and the perception frame to be fused.
[0105] In some embodiments, after the above step S130 determines that the first candidate perception frame matches the first tracked perception frame successfully, the driving environment perception method provided by the embodiment of the present application further includes performing the following steps a to c for each perception frame in the updated historical perception frame set:
[0106] Step a: If the perception frame is continuously used as a corresponding matching perception frame by the to-be-fused perception frames in the environment images exceeding a first preset number, the perception frame is marked as a stable tracking perception frame.
[0107] That is, if a candidate perception frame obtained from the environment image continuously exceeding the first preset number of times uses a perception frame in the updated historical perception frame set as a matching perception frame, then the perception frame in the updated historical perception frame set is marked as a stable tracking perception frame.
[0108] Exemplarily, the first preset number may be 5.
[0109] Step b: if the perception frames to be fused in the environment images exceeding a second preset number continuously do not use the perception frame as the corresponding matching perception frame, then the perception frame is deleted from the historical perception frame set.
[0110] That is, if the candidate perception frames obtained from the environment images continuously exceeding the second preset number of times do not use a perception frame in the updated historical perception frame set as a matching perception frame, the perception frame is deleted from the updated historical perception frame set.
[0111] Exemplarily, the second preset number may be 10.
[0112] Specifically, if the second tracked perception frame fails to successfully match the candidate perception frame in the environment image for more than 10 consecutive frames, the second tracked perception frame is deleted from the historical perception frame set.
[0113] Step c: displaying all the stable tracking perception frames in the updated historical perception frame set.
[0114] In some embodiments, before displaying all the stable tracking perception frames in the updated historical perception frame set, the driving environment perception method provided by the embodiment of the present application further includes: merging the stable tracking perception frames, and providing the following two merging methods of the stable tracking perception frames:
[0115] Merging method (1): horizontally merging two stable tracking perception frames that meet the first merging condition.
[0116] like Figure 3 As shown, the first merging condition includes: the merging angle α is less than the threshold angle, the longitudinal distance m and the lateral distance n are less than half of the width a of the larger stable tracking perception frame 300; the direction of the stable tracking perception frame is the length direction y of the stable tracking perception frame.
[0117] Merging method (2): vertically merge any two stable tracking perception frames that meet the second merging condition.
[0118] like Figure 4 As shown, the second merging condition includes: the direction angle is less than the threshold angle, the longitudinal distance m is less than one-third of the length b of the larger stable tracking perception frame 400, and the lateral distance n is less than twice the width a of the larger stable tracking perception frame 400; the direction of the stable tracking perception frame is the length direction y of the stable tracking perception frame.
[0119] In some embodiments, in order to further enable the driving environment perception method to perceive a more stable tracked perception frame, the embodiment of the present application also provides a method for filtering abnormal state perception frames in the historical perception frame set, specifically including:
[0120] like Figure 5 As shown, if the distance h between two perception frames in the updated historical perception frame set is less than the distance threshold and the angle β belongs to the preset angle range, one of the two perception frames is deleted from the updated historical perception frame set.
[0121] The present application embodiment provides a driving environment perception device, see Figure 6 , Figure 6 600 is a schematic diagram of the structure of a driving environment perception device provided in an embodiment of the present application. The driving environment perception device 600 includes:
[0122] An acquisition module 61 is used to acquire at least one candidate perception box of a target object from a current environment image;
[0123] A matching module 62, configured to match each candidate perception frame with a perception frame in a set of historical perception frames, and obtain at least one perception frame to be fused by using a degree of overlap between the candidate perception frame and the perception frame in the set of historical perception frames;
[0124] The fusion module 63 is used to, for each perception frame to be fused, use the perception frame in the historical perception frame set that has the largest overlap with the perception frame to be fused as the matching perception frame corresponding to the perception frame to be fused, and for each matching perception frame, fuse the matching perception frame with the corresponding perception frame to be fused to obtain an updated historical perception frame set.
[0125] As an optional implementation of the embodiment of the present application, the acquisition module 61 is specifically used to identify at least one observation perception frame from the current environment image; and select an observation perception frame whose length and width ratio falls within a preset range as at least one candidate perception frame of the target object.
[0126] As an optional implementation of the embodiment of the present application, the matching module 62 is specifically used to perform the following steps for each candidate perceptual frame: obtaining the degree of overlap between the candidate perceptual frame and each perceptual frame in the set of historical perceptual frames; obtaining the maximum degree of overlap of the candidate perceptual frame according to the degree of overlap between the candidate perceptual frame and each perceptual frame in the set of historical perceptual frames, the maximum degree of overlap of the candidate perceptual frame being the maximum value of the degree of overlap between the candidate perceptual frame and each perceptual frame in the set of historical perceptual frames; when the maximum degree of overlap of the candidate perceptual frame is greater than the threshold degree of overlap, the candidate perceptual frame is determined as the perceptual frame to be fused.
[0127] As an optional implementation of the embodiment of the present application, the matching module 62 is specifically used to perform the following steps for each matching perception frame: if the matching perception frame corresponds to multiple perception frames to be fused, the multiple perception frames to be fused corresponding to the matching perception frame are merged to obtain a merged perception frame, and the matching perception frame and the merged perception frame are weightedly fused; if the matching perception frame corresponds to one perception frame to be fused, the matching perception frame and the corresponding perception frame to be fused are weightedly fused.
[0128] As an optional implementation of the embodiment of the present application, the fusion module 63 is also used to, after obtaining the updated historical perception frame set, for each perception frame in the updated historical perception frame set, if the number of times the perception frame continuously serves as the matching perception frame corresponding to the perception frame to be fused exceeds a first preset number, mark the perception frame as a stable tracking perception frame; if the number of times the perception frame continuously fails to serve as the matching perception frame corresponding to the perception frame to be fused exceeds a second preset number, delete the perception frame from the historical perception frame set; and display all the stable tracking perception frames in the updated historical perception frame set.
[0129] As an optional implementation of the embodiment of the present application, the fusion module 63 is further configured to horizontally merge the stable tracking perception frames that meet the first merging condition before displaying all the stable tracking perception frames in the updated historical perception frame set;
[0130] Among them, the first merging condition includes: the direction angle is less than the threshold angle, the longitudinal distance and the lateral distance are both less than half the width of the larger stable tracking perception frame; the direction of the stable tracking perception frame is the length direction of the stable tracking perception frame.
[0131] As an optional implementation of the embodiment of the present application, the fusion module 63 is further configured to vertically merge the stable tracking perception frames that meet the second merging condition before displaying all the stable tracking perception frames in the updated historical perception frame set;
[0132] Among them, the second merging condition includes: the direction angle is less than the threshold angle, the longitudinal distance is less than one-third of the length of the larger stable tracking perception frame, and the lateral distance is less than twice the width of the larger stable tracking perception frame; the direction of the stable tracking perception frame is the length direction of the stable tracking perception frame.
[0133] As an optional implementation of the embodiment of the present application, the fusion module 63 is also used to, after obtaining the updated historical perception frame set, if the distance between two perception frames in the updated historical perception frame set is less than a distance threshold and the angle belongs to a preset angle range, delete one of the two perception frames from the updated historical perception frame set.
[0134] Regarding the above-mentioned driving environment perception device, the specific execution method of each module therein has been described in detail in the above-mentioned embodiment of the relevant method and will not be repeated here.
[0135] The present application also provides an electronic device. Figure 7 , Figure 7 701, a memory 702, and a computer program stored in the memory 702 and executable on the processor 701. When the computer program is executed by the processor 701, the driving environment perception method described in any of the above embodiments is implemented.
[0136] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the embodiments of the present application may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the embodiments of the present application may take the form of a computer program product implemented on one or more computer-usable storage media that include computer-usable program code.
[0137] The processor 701 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor, etc.
[0138] The memory 702 may include non-permanent memory in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.
[0139] Computer-readable storage media include permanent and non-permanent, removable and non-removable storage media. Storage media can be implemented by any method or technology to store information, and the information can be computer-readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer-readable media does not include temporary computer-readable media (transitory media), such as modulated data signals and carrier waves.
[0140] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the embodiments of the present application, rather than to limit them. Although the embodiments of the present application have been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present application.
Claims
1. A driving environment perception method, characterized in that: include: Obtain at least one candidate perception box of the target object from the current environment image; Matching each candidate perception frame with a perception frame in a set of historical perception frames, and obtaining at least one perception frame to be fused using a degree of overlap between the candidate perception frame and the perception frame in the set of historical perception frames; For each perception frame to be fused, taking the perception frame in the historical perception frame set that has the largest overlap with the perception frame to be fused as the matching perception frame corresponding to the perception frame to be fused; For each matching perception frame, the matching perception frame is fused with the corresponding perception frame to be fused to obtain the updated historical perception frame set.
2. The method according to claim 1, characterized in that The step of acquiring at least one candidate perception box of the target object from the current environment image includes: Identifying at least one observation perception frame from the current environment image; An observation perception frame whose ratio of length to width falls within a preset range is selected as at least one candidate perception frame of the target object.
3. The method according to claim 1, characterized in that: The step of matching each candidate perception frame with a perception frame in a set of historical perception frames, and obtaining at least one perception frame to be fused by using the overlap between the candidate perception frame and the perception frame in the set of historical perception frames, includes performing the following steps for each candidate perception frame: Obtaining the overlap between the candidate perception frame and each perception frame in the historical perception frame set; According to the overlap between the candidate perception frame and each perception frame in the set of historical perception frames, a maximum overlap between the candidate perception frame and each perception frame in the set of historical perception frames is obtained, where the maximum overlap between the candidate perception frame and each perception frame in the set of historical perception frames is the maximum value of the overlap between the candidate perception frame and each perception frame in the set of historical perception frames; When the maximum overlap of the candidate perception frames is greater than the threshold overlap, the candidate perception frame is determined as the perception frame to be fused.
4. The method according to claim 1, characterized in that For each matching perception frame, fusing the matching perception frame with the corresponding perception frame to be fused includes performing the following steps for each matching perception frame: If the matched perception frame includes multiple perception frames to be fused, the multiple perception frames to be fused corresponding to the matched perception frame are merged to obtain a merged perception frame, and weighted fusion is performed on the matched perception frame and the merged perception frame; If the matching perception frame corresponds to one perception frame to be fused, weighted fusion is performed on the matching perception frame and the corresponding perception frame to be fused.
5. The method according to claim 1, characterized in that: After obtaining the updated historical perception frame set, the method further includes: For each perception frame in the updated historical perception frame set, if the perception frame is used as a corresponding matching perception frame by the to-be-fused perception frames in the environment images for more than a first preset number of consecutive times, the perception frame is marked as a stable tracking perception frame; if the to-be-fused perception frames in the environment images for more than a second preset number of consecutive times do not use the perception frame as a corresponding matching perception frame, the perception frame is deleted from the historical perception frame set; All of the stable tracking perception frames in the updated historical perception frame set are displayed.
6. The method according to claim 5, characterized in that Before displaying all the stable tracking perception frames in the updated historical perception frame set, the method further includes: horizontally merging any two of the stable tracking perception frames that meet the first merging condition; Among them, the first merging condition includes: the direction angle is less than the threshold angle, the longitudinal distance and the lateral distance are both less than half the width of the larger stable tracking perception frame; the direction of the stable tracking perception frame is the length direction of the stable tracking perception frame.
7. The method according to claim 5, characterized in that Before displaying all the stable tracking perception frames in the updated historical perception frame set, the method further includes: Vertically merging any two of the stable tracking perception frames that meet the second merging condition; Among them, the second merging condition includes: the direction angle is less than the threshold angle, the longitudinal distance is less than one-third of the length of the larger stable tracking perception frame, and the lateral distance is less than twice the width of the larger stable tracking perception frame; the direction of the stable tracking perception frame is the length direction of the stable tracking perception frame.
8. The method according to any one of claims 1 to 7, characterized in that: After obtaining the updated historical perception frame set, the method further includes: If the distance between two perception frames in the updated historical perception frame set is less than the distance threshold and the angle between them belongs to the preset angle range, one of the two perception frames is deleted from the updated historical perception frame set.
9. A driving environment perception device, characterized in that: include: An acquisition module, used to acquire at least one candidate perception box of the target object from the current environment image; A matching module, used to match each candidate perception frame with a perception frame in a set of historical perception frames, and obtain at least one perception frame to be fused by using the overlap between the candidate perception frame and the perception frame in the set of historical perception frames; The fusion module is used to, for each perception frame to be fused, use the perception frame with the largest overlap with the perception frame to be fused in the historical perception frame set as the matching perception frame corresponding to the perception frame to be fused, and for each matching perception frame, fuse the matching perception frame with the corresponding perception frame to be fused to obtain the updated historical perception frame set.
10. An electronic device, characterized in that: include: A processor, a memory, and a computer program stored in the memory and executable on the processor, wherein the computer program, when executed by the processor, implements the driving environment perception method as claimed in any one of claims 1 to 8.