A potentially dangerous object detection method and apparatus

Abstract

The application discloses a potential dangerous target detection method and device, which are used for accurately and quickly detecting a potential dangerous target and guaranteeing automobile driving safety. The method comprises the following steps: acquiring at least two continuous images, the at least two continuous images being obtained by image acquisition on scenes on both sides of the automobile; determining an ROI region in a first image and determining motion information of each feature point included in the ROI region, the motion information of each feature point comprising a motion direction of the feature point; the first image being any one of the at least two images; clustering each feature point included in the ROI region according to the motion information of each feature point, so as to obtain at least one target located in the ROI region; determining a motion direction of each target according to the motion direction of at least one feature point included in each target; and determining a target with the same motion direction as the driving direction of the automobile in the at least one target as a potential dangerous target.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of communication, and particularly relates to a potential dangerous target detection method and device. BACKGROUND

[0002] With the increase of the number of automobiles, automobile safety is also paid more and more attention by people. An automatic emergency brake (AEB) system can ensure the automobile safety to a certain extent. The working principle of the AEB system is as follows: a camera is used to detect a target existing in front of the automobile, and when the distance between the detected target and the automobile does not exceed an alarm distance, it is considered that the target has a collision risk, and at this time, the automobile automatically adopts a braking measure.

[0003] In the prior art, the detection of the target can be realized based on machine learning. This detection method has a high detection rate for a complete target located in front of the automobile. However, for a target that can appear on both sides of the automobile and can quickly cut into the front of the automobile, the machine learning detection method can not detect the target or detect the target too late. This is because the target is only partially visible in the collected image and appears for a short time, and the machine learning detection method is difficult to detect the target when the target is only partially visible, resulting in late detection. Moreover, since the detection result of the target that is only partially visible is seriously dependent on sample data in machine learning, the target can not be detected. In the case of not detecting the target or detecting the target too late, the response time of the AEB function is compressed, which cannot guarantee the driving safety of the automobile. SUMMARY

[0004] Embodiments of the present application provide a potential dangerous target detection method and device, which can quickly and accurately detect a potential dangerous target to respond in advance and guarantee the driving safety of the automobile.

[0005] In a first aspect, the embodiments of the present application provide a potential dangerous target detection method, which can be executed by a detection device in a vehicle or a remote server, and the execution result is sent to the vehicle. The method comprises the following steps: acquiring continuous at least two frames of images, the continuous at least two frames of images being obtained by image acquisition of scenes on both sides of the vehicle; then determining a ROI region of interest in a first image, and determining motion information of each feature point included in the ROI region, the motion information of each feature point including a motion direction of the feature point; the first image is any one of the at least two frames of images; then clustering each feature point included in the ROI region according to the motion information of each feature point, to obtain at least one target located in the ROI region; determining a motion direction of each target according to the motion direction of at least one feature point included in each target; and finally determining a target having the same motion direction as the driving direction of the vehicle as a potential dangerous target.

[0006] In a possible scenario, an image acquisition device can be installed on the vehicle, and the scenes on both sides of the vehicle are image-acquired by the image acquisition device, so that the continuous at least two frames of images described above can be acquired. Of course, the scenes on both sides of the vehicle can also be image-acquired by a roadside device at the location of the vehicle, so that the continuous at least two frames of images described above can also be obtained, which is not limited in the present application.

[0007] The complete target located in front of the vehicle can be accurately detected, but the image acquisition device cannot capture a complete image of the target that may appear on both sides of the vehicle, so that when the detection is performed by the machine learning detection method, the target may not be detected or the detection time of the target may be too long. In the embodiments of the present application, the motion information of each feature point in the ROI region is clustered to determine the motion direction of the target in the ROI region, and by analyzing the motion directions of the target and the vehicle, the potential dangerous target can be determined. The detection of the potential dangerous target is obtained by analyzing the motion direction, and does not depend on the machine learning detection method, so that the potential dangerous target can be quickly and accurately detected, the response can be made in advance, and the driving safety of the vehicle is ensured.

[0008] In a possible design, when the ROI region in the first image is determined, the ROI region in the first image can be determined according to parameters of an image acquisition device used for image acquisition of the scenes on both sides of the vehicle and motion information of the vehicle. The parameters of the image acquisition device include intrinsic parameters and extrinsic parameters, the intrinsic parameters are used for conversion between a world coordinate system and an image acquisition device coordinate system, and the extrinsic parameters are used for conversion between the image acquisition device coordinate system and a two-dimensional coordinate system; and the motion information of the vehicle includes a driving direction and a driving speed of the vehicle.

[0009] In this design, the ROI region can be determined through the internal and external parameters of the image acquisition device and the motion information of the vehicle, i.e., the region where potential dangerous targets possibly exist on both sides of the vehicle is determined to ensure the safety of the vehicle.

[0010] In a possible design, when the ROI region in the first image is determined according to the parameters of the image acquisition device used for image acquisition of the scenes on both sides of the vehicle and the motion information of the vehicle, the longitudinal length of the potential dangerous region can be determined according to the driving direction and speed of the vehicle and the collision time corresponding to the safety distance of the vehicle; the lateral length of the potential dangerous region can be determined according to the width of the lane where the vehicle is located; and the ROI region in the first image can be determined according to the longitudinal length, the lateral length of the potential dangerous region and the parameters of the image acquisition device.

[0011] In a possible design, when the at least one target located in the ROI region is obtained by clustering the feature points included in the ROI region, the feature points included in the ROI region can be first clustered to obtain at least one first detection region, and the at least one first detection region is located in the ROI region; then the first confidence degree corresponding to each of the at least one first detection region is determined, and the first confidence degree corresponding to each of the at least one first detection region is determined when the feature points included in the ROI region are clustered; then the first detection region with a first confidence degree less than a first confidence degree threshold in the at least one first detection region is filtered out, and the remaining at least one first detection region is obtained as at least one second detection region, and the first confidence degree threshold is greater than 0 and less than 1; then the first confidence degree corresponding to each of the at least one second detection region is weighted according to the determined weight value to obtain the second confidence degree corresponding to each of the at least one second detection region; then the second detection region with a second confidence degree less than a second confidence degree threshold in the at least one second detection region is filtered out, and the remaining at least one second detection region is obtained, and the second threshold is greater than the first threshold and less than 1; finally, the target included in each of the remaining second detection regions is taken as the at least one target located in the ROI region.

[0012] In this design, the detection rate of potential dangerous targets can be improved through the preliminary screening by the lower first confidence degree threshold, then the first confidence degree is increased to the second confidence degree through weighting according to the weight value, and then the false detection rate is reduced through the re-screening by the higher second confidence degree threshold, so that the accuracy of detection and the positioning precision are improved.

[0013] In a possible design, the weight value can be determined according to at least one of the following parameters:

[0014] an intersection-over-union ratio of the first detection region and the second detection region, the intersection-over-union ratio being a ratio of a size of an intersection region of the first detection region and the second detection region to a size of a union region of the first detection region and the second detection region;

[0015] a number and a position of feature points included in the second detection region;

[0016] a motion amount of each feature point included in the second detection region;

[0017] a motion direction of each feature point included in the second detection region.

[0018] The adaptive adjustment of the weight value through at least one parameter further ensures the accuracy of potential dangerous target detection and the positioning precision.

[0019] In a second aspect, an embodiment of the present application provides a potential dangerous target detection device, which can have the functions in the above-mentioned first aspect or any possible design in the first aspect. The functions of the above-mentioned potential dangerous target detection device can be realized by hardware, or by hardware executing corresponding software, and the hardware or software includes one or more modules corresponding to the above-mentioned functions.

[0020] For example, the device can include an interface unit and a processing unit. The interface unit is configured to acquire at least two continuous images, and the at least two continuous images are obtained by image acquisition of scenes on both sides of a vehicle. The processing unit is configured to determine a ROI region of interest in a first image, and determine motion information of each feature point included in the ROI region, wherein the motion information of each feature point includes a motion direction of the feature point. The first image is any one of the at least two images. The processing unit is further configured to cluster each feature point included in the ROI region according to the motion information of each feature point, to obtain at least one target located in the ROI region. Finally, the processing unit is configured to determine a motion direction of each target according to a motion direction of at least one feature point included in each target, and determine a target with a same motion direction as a driving direction of the vehicle as a potential dangerous target.

[0021] In a possible design, the processing unit, specifically when determining the ROI region in the first image, can determine the ROI region in the first image according to parameters of an image acquisition device used for acquiring images of scenes on both sides of the vehicle and motion information of the vehicle. The parameters of the image acquisition device can include intrinsic parameters and extrinsic parameters, the intrinsic parameters being used for conversion between a world coordinate system and a coordinate system of the image acquisition device, and the extrinsic parameters being used for conversion between the coordinate system of the image acquisition device and a two-dimensional coordinate system. The motion information of the vehicle includes a driving direction and a driving speed of the vehicle.

[0022] In a possible design, the processing unit, specifically when determining the ROI region in the first image according to the parameters of the image acquisition device used for acquiring images of scenes on both sides of the vehicle and the motion information of the vehicle, can determine a longitudinal length of the potential dangerous region according to the driving direction and the driving speed of the vehicle and a collision time corresponding to a safety distance of the vehicle, determine a transverse length of the potential dangerous region according to a width of a lane where the vehicle is located, and finally determine the ROI region in the first image according to the longitudinal length and the transverse length of the potential dangerous region and the parameters of the image acquisition device.

[0023] In a possible design, the processing unit, specifically when clustering each feature point included in the ROI region to obtain at least one target located in the ROI region, can first cluster each feature point included in the ROI region to obtain at least one first detection region, where the at least one first detection region is located in the ROI region; then determine first confidence degrees respectively corresponding to the at least one first detection region, where the first confidence degrees respectively corresponding to the at least one first detection region are determined when clustering each feature point included in the ROI region; filter out a first detection region whose first confidence degree is less than a first confidence degree threshold from the at least one first detection region to obtain remaining at least one first detection region as at least one second detection region, where the first confidence degree threshold is greater than 0 and less than 1; further weight the first confidence degrees respectively corresponding to the at least one second detection region according to the determined weight value to obtain second confidence degrees respectively corresponding to the at least one second detection region; filter out a second detection region whose second confidence degree is less than a second confidence degree threshold from the at least one second detection region to obtain remaining at least one second detection region, where the second threshold is greater than the first threshold and less than 1; and finally take a target respectively included in each remaining second detection region as the at least one target located in the ROI region.

[0024] In a possible design, the weight value can be determined according to at least one of the following parameters:

[0025] an intersection-union ratio of the first detection region and the second detection region, the intersection-union ratio being a ratio of a size of an intersection region of the first detection region and the second detection region to a size of a union region of the first detection region and the second detection region;

[0026] a number and a position of feature points contained in the second detection region;

[0027] a motion amount of each feature point contained in the second detection region;

[0028] a motion direction of each feature point contained in the second detection region.

[0029] In a third aspect, an embodiment of the present application provides a potential dangerous target detection device, which can have the functions of the first aspect or any possible design of the first aspect. The functions of the potential dangerous target detection device can be implemented by hardware, or by hardware executing corresponding software, and the hardware or software includes one or more modules corresponding to the functions.

[0030] The device includes at least one processor and can also include at least one memory. The at least one processor is coupled to the at least one memory and can be used to execute computer program instructions stored in the memory to enable the device to perform the method of the first aspect or any possible design of the first aspect. Optionally, the device also includes a communication interface, and the processor is coupled to the communication interface. When the device is a server, the communication interface can be a transceiver or an input / output interface; when the device is a chip included in the server, the communication interface can be an input / output interface of the chip. Optionally, the transceiver can be a transceiver circuit, and the input / output interface can be an input / output circuit.

[0031] In a fourth aspect, an embodiment of the present application provides a chip system, including: a processor, the processor being coupled to a memory, the memory being used to store programs or instructions, when the programs or instructions are executed by the processor, enabling the chip system to perform the method of the first aspect or any possible design of the first aspect.

[0032] Optionally, the chip system also includes an interface circuit, which is used to receive code instructions and transmit to the processor.

[0033] Optionally, the processor in the chip system can be one or more, and the processor can be implemented by hardware or software. When implemented by hardware, the processor can be a logic circuit, an integrated circuit, etc. When implemented by software, the processor can be a general-purpose processor, which is implemented by reading software code stored in the memory.

[0034] Optionally, the memory in the chip system can also be one or more. The memory can be integrated with the processor, or can be arranged separately from the processor, and the application does not make any limitation. For example, the memory can be a non-transient processor, such as a read-only memory (ROM), which can be integrated on the same chip as the processor, or can be arranged on a different chip, and the application does not make any limitation on the type of memory and the arrangement of the memory and the processor.

[0035] In a fifth aspect, an embodiment of the present application provides a potential dangerous target detection device, comprising a processor and an interface circuit; the interface circuit is configured to receive code instructions and transmit the code instructions to the processor; the processor is configured to run the code instructions to execute the method in the first aspect or any possible design of the first aspect.

[0036] In a sixth aspect, an embodiment of the present application provides a computer readable storage medium, which stores a computer program or instructions, and when the computer program or instructions are executed, the computer executes the method in the first aspect or any possible design of the first aspect.

[0037] In a seventh aspect, an embodiment of the present application provides a computer program product, when a computer reads and executes the computer program product, the computer executes the method in the first aspect or any possible design of the first aspect.

[0038] The technical effects that can be achieved by any one of the second aspect to the seventh aspect and any possible design in any one of the aspects thereof can refer to the technical effects described above for the first aspect and the corresponding design in the first aspect, which will not be repeated here. BRIEF DESCRIPTION OF DRAWINGS

[0039] Figure 1 A potential dangerous target schematic diagram provided by an embodiment of the present application;

[0040] Figure 2a A target detection architecture schematic diagram;

[0041] Figure 2b A target detection flowchart;

[0042] Figure 3 、 Figure 7 、 Figure 8 A potential dangerous target detection flowchart provided by an embodiment of the present application;

[0043] Figure 4 A coordinate system conversion schematic diagram;

[0044] Figure 5An ROI diagram provided for an embodiment of the present application;

[0045] Figure 6 An object motion direction diagram provided for an embodiment of the present application;

[0046] Figure 9 、 Figure 10 An potential dangerous target detection device structure diagram provided for an embodiment of the present application. DETAILED DESCRIPTION

[0047] The present application provides a potential dangerous target detection method and device, aiming to more quickly and accurately realize detection of potential dangerous targets, so as to make a response in time and ensure vehicle safety. The method and device are based on the same technical concept, and since the principles of the method and device for solving problems are similar, the implementation of the device and the method can be mutually referred to, and the repeated parts will not be described again.

[0048] The following explains some terms of the embodiments of the present application, so as to facilitate understanding by those skilled in the art.

[0049] 1) Potential dangerous target, indicating a target appearing on both sides of the vehicle and possibly quickly switching to the front of the vehicle, which will cause a safety hazard and affect vehicle safety. As shown in Figure 1 , the potential dangerous target is indicated by the dashed box, and the driving directions of the vehicle and the potential dangerous target are indicated by the arrows. Since the potential dangerous target is located on both sides of the vehicle, the vehicle driver cannot timely perceive the existence of the potential dangerous target, nor can the vehicle driver timely respond (such as making braking or avoiding operations) to the potential dangerous target quickly cutting into the front of the vehicle at a close distance, which causes a safety hazard and affects vehicle safety. Therefore, it is very important to quickly and accurately detect the potential dangerous target and make the vehicle driver respond in time or in advance.

[0050] The target or the potential dangerous target can be a vehicle and / or a pedestrian, and the vehicle can be a motor vehicle (i.e., a vehicle) and / or a non-motor vehicle (such as Figure 1 , an electric bicycle).

[0051] The driving direction of the target can be the same as or different from the driving direction of the vehicle, for example Figure 1 , the driving direction of the target is the same as the driving direction of the vehicle. When the driving direction of the target is the same as the driving direction of the vehicle, and the potential dangerous target is located on both sides of the vehicle, it is more difficult for the vehicle driver to observe. When the target quickly cuts into the front of the vehicle at a close distance, the vehicle driver is also more difficult to respond in time, and thus the possibility of a safety hazard is higher. In the embodiments of the present application, this kind of target is mainly taken as a potential dangerous target for illustration.

[0052] The automobile referred to in the embodiments of the present application mainly refers to an automobile in a driving state.

[0053] For the sake of distinction, in the embodiments of the present application, the driving direction of the automobile is also referred to as the driving direction, the driving speed of the automobile is also referred to as the driving speed, the driving direction of the target is also referred to as the movement direction, and the driving speed of the target is also referred to as the movement speed.

[0054] 2) Coordinate systems involved in the image processing process: world coordinate system, camera coordinate system, image coordinate system and pixel coordinate system.

[0055] The world coordinate system is an absolute coordinate system, which is used to describe the real positions of the camera and the object in the three-dimensional space, and the unit is meter (m). The world coordinate system can also be regarded as a reference coordinate system. The world coordinate system can be represented as O w -X w Y w Z w , wherein O w is the origin of the world coordinate system, X w , Y w and Z w constitute three coordinate axes of the world coordinate system. The coordinates of a certain point in the world coordinate system can be represented as (X, Y, Z) or (X w , Y w , Z w ).

[0056] The camera coordinate system is a three-dimensional rectangular coordinate system with the focusing center of the camera (i.e. the optical center of the camera) as the origin and the optical axis of the camera as the Z axis, and the unit is m. The camera coordinate system can be represented as O c -X c Y c Z c , wherein O c is the origin of the camera coordinate system, X c , Y c and Z c constitute three coordinate axes of the camera coordinate system. The coordinates of a certain point in the camera coordinate system can be represented as (X c , Y c , Z c ).

[0057] The image coordinate system is a two-dimensional coordinate system with the optical center of the camera as the origin, and the optical center of the camera is the image center, and the unit is millimeter (mm). The image coordinate system can be represented as o-xy, where o is the origin of the image coordinate system, and x and y constitute two coordinate axes of the image coordinate system. The coordinates of a point in the image coordinate system can be represented as (x, y). f represents the focal length of the camera, which is the distance between o and O c

[0058] The pixel coordinate system is a two-dimensional coordinate system with the upper left corner of the image as the origin, and the unit is pixel. Two coordinate axes of the pixel coordinate system are composed of u and v. The coordinates of a point in the pixel coordinate system can be identified as (u, v).

[0059] The above coordinate systems can be converted to each other.

[0060] It can be understood that only the "camera" is taken as an example for description in the embodiments of the present application, and the type of the image acquisition device is not limited, and the "camera" can be replaced by other types of image acquisition devices, such as a camera, a driving recorder, etc.

[0061] 3) Parameters of the camera, including extrinsic parameters and intrinsic parameters, the extrinsic parameters are used to determine the relative position relationship between the world coordinate system and the camera coordinate system, and the intrinsic parameters are used to determine the projection relationship of the camera from three-dimensional space to two-dimensional image. That is, the parameters of the camera can realize the conversion between the above coordinate systems.

[0062] 4) Region of interest (ROI), which is an image region selected in the image, and the content of the ROI is more interesting or focused than the content of other regions in the image, and can be used as a focus in the image processing process. The content in the region of interest is the target of interest, which includes the potential dangerous target in the embodiments of the present application.

[0063] In the present application, "and / or" describes the association relationship of the associated objects, which means that there can be three kinds of relationships, for example, A and / or B, which can represent three cases: A exists alone, A and B exist together, and B exists alone. The character " / " generally represents an "or" relationship between the front and rear associated objects.

[0064] In the present application, multiple refers to two or more.

[0065] In addition, it should be understood that in the description of the present application, the words "first", "second", etc. are only used for distinguishing purposes of description, and cannot be understood as indicating or implying relative importance, nor can it be understood as indicating or implying order. ​

[0066] In order to facilitate understanding of the embodiments of the present application, the application scenarios used in the present application are first described.

[0067] With the increase of the number of automobiles, automobile safety is also paid more and more attention by people. At present, in order to ensure automobile safety, automobiles are all equipped with safety belts and airbags, in addition, AEB, lane keeping assistance (LKA) and other active safety systems also gradually become the standard configuration of automobiles. The working principle of the AEB system is: using a camera to detect the target existing in front of the automobile, when the distance between the detected target and the automobile does not exceed the alarm distance, it is considered that the target has collision risk, at this time the automobile will automatically take braking measures.

[0068] In the prior art, the detection of the target can be realized based on machine learning, and this detection method has a high detection rate for the complete target located in front of the automobile. However, for the target that may appear on both sides of the automobile and may quickly cut into the front of the automobile, the machine learning detection method may not detect the target or detect the target too late. This is because such a target is only partially visible in the collected image and appears for a short time, and the machine learning detection method is difficult to detect when the target is only partially visible, resulting in late detection, and because the detection result of such a partially visible target is seriously dependent on the sample data at the time of machine learning, it may also not detect the target. In the case of not detecting the target or detecting the target too late, the response time of the AEB function is compressed, resulting in the inability to ensure automobile safety.

[0069] In order to be able to respond to potential dangerous targets in advance, the following methods are also proposed in the prior art.

[0070] Method one, fuse the ranging ability of laser radar and the collection function of visual camera, determine the accurate position of the target, so as to realize early response and ensure automobile safety.

[0071] For example, as shown in FIG. 1, a laser radar is arranged on the front of the automobile, and a visual camera is arranged on the side of the automobile. Figure 2aAs shown, the car includes a vehicle speed sensor, a laser radar, a visual camera, a digital image processor, a data processor, a pre-warning indicator and a controller area network (CAN) bus. The laser radar is arranged on the roof of the car. The laser radar can detect targets located in front of and on both sides of the car, measure the distance between the targets and the car, and the movement speed of the targets. The visual camera collects images of the targets, and the digital image processor can process the images of the targets to obtain relevant information of the targets. The data processor can fuse the distance information measured by the laser radar and the relevant information of the targets obtained by the visual camera to determine the accurate position of the targets. The vehicle speed sensor can collect the driving speed of the car and send the driving speed to the data processor through the CAN bus. The data processor determines whether the distance between the targets and the car is not more than the alarm distance according to the accurate position of the targets, the movement speed of the targets and the driving speed of the car. If the distance between the targets and the car is not more than the alarm distance, the pre-warning indicator is used for pre-warning to achieve early response.

[0072] In this way, an additional laser radar needs to be added in the car. At present, the application cost of the laser radar is high and the laser radar is easy to be damaged, so the commercial value of this way is limited, and there are few car-grade laser radars, which cannot well adapt to the demand of car safety.

[0073] Way two, the detection function of the drivable area of the car and the detection function of the target are combined to determine whether there is a potential dangerous target, so as to achieve early response and ensure the safety of the car. As shown, Figure 2b The method comprises the following steps:

[0074] S201: According to the drivable area detection function, a set of drivable area points is obtained in the image collected by the front-view camera. The front-view camera can be installed in the car.

[0075] S202: The set of drivable area points is preprocessed.

[0076] S203: The pixel points in the preprocessed set of drivable area points are clustered to determine at least one target included in the drivable area.

[0077] S204: Coordinates of the at least one target in the drivable area are generated.

[0078] S205: The at least one target is tracked to determine whether there is a potential dangerous target of close-range cut-in. If yes, S206 is executed; if no, the S201 is returned to continue collecting images.

[0079] S206: calculating the acceleration and deceleration of the vehicle according to different situations.

[0080] S207: according to the movement speed of the potential dangerous target and the driving speed of the vehicle itself, performing long-distance control of the vehicle in advance to realize early response.

[0081] In this way, the detection process of the drivable area is also realized based on the deep learning method, and a large number of samples are also required, but the labeled samples required for drivable area detection are difficult to obtain, and the labeling accuracy is not high, so the drivable area cannot be quickly and accurately detected, and therefore this way cannot well adapt to the needs of vehicle safety. And the labeled samples required for drivable area detection are all manually labeled, which also increases the data dependence and application cost.

[0082] In summary, the several ways proposed in the prior art cannot quickly and accurately detect the potential dangerous target, so as to ensure the safety of vehicle driving.

[0083] Based on this, the embodiments of the present application provide a potential dangerous target detection method and device to quickly and accurately detect the potential dangerous target to respond in advance and ensure the safety of vehicle driving. As shown in the following steps, the following processes can be executed by a vehicle-mounted device, and of course, the road-side device can also be executed: Figure 3

[0084] S301: The detection module determines the ROI region according to the parameters of the camera and the motion information of the vehicle. The detection module can be located in the vehicle or in other devices outside the vehicle, such as road-side devices located on both sides of the road where the vehicle travels, and the detected information is sent to a remote server for processing.

[0085] The camera can be arranged on the vehicle, and the camera can capture (all or part of) images of the scenes on both sides of the vehicle, for example, the camera can be a driving recorder installed on the vehicle. The parameters of the camera generally refer to the internal and external parameters of the camera, that is, the internal and external parameters of the camera. The parameters of the camera are used to realize the conversion between the two-dimensional coordinate system (two-dimensional image / two-dimensional plane) where the camera is located and the world coordinate system (three-dimensional space). Of course, the camera can also not be located on the vehicle, such as the camera arranged in the road-side device at the location of the vehicle, and the road-side device captures images of the scenes on both sides of the vehicle.

[0086] ​The motion information of the vehicle includes, but is not limited to, the driving direction and the driving speed of the vehicle. Alternatively, the camera can capture a plurality of continuous images, and the detection module determines the driving direction and the driving speed of the vehicle according to the plurality of continuous images. Alternatively, the power system of the vehicle directly captures the driving direction and the driving speed of the vehicle and reports them to the detection module.

[0087] The ROI region is related to the motion information of the vehicle, wherein the driving direction and the driving speed can determine the size of the ROI region on both sides of the vehicle. The left and right sides of the vehicle refer to the left and right sides of the vehicle relative to the face of the driver when driving the vehicle.

[0088] In the S301, the detection module can determine the size of the potential dangerous region on both sides of the vehicle according to the driving direction and the driving speed of the vehicle, and then determine the size of the ROI region on both sides of the vehicle according to the size of the potential dangerous region and the parameters of the camera. The potential dangerous region is a region in the three-dimensional space, i.e., a region in the world coordinate system, and the ROI region is a region in the two-dimensional image, i.e., a region in the image coordinate system or the pixel coordinate. For example, the world coordinate system needs to be converted to the camera coordinate system, and then projected to the image coordinate system from the camera coordinate system. The image coordinate system can also be converted to the pixel coordinate system. Wherein Figure 4 The pinhole imaging principle of the camera is shown in the schematic diagram, which embodies the projection relationship between the camera coordinate system and the image coordinate system. The coordinates of point P in the camera coordinate system are (X, Y, Z), and the projection point of point P in the image coordinate system is point p, and the coordinates of point p are (x, y, f). The conversion formula from the world coordinate system to the pixel coordinate system is shown in Formula 1, and the camera intrinsic parameters are The camera extrinsic parameters are In the camera extrinsic parameters, R and T represent the rotation matrix and the translation vector when the world coordinate system is converted to the camera coordinate system, respectively. In the camera intrinsic parameters, u0 and v0 represent the coordinate values of the origin of the camera coordinate system in the pixel coordinate system, respectively. In the camera intrinsic parameters, f x and f y respectively represent the components of the camera focal length in the x-axis direction (horizontal direction) and the y-axis direction (vertical direction) of the camera coordinate system. dx and dy represent the length of one pixel in the x-axis direction and the y-axis direction, respectively, with the unit of mm.

[0089]

[0090] For example, the moving direction of the vehicle is forward, the moving speed of the vehicle is 20 km / h, and the safety distance of the vehicle corresponds to a time to collision (TTC) of 1 second. Assuming that 1 m / s is equal to 3.6 km / h, the detection module can determine that the longitudinal length of the potential danger area is one safety distance, i.e., the longitudinal length of the potential danger area = 20 km / h ÷ 3.6 km / h ≈ 5.5 m, and the transverse length of the potential danger area can be the transverse distance of one lane from the front bumper of the vehicle (for example, 3.75 m). According to the potential danger area, the detection module can convert the ROI area of the two-dimensional image, for example, the ROI area located on both sides of the vehicle as shown in the dashed box in FIG. 3, i.e., the area framed by the dashed box is the ROI area. Figure 5

[0091] The ROI area is a region that needs to be concerned about the possible potential danger target, that is, the ROI area can include a potential danger target. How the detection module detects the potential danger target in the ROI area will be described in detail in the following steps.

[0092] The detection module can determine the ROI area in each frame of the collected image. In this way, the ROI area can be dynamically adjusted according to the change of the motion information of the vehicle.

[0093] In this S301, the detection module calculates and generates the ROI area according to the internal and external parameters of the camera and the motion information of the vehicle, which can greatly reduce the calculation amount and reduce the noise interference brought by other irrelevant regions. The detection process of the ROI area does not depend on the method of deep learning, and does not need to label a large number of ROI samples, so it can quickly and accurately detect the ROI.

[0094] S302: The detection module performs feature point extraction in the ROI area.

[0095] The feature point extraction process can also be regarded as a corner point detection process. The corner point can be understood as a pixel point with specific characteristics.

[0096] ​For example, the HARRIS corner detection or the FAST corner detection can be used for the corner detection. The basic principle of the HARRIS corner detection is that a fixed window is used to slide in any direction in the image, and the pixel gray level change degree in the window before and after the sliding is compared. If there is a large gray level change in any direction, the corner exists in the window. The basic principle of the FAST corner detection is that if a pixel point and enough pixel points in the surrounding area are in different areas, the pixel point can be a corner. In the gray image, the gray value of the pixel point is greater than or less than the gray value of enough pixel points in the surrounding area.

[0097] The feature point extracted in the S302 can or can not belong to the potential dangerous target.

[0098] The detection module can extract the feature point in the ROI region of each frame of image.

[0099] Optionally, after extracting one or more feature points, the detection module can also determine the position of the feature point in the two-dimensional image. The position of the feature point in the two-dimensional image can be represented by the coordinates in the image coordinate system or the pixel coordinate system. In the embodiments of the present application, the coordinates in the image coordinate system are mainly taken as an example for description.

[0100] S303: The detection module determines the inter-frame motion information of the feature point in the ROI region.

[0101] In the S303, the detection module can determine the inter-frame motion information of the feature point in the ROI region in the image according to the continuous (at least) two frames of images. For example, the detection module determines that the ROI region in the current frame includes a first feature point, and the ROI region in the previous frame image of the current frame also includes the first feature point. It can be understood that the first feature point in the previous frame image is a feature point corresponding to the first feature point in the current frame image. The first feature point corresponding to the feature point refers to a feature point corresponding to the position of the first feature point, and / or a feature point corresponding to the feature possessed by the first feature point. The position of the first feature point in the previous frame image is (x1, y1), and the position of the first feature point in the current frame image is (x2, y2). The detection module determines the inter-frame motion of the first feature point as (x1, y1) to (x2, y2). The detection module can determine the direction and speed of the inter-frame movement of the first feature point according to (x1, y1) and (x2, y2). For example, the detection module can determine the direction of the inter-frame movement of the first feature point according to the direction of (x1, y1) to (x2, y2). The detection module can determine the speed of the inter-frame movement of the first feature point according to the distance between (x1, y1) and (x2, y2) and the time interval between the two frames of images. The detection module can determine the size of the inter-frame movement of the first feature point, i.e., the motion amount, according to the distance between (x1, y1) and (x2, y2).

[0102] The detection module can track the position of the first feature point in the previous frame image based on the invariance of illumination between adjacent two frames according to the pyramid optical flow algorithm (Lukas-Kanade optical flow algorithm), to obtain the position of the first feature point in the current frame image.

[0103] S304: The detection module obtains the motion information of the vehicle.

[0104] The motion information of the vehicle includes the driving direction and driving speed of the vehicle, and can also include the gear position and steering wheel angle of the vehicle.

[0105] In the S304, the detection module can obtain the motion information of the vehicle through the CAN bus.

[0106] S305: The clustering module clusters the feature points into a bonding box (BBOX), and feeds back the BBOX to the detection module. The detection module can filter the BBOX with a first confidence value less than a first confidence threshold value according to the first confidence value of the BBOX.

[0107] In the S305, the clustering module can cluster the feature points in the ROI to obtain at least one BBOX, each of which includes one target. That is, at least one target is clustered in the ROI region. Generally, due to the complexity of the actual scene, multiple BBOXs and multiple targets can be clustered in the ROI region. Each BBOX is a detection frame, and the region framed by the detection frame is a detection region. Therefore, in the embodiments of the present application, the detection frame and the detection region can be understood equivalently, and the concepts of the detection frame and the detection region can be used interchangeably.

[0108] In the S305, the clustering module can cluster the feature points that meet the conditions in a minimum circumscribed rectangle, and the BBOX is the minimum circumscribed rectangle. Since the feature points are not uniformly distributed, the clustered BBOXs may not be very accurate, but they have guiding significance for determining potential dangerous targets in subsequent steps. Since the clustered BBOXs may not be accurate, the BBOXs can also be regarded as rough BBOXs obtained by clustering.

[0109] The multiple feature points clustered into one BBOX can have one or more of the following characteristics: the motion speed of the feature points is greater than or equal to the driving speed of the car, the motion direction of the feature points is the same as the driving direction of the car, the motion direction of the feature points is opposite to the driving direction of the car, or the number of feature points exceeds a first number. The first number is a preset minimum number of feature points included in a BBOX, which can limit the size of the BBOX region. For example, the feature points that meet the conditions are feature points whose motion state meets the motion state of a potential dangerous target, that is, the motion direction of the feature points that meet the conditions is the same as the driving direction of the car, and the motion speed is greater than or equal to the driving speed of the car. Generally, the multiple feature points clustered into one BBOX belong to the same target, and the BBOX may include a potential dangerous target. The BBOX of the potential dangerous target can be as shown by the dashed box in FIG. 3. Figure 1

[0110] For targets located on both sides of the car and quickly cutting into the front of the car at a close distance, the car driver is more difficult to be observed and more difficult to respond in time, so such targets can be regarded as potential dangerous targets, and the motion direction of the potential dangerous target is the same as the driving direction of the car. Targets different from the driving direction of the car are easy to be observed by the car driver, so such targets can be regarded as non-potential dangerous targets, that is, the motion direction of the non-potential dangerous target is different from the driving direction of the car. Optionally, Figure 6 ​As shown, the potential dangerous target and the non-potential dangerous target can be marked in the image in different marking manners to be more obviously distinguished, for example, the potential dangerous target and the non-potential dangerous target can be marked by using different color marking boxes, and / or the arrow corresponding to the motion direction of the potential dangerous target is different in color from the arrow corresponding to the motion direction of the non-potential dangerous target.

[0111] The BBOX includes position information of the BBOX, category information of a target in the BBOX, and a first confidence degree belonging to the category, the confidence degree of the target is also used to represent the confidence degree of the BBOX, the first confidence degree can be regarded as a confidence degree of the target in the BBOX belonging to the category, and the confidence degree is generally represented by probability, and the value range of the first confidence degree is [0, 1].

[0112] In the embodiment of the application, at least two confidence degree thresholds are set, for example, two confidence degree thresholds including a first confidence degree threshold and a second confidence degree threshold, the first confidence degree threshold is less than the second confidence degree threshold, and the first confidence degree threshold is not less than 0, and the second confidence degree threshold is not greater than 1. In the S305, by setting a lower first confidence degree threshold (hereinafter referred to as a low threshold), more BBOXs can be released, and the detection rate of the potential dangerous target can be improved.

[0113] In the S305, the detection module can detect the BBOX in the ROI region in a sliding window manner according to machine learning and deep learning technology, and at this time, the clustering module can be trained by using machine learning and deep learning technology. Or the detection module can input the image (in which the ROI region is marked) into a neural network to obtain the BBOX, and at this time, the clustering module can be a clustering model based on a neural network architecture.

[0114] In order to facilitate distinction, the BBOX obtained by the clustering module is represented by Bp, and the BBOX obtained by the first confidence degree threshold screening is represented by Bi.

[0115] S306: The detection module increases the confidence degree of the BBOX to a second confidence degree, filters the BBOX with the second confidence degree less than the second confidence degree threshold, and outputs a final detection result.

[0116] The detection module can improve the confidence degree of the BBOX by weighting, further screen by using a higher second confidence degree threshold (hereinafter referred to as a high threshold), so as to reduce the false detection rate and improve the detection accuracy.

[0117] For example, the detection module increases the BBOX from the first confidence {Bi} to the second confidence {Wi*Bi}, and the second confidence is in the range of [0, 1]. Wherein Wi is an adaptive weight, which can be adjusted according to at least one of the following parameters:

[0118] Parameter one: intersection over union (IOU) of Bi and Bp. Bi can be a BBOX obtained by screening the first confidence threshold, and Bp can be a BBOX obtained by clustering the feature points. The intersection over union of Bi and Bp is the ratio of the size of the intersection region of Bi and Bp to the size of the union region of Bi and Bp. The higher the IOU value, the higher the classification confidence of Bi, the higher the positioning accuracy of Bi, and the larger the corresponding Wi value.

[0119] Parameter two: the number and position of feature points contained in Bi. The position of the feature points can be used to represent the positioning accuracy of Bi. The more the number of feature points, the higher the classification confidence of Bi, the higher the positioning accuracy of Bi, and the larger the corresponding Wi value.

[0120] Parameter three: the motion direction of the feature points contained in Bi. Different motion directions of the feature points have different contributions to the weighting coefficient Wi. The feature points that are more consistent with the motion direction of the target (i.e., the potential dangerous target) have a greater contribution to Wi, for example, the feature points that have the same motion direction as the driving direction of the car have a greater contribution to Wi, and the corresponding Wi value is larger.

[0121] Parameter four: the motion amount of the feature points contained in Bi. Different motion amounts of the feature points have different contributions to Wi. The greater the motion amount of the feature points, the greater the contribution to Wi, and the smaller the motion amount of the feature points, the smaller the contribution to Wi, which can ensure that only the targets that cut in quickly are detected.

[0122] In the S306, by increasing the confidence of the candidate detection region and setting a high threshold, the false detection rate can be reduced, and the detection accuracy and positioning precision can be improved.

[0123] In the S306, the detection module can further screen the weighted classification confidence and positioning accuracy, and then perform non-maximum suppression (NMS) to obtain the final detection result. For example, the NMS can further screen the BBOX according to the weighted classification confidence and positioning accuracy, and output the final detection result. The score threshold can be any value, which is not limited in the embodiments of the present application.

[0124] wherein S305 and S306 can be regarded as using double thresholds for detection within the ROI, the double thresholds including a low threshold Tlow and a high threshold Thigh, wherein 0 < Tlow < Thigh < 1. It can be understood that the method of double threshold detection can also be directly applied to the collected image. The process of double threshold detection can refer to Figure 7 , including the following steps:

[0125] S701: The detection module inputs the collected image into the neural network, and obtains an output first detection region based on the neural network.

[0126] The neural network can realize the function of the clustering module in S305 described above, and the first detection region is the BBOX obtained by clustering in S305 described above.

[0127] S702: The detection module determines whether the first detection region is located within the ROI region. For the first detection region located within the ROI, S703 is performed, and for the first detection region not located within the ROI, S705 is performed.

[0128] S703: The detection module uses a low threshold Tlow to filter the first detection region whose first confidence does not exceed the Tlow, to obtain a second detection region. The first confidence of the second detection region is equal to or greater than the Tlow.

[0129] S704: The detection module calculates the weight Wi of the second detection region, and uses the weight Wi to weight the first confidence, to obtain the second confidence {Wi*Bi} of the second detection region.

[0130] S705: The detection module uses a high threshold Thigh to further filter the second detection region whose second confidence does not exceed the Thigh, to obtain a third detection region. The second confidence of the third detection region is equal to or greater than the Thigh.

[0131] S706: The detection module further filters the third detection region through an NMS algorithm, to output a final detection result.

[0132] Exemplarily, the input of the NMS algorithm is the positions of the plurality of third detection regions and the second confidence of each third detection region. The screening condition of the NMS algorithm includes: for the third detection regions with overlap, deleting the third detection region with the second confidence exceeding a third confidence threshold from the overlapping third detection regions, and deleting the third detection region not exceeding the third confidence threshold; for the third detection regions without overlap, retaining the third detection region. The NMS outputs one or more third detection regions satisfying the deletion screening condition as the final detection region, and the final detection region includes the potential dangerous target.

[0133] The prior art has a high detection rate for complete targets, but for fast-cut-in targets, it is necessary to detect them in advance during the process of entering the field of vision (FOV) of the camera from outside the FOV, and the faster the detection is, the more serious the target occlusion is, and the higher the detection capability is required. The mainstream machine learning and deep learning detection method relies heavily on sample data when detecting potential dangerous targets, and consumes more computing power to ensure the accuracy of the detection model for severely occluded targets. More computing power requires higher system resources and reduces the real-time detection of the system. The potential dangerous target method provided in the embodiments of the present application reduces the dependence of occluded targets on samples, can improve the fast and accurate detection of cut-in targets with less computing power, and only uses an image acquisition device in the embodiments of the present application, without introducing other sensors, thereby reducing the system cost.

[0134] In combination with the above embodiments and the accompanying drawings, as Figure 8 shown, the embodiments of the present application provide a potential dangerous target detection method. The method includes the following steps:

[0135] S801: A first device acquires continuous at least two images, and the continuous at least two images are obtained by image acquisition of scenes on both sides of a vehicle.

[0136] The first device can be the vehicle or located in the vehicle, or the first device is another device outside the vehicle, such as a remote server. The server performs potential dangerous target detection, and then sends the detection result to the vehicle.

[0137] In one possible scenario, an image acquisition device is installed on the vehicle, which can acquire images of scenes on both sides of the vehicle, such as acquiring continuous at least two images. In another possible scenario, a road side unit at the location of the vehicle can also acquire images of scenes on both sides of the vehicle, that is, an image acquisition device is installed in the road side unit.

[0138] The image collected by the image collection device can include a complete target image or a target image that is only partially visible.

[0139] S802: The first device determines an ROI region in a first image and determines motion information of each feature point included in the ROI region, the motion information of each feature point including a motion direction of the feature point, the first image being any one of the at least two images.

[0140] In the S802, the first device can also determine the ROI region in the first image according to parameters of the image collection device and motion information of the vehicle.

[0141] The parameters of the image collection device include intrinsic parameters and extrinsic parameters, the intrinsic parameters being used for conversion between a world coordinate system and an image collection device coordinate system, and the extrinsic parameters being used for conversion between the image collection device coordinate system and a two-dimensional coordinate system; and the motion information of the vehicle includes a driving direction and a driving speed of the vehicle.

[0142] For example, the first device can determine a longitudinal length of a potential danger region according to the driving direction and the driving speed of the vehicle and a collision time corresponding to a safety distance of the vehicle, determine a transverse length of the potential danger region according to a width of a lane where the vehicle is located, and determine the ROI region in the first image according to the longitudinal length, the transverse length of the potential danger region, and the parameters of the image collection device.

[0143] S803: The first device clusters each feature point included in the ROI region according to the motion information of the feature point, to obtain at least one target located in the ROI region.

[0144] The first device can cluster a plurality of feature points whose feature information meets a condition.

[0145] S804: The first device determines a motion direction of each target according to a motion direction of at least one feature point included in the target.

[0146] The motion directions of the feature points in the first detection region obtained by clustering are the same, and therefore the motion direction of at least one feature point can be used to determine the motion direction of a target in the first detection region.

[0147] S805: The first device determines a target whose motion direction is the same as the driving direction of the vehicle as a potential danger target.

[0148] In the S805, each feature point included in the ROI region is clustered to obtain at least one first detection region, the at least one first detection region is located in the ROI region; a first confidence corresponding to each of the at least one first detection region is determined; the first confidence corresponding to each of the at least one first detection region is determined when the clustering is performed on each feature point included in the ROI region; a first detection region with a first confidence less than a first confidence threshold in the at least one first detection region is filtered out to obtain at least one remaining first detection region as at least one second detection region, the first confidence threshold is greater than 0 and less than 1; the first confidence corresponding to each of the at least one second detection region is weighted according to the determined weight value to obtain a second confidence corresponding to each of the at least one second detection region; a second detection region with a second confidence less than a second confidence threshold in the at least one second detection region is filtered out to obtain at least one remaining second detection region, the second threshold is greater than the first threshold and less than 1; a target included in each of the remaining second detection regions is taken as at least one target located in the ROI region.

[0149] The weight value is determined according to at least one of the following parameters:

[0150] An intersection union ratio of the first detection region and the second detection region, the intersection union ratio being a ratio of a size of an intersection region of the first detection region and the second detection region to a size of a union region of the first detection region and the second detection region;

[0151] A number and a position of a feature point included in the second detection region;

[0152] A motion amount of each feature point included in the second detection region;

[0153] A motion direction of each feature point included in the second detection region.

[0154] The specific implementation manners shown in the embodiments of the present application can refer to the introduction of the related embodiments described above. Figure 8

[0155] Each embodiment in the embodiments of the present application can be used in combination with each other, or can be used alone.

[0156] Figure 8 ​The schemes provided in the application are mainly introduced from the perspective of method flow. It can be understood that, in order to realize the above functions, the device can include hardware structure and / or software module corresponding to the execution of each function. Those skilled in the art should easily realize that, in combination with the units and algorithm steps of each example described in the embodiments disclosed herein, the application can be realized in the form of hardware or a combination of hardware and computer software. Whether a certain function is realized in the form of hardware or computer software driving hardware depends on the specific application and design constraints of the technical solution. The skilled person can use different methods to realize the described functions for each specific application, but such implementation should not be considered beyond the scope of the application.

[0157] In the case of using an integrated unit, Figure 9 A possible exemplary block diagram of a potential dangerous target detection device involved in the embodiments of the application is shown, which can exist in the form of software. The potential dangerous target device 900 can include an interface unit 901 and a processing unit 902.

[0158] The potential dangerous target device 900 can be the above-mentioned automobile or also a semiconductor chip arranged in the automobile or other equipment. Specifically, in one embodiment, the interface unit 901 is configured to acquire continuous at least two frames of images, the continuous at least two frames of images being obtained by image acquisition of scenes on both sides of the automobile.

[0159] The processing unit 902 is configured to determine a ROI region of interest in a first image, and determine motion information of each feature point included in the ROI region, the motion information of each feature point including a motion direction of the feature point; the first image is any one of the at least two frames of images; and

[0160] According to the motion information of each feature point, each feature point included in the ROI region is clustered to obtain at least one target located in the ROI region.

[0161] According to the motion direction of at least one feature point included in each target, a motion direction of each target is determined; a target with the same motion direction as the driving direction of the automobile among the at least one target is determined as a potential dangerous target.

[0162] In a possible design, when the processing unit 902 determines the ROI region in the first image, it is specifically configured to: according to parameters of an image acquisition device used for image acquisition of scenes on both sides of the automobile and motion information of the automobile, determine the ROI region in the first image.

[0163] The parameters of the image acquisition device include an intrinsic parameter and an extrinsic parameter, the intrinsic parameter is used for conversion between a world coordinate system and an image acquisition device coordinate system, and the extrinsic parameter is used for conversion between the image acquisition device coordinate system and a two-dimensional coordinate system; and the motion information of the vehicle includes a driving direction and a driving speed of the vehicle.

[0164] In a possible design, when the processing unit 902 determines the ROI region in the first image according to the parameters of the image acquisition device used for image acquisition of the scenes on both sides of the vehicle and the motion information of the vehicle, specifically, the processing unit 902 determines a longitudinal length of a potential danger region according to the driving direction and the driving speed of the vehicle and a collision time corresponding to a safety distance of the vehicle; determines a transverse length of the potential danger region according to a width of a lane where the vehicle is located; and determines the ROI region in the first image according to the longitudinal length, the transverse length of the potential danger region and the parameters of the image acquisition device.

[0165] In a possible design, when the processing unit 902 clusters each feature point included in the ROI region to obtain at least one target located in the ROI region, specifically, the processing unit 902 clusters each feature point included in the ROI region to obtain at least one first detection region, the at least one first detection region is located in the ROI region; determines first confidence degrees respectively corresponding to the at least one first detection region, the first confidence degrees respectively corresponding to the at least one first detection region are determined when each feature point included in the ROI region is clustered; filters out a first detection region with a first confidence degree less than a first confidence threshold from the at least one first detection region, to obtain remaining at least one first detection region as at least one second detection region, the first confidence threshold is greater than 0 and less than 1; weights the first confidence degrees respectively corresponding to the at least one second detection region according to the determined weight value, to obtain second confidence degrees respectively corresponding to the at least one second detection region; filters out a second detection region with a second confidence degree less than a second confidence threshold from the at least one second detection region, to obtain remaining at least one second detection region, the second threshold is greater than the first threshold and less than 1; and takes a target respectively included in each remaining second detection region as the at least one target located in the ROI region.

[0166] In a possible design, the weight value is determined according to at least one of the following parameters:

[0167] An intersection-over-union of the first detection region and the second detection region, the intersection-over-union is a ratio of a size of an intersection region of the first detection region and the second detection region to a size of a union region of the first detection region and the second region.

[0168] The number and location of feature points contained within the second detection area;

[0169] The amount of motion of each feature point contained within the second detection area;

[0170] The direction of motion of each feature point contained within the second detection area.

[0171] The division of units in this embodiment is illustrative and represents only one logical functional division. In actual implementation, other division methods may be used. The functional units in this embodiment can be integrated into one processing module, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated units can be implemented in hardware or as software functional units.

[0172] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) or processor to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0173] like Figure 10 As shown in the illustration, this application also provides a schematic diagram of another possible potential hazardous target detection device, which includes at least one processor 1002 and at least one communication interface 1004. Further, the potential hazardous target detection device may also include a memory 1006 for storing computer programs or instructions. The memory 1006 can be either in-processor memory or external memory. Figure 10 When the unit modules described herein are implemented in software, the software or program code required for the processor 1002 to perform corresponding actions is stored in the memory 1006. The processor 1002 is used to execute the programs or instructions in the memory 1006 to implement the above embodiments. Figure 8The steps shown. The communication interface 1004 is configured to enable communication between the apparatus and other apparatuses.

[0174] In the case where the memory 1006 is disposed outside the processor, the memory 1006, the processor 1002 and the communication interface 1004 are connected with each other through a bus 1008, which can be a peripheral component interconnect (PCI) bus or an extended industry standard architecture (EISA) bus, etc. It should be understood that the bus can be divided into an address bus, a data bus, a control bus, etc. For the sake of representation, Figure 10 In the figures, only one thick line is used to represent the bus, but it does not mean that there is only one bus or only one type of bus.

[0175] It should be noted that the operations and / or functions of each module in the apparatus 1000 are respectively implemented to realize the corresponding flow of the method shown in Figure 8 In the figures, only one thick line is used to represent the bus, but it does not mean that there is only one bus or only one type of bus.

[0176] The embodiments of the present application further provide a chip system, comprising: a processor, the processor being coupled with a memory, the memory being configured to store programs or instructions, when the programs or instructions are executed by the processor, the chip system implements the method in any of the above method embodiments.

[0177] Optionally, the processor in the chip system can be one or more. The processor can be implemented by hardware or software. When implemented by hardware, the processor can be a logic circuit, an integrated circuit, etc. When implemented by software, the processor can be a general-purpose processor, which realizes by reading software codes stored in the memory.

[0178] Optionally, the memory in the chip system can also be one or more. The memory can be integrated with the processor or disposed separately from the processor, which is not limited in the present application. For example, the memory can be a non-transient processor, such as a read-only memory (ROM), which can be integrated on the same chip as the processor or disposed on different chips respectively, and the type of the memory and the disposition of the memory and the processor are not limited in the present application.

[0179] For example, the chip system can be a field programmable gate array (FPGA), an application specific integrated circuit (ASIC), a system on chip (SoC), a central processing unit (CPU), a network processor (NP), a digital signal processor (DSP), a micro controller unit (MCU), a programmable logic device (PLD) or other integrated chip.

[0180] It should be understood that each step in the above method embodiments can be completed by integrated logic circuits of hardware in the processor or instructions in the form of software. The method steps disclosed in combination with the embodiments of the present application can be directly embodied as hardware processor execution, or executed by a combination of hardware and software modules in the processor.

[0181] The embodiments of the present application further provide a computer readable storage medium, wherein the computer readable storage medium stores computer readable instructions, and when the computer reads and executes the computer readable instructions, the computer executes the method in any of the above method embodiments.

[0182] The embodiments of the present application further provide a computer program product, and when the computer reads and executes the computer program product, the computer executes the method in any of the above method embodiments.

[0183] It should be understood that the processor mentioned in the embodiments of the present application can be a central processing unit (CPU), and can also be other general-purpose processors, digital signal processors (DSP), application specific integrated circuits (ASIC), ready programmable gate arrays (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor.

[0184] It should also be understood that the memory mentioned in the embodiments of this application can be volatile memory or non-volatile memory, or may include both volatile and non-volatile memory. The non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. The volatile memory can be random access memory (RAM), which is used as an external cache. By way of example, but not limitation, many forms of RAM are available, such as static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous linked dynamic random access memory (SLDRAM), and direct rambus RAM (DR RAM).

[0185] It should be noted that when the processor is a general-purpose processor, DSP, ASIC, FPGA, or other programmable logic device, discrete gate or transistor logic device, or discrete hardware component, the memory (storage module) is integrated into the processor.

[0186] It should be noted that the memories described herein are intended to include, but are not limited to, these and any other suitable types of memories.

[0187] It should be understood that, in the various embodiments of this application, the sequence number of each process does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this invention.

[0188] Those skilled in the art can clearly understand that the units and algorithm steps of each example described in combination with the embodiments disclosed herein can be realized by electronic hardware or a combination of computer software and electronic hardware. Whether the functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the present application.

[0189] Those skilled in the art can clearly understand that, for the convenience and brevity of the description, the specific working processes of the above-described system, device and unit can refer to the corresponding processes in the foregoing method embodiments, which will not be repeated here.

[0190] In several embodiments provided in the present application, it should be understood that the disclosed system, device and method can be implemented in other ways. For example, the above-described device embodiments are merely schematic, for example, the division of the units is only a logical function division, and actual implementation can have another division manner, for example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between the units shown or discussed can be indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.

[0191] The units described as separate components can or can not be physically separated, and the components shown as units can or can not be physical units, that is, they can be located in one place, or can be distributed on multiple network units. Part or all of the units can be selected according to actual needs to achieve the purpose of the embodiment.

[0192] In addition, each functional unit in each embodiment of the present application can be integrated into a processing unit, or each unit can exist physically independently, or two or more units can be integrated into one unit.

[0193] If the functions are implemented in the form of software function units and sold or used as independent products, they can be stored in a computer readable storage medium. Based on this understanding, the technical solutions of the present application essentially or the parts that contribute to the prior art or parts of the technical solutions can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes a plurality of instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present application. The aforementioned storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, and various media that can store program codes.

[0194] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art can easily think of changes or replacements within the technical scope disclosed in the present application, which should be covered within the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

Claims

1. A method of detecting a potentially dangerous object, characterized by, The method comprises the following steps: acquiring at least two continuous images, which are obtained by image acquisition of scenes on both sides of the vehicle; determining a ROI region in a first image, and determining motion information of each feature point included in the ROI region, wherein the motion information of each feature point includes a motion direction of the feature point, and the first image is any one of the at least two images; clustering each feature point included in the ROI region according to the motion information of each feature point to obtain at least one target located in the ROI region; determining a motion direction of each target according to the motion direction of at least one feature point included in each target; determining a potential dangerous target from the at least one target whose motion direction is the same as the driving direction of the vehicle; wherein the clustering of each feature point included in the ROI region to obtain at least one target located in the ROI region comprises: clustering each feature point included in the ROI region to obtain at least one first detection region, wherein the at least one first detection region is located in the ROI region; determining a first confidence degree corresponding to each of the at least one first detection region, wherein the first confidence degree corresponding to each of the at least one first detection region is determined when clustering each feature point included in the ROI region; filtering out a first detection region with a first confidence degree less than a first confidence degree threshold from the at least one first detection region to obtain at least one remaining first detection region as at least one second detection region, wherein the first confidence degree threshold is greater than 0 and less than 1; weighting the first confidence degree corresponding to each of the at least one second detection region according to the determined weight value to obtain a second confidence degree corresponding to each of the at least one second detection region; filtering out a second detection region with a second confidence degree less than a second confidence degree threshold from the at least one second detection region to obtain at least one remaining second detection region, wherein the second confidence degree threshold is greater than the first confidence degree threshold and less than 1; regarding a target included in each of the remaining second detection regions as at least one target located in the ROI region.

2. The method of claim 1, wherein, The method comprises the following steps: determining a ROI region in a first image according to parameters of an image acquisition device used for image acquisition of scenes on both sides of the vehicle and motion information of the vehicle; the parameters of the image acquisition device include intrinsic parameters and extrinsic parameters, wherein the intrinsic parameters are used for conversion between a world coordinate system and an image acquisition device coordinate system, and the extrinsic parameters are used for conversion between the image acquisition device coordinate system and a two-dimensional coordinate system; and the motion information of the vehicle includes a driving direction and a driving speed of the vehicle.

3. The method of claim 2, wherein, The method comprises the following steps: Determine a longitudinal length of the potential danger area according to the driving direction and the driving speed of the vehicle and a collision time corresponding to a safety distance of the vehicle; Determine a lateral length of the potential danger area according to a width of a lane where the vehicle is located; Determine a ROI area in the first image according to the longitudinal length, the lateral length of the potential danger area and parameters of the image acquisition device.

4. The method of claim 1, wherein, The weight value is determined according to at least one of the following parameters: An intersection-over-union of the first detection area and the second detection area, the intersection-over-union being a ratio of a size of an intersection area of the first detection area and the second detection area to a size of a union area of the first detection area and the second detection area; A number and a position of feature points contained in the second detection area; A motion amount of each feature point contained in the second detection area; A motion direction of each feature point contained in the second detection area.

5. A potentially dangerous object detection apparatus characterized by comprising: The interface unit is configured to acquire at least two continuous images, the at least two continuous images being obtained by image acquisition of scenes on both sides of the vehicle; The processing unit is configured to determine a ROI area of interest in a first image and determine motion information of each feature point contained in the ROI area, the motion information of each feature point including a motion direction of the feature point, the first image being any one of the at least two images; According to the motion information of each feature point, cluster each feature point contained in the ROI area to obtain at least one target located in the ROI area; Determine a motion direction of each target according to a motion direction of at least one feature point contained in each target, and determine a target having the same motion direction as the driving direction of the vehicle as a potential danger target. The processing unit is configured to: Cluster each feature point contained in the ROI area to obtain at least one first detection area, the at least one first detection area being located in the ROI area; Determine a first confidence degree corresponding to each of the at least one first detection area, the first confidence degree corresponding to each of the at least one first detection area being determined when clustering each feature point contained in the ROI area; Filter out a first detection area having a first confidence degree less than a first confidence degree threshold from the at least one first detection area to obtain at least one remaining first detection area as at least one second detection area, the first confidence degree threshold being greater than 0 and less than 1; According to the determined weight value, weight the first confidence degree corresponding to each of the at least one second detection area to obtain a second confidence degree corresponding to each of the at least one second detection area. ​ ​ filtering out a second detection region with a second confidence less than a second confidence threshold in the at least one second detection region, to obtain a remaining at least one second detection region, the second confidence threshold being greater than the first confidence threshold and less than 1; regarding a target included in each of the remaining second detection regions as at least one target located in the ROI region.

6. The apparatus of claim 5, wherein, When determining the ROI region in the first image, the processing unit is specifically configured to: determine the ROI region in the first image according to parameters of an image acquisition device used for image acquisition of scenes on both sides of the vehicle and motion information of the vehicle; the parameters of the image acquisition device include intrinsic parameters and extrinsic parameters, the intrinsic parameters are used for conversion between a world coordinate system and an image acquisition device coordinate system, and the extrinsic parameters are used for conversion between the image acquisition device coordinate system and a two-dimensional coordinate system; and the motion information of the vehicle includes a driving direction and a driving speed of the vehicle.

7. The apparatus of claim 6, wherein, When determining the ROI region in the first image according to the parameters of the image acquisition device used for image acquisition of scenes on both sides of the vehicle and the motion information of the vehicle, the processing unit is specifically configured to: determine a longitudinal length of a potential danger region according to the driving direction and the driving speed of the vehicle and a collision time corresponding to a safety distance of the vehicle; determine a transverse length of the potential danger region according to a width of a lane where the vehicle is located; determine the ROI region in the first image according to the longitudinal length and the transverse length of the potential danger region and the parameters of the image acquisition device.

8. The apparatus of claim 5, wherein, The weight value is determined according to at least one of the following parameters: an intersection-over-union of the first detection region and the second detection region, the intersection-over-union being a ratio of a size of an intersection region of the first detection region and the second detection region to a size of a union region of the first detection region and the second detection region; a number and a position of feature points included in the second detection region; a motion amount of each feature point included in the second detection region; a motion direction of each feature point included in the second detection region.

9. A potentially dangerous object detection apparatus characterized by comprising: The apparatus includes at least one processor coupled with at least one memory: the at least one processor is configured to execute computer programs or instructions stored in the at least one memory, so that the apparatus performs the method according to any one of claims 1 to 4.

10. A potentially dangerous object detection apparatus characterized by comprising: The apparatus includes a processor and an interface circuit; the interface circuit is configured to receive code instructions and transmit the code instructions to the processor; the processor is configured to run the code instructions to perform the method according to any one of claims 1 to 4.

11. A computer program storage medium, characterized in that, The computer program instructions are stored in the memory, and when the instructions are executed, the method according to any one of claims 1 to 4 is implemented.

Images