Method and device for predicting collision time based on image

Abstract

The invention provides a method and device for predicting collision time based on an image and a storage medium. The method comprises the following steps: collecting images of a plurality of frames, wherein the images of the plurality of frames comprise a first image corresponding to a first frame and a second image corresponding to a second frame; according to a first bounding box corresponding to the first image and a second bounding box corresponding to the second image, calculating a reference scale ratio between the two bounding boxes; generating a plurality of scale proportions associated with the reference scale proportion; scaling the second bounding box according to each scale ratio to obtain a third bounding box for capturing a first image block from the first image; capturing a second image block from the second image through a second bounding box; and performing similarity calculation on the first image block and the second image block corresponding to each scale proportion to screen out at least one scale proportion so as to calculate a predicted value of the collision time of the target object. According to the method for predicting the collision time based on the image, the accuracy of collision time prediction can be improved under the condition of low calculation overhead.

Description

Technical Field

[0001] The present disclosure relates to a technology for predicting collision time, and more particularly to a method, device and computer-readable storage medium for predicting collision time by image similarity. Background Art

[0002] In the field of autonomous driving, high-precision time to contact (TTC) estimation is required for collision time prediction. Inaccurate TTC estimation will affect the path planning and safety of the autonomous driving system. In this paper, TTC can represent the predicted value of the collision time between the vehicle and the target object (e.g., an obstacle or a preceding vehicle).

[0003] In the related art, TTC can be predicted by optical flow or detection box (or bounding box) scale ratio. The collision time prediction method based on optical flow is to detect the change of the intensity of image pixels over time and then infer the speed and direction of the object, and then estimate the predicted value of the collision time with the target object. However, optical flow estimation of TTC depends on accurate optical flow information, which will result in significant computational overhead. On the other hand, the collision time prediction method based on the detection box scale ratio calculates the scale ratio based on the detection box size in the two frames of images, and then directly calculates TTC. The formula for calculating TTC based on the scale ratio: Where α is the scale ratio from time t0 to time t1. However, when the detection box does not accurately mark the target object (for example, the size of the detection box in the two frames is inconsistent or inaccurate), a large TTC error will be caused.

[0004] Therefore, in order to achieve higher-precision TTC prediction, it is necessary to propose a more optimized TTC prediction method to improve the path planning efficiency and safety of the autonomous driving system. Summary of the invention

[0005] In view of this, the present disclosure proposes a method, an apparatus and a computer-readable storage medium for predicting collision time based on an image, so as to achieve high-precision collision time prediction.

[0006] The first aspect of the present disclosure proposes a method for predicting collision time based on an image, comprising: collecting multiple frames of images according to a target object, the multiple frames of images including a first image corresponding to a first frame and a second image corresponding to a second frame, the second frame being after the first frame in terms of timing; calculating a reference scale ratio between the two bounding boxes according to a first bounding box corresponding to the first image and a second bounding box corresponding to the second image, wherein the first bounding box and the second bounding box correspond to the target object; generating multiple scale ratios associated with the reference scale ratio; scaling the second bounding box according to each scale ratio to obtain a third bounding box for extracting a first image block from the first image; extracting a second image block from the second image through the second bounding box; performing similarity calculation on the first image block and the second image block corresponding to each scale ratio to screen out at least one scale ratio; and calculating a predicted value of the collision time of the target object according to the at least one screened scale ratio.

[0007] The second aspect of the present disclosure provides a device for predicting collision time based on an image, comprising one or more processors and a memory storing a program. The program comprises instructions, which, when executed by the processor, enable the device to perform the above-mentioned method for predicting collision time based on an image.

[0008] A third aspect of the present disclosure provides a computer-readable storage medium storing a program, wherein the program includes instructions, which, when executed by one or more processors of a computing device, cause the computing device to execute the above-mentioned method for predicting collision time based on an image.

[0009] Based on the above disclosure, since the above-mentioned method for predicting collision time based on image includes performing similarity calculation on the first image block and the second image block corresponding to each scale ratio to screen out at least one scale ratio; and calculating the predicted value of the collision time of the target object based on the at least one screened scale ratio, the present disclosure can screen out the predicted value calculated by the appropriate scale ratio through a small amount of similarity calculation results, so as to improve the accuracy of the predicted value of the collision time, thereby reducing the computing overhead and optimizing the path planning and safety of the autonomous driving system. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] The accompanying drawings exemplarily illustrate the embodiments and constitute a part of the specification, and together with the text description of the specification, are used to explain the exemplary implementation of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention, and for those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative work. In all drawings, the same reference numerals refer to similar but not necessarily identical elements.

[0011] Figure 1is a schematic diagram showing a vehicle in which the various techniques disclosed herein may be implemented;

[0012] Figure 2 is a flowchart showing a method for predicting collision time based on an image according to an embodiment of the present disclosure;

[0013] Figure 3 is a schematic diagram showing a process of predicting collision time based on an image according to an embodiment of the present disclosure;

[0014] Figure 4 is a schematic diagram showing a computing device according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0015] In order to make the purpose, technical scheme and advantages of the present invention clearer, the present invention will be further described in detail below in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0016] In this disclosure, the term "plurality" means two or more, unless otherwise specified. In this disclosure, the term "and / or" describes the association relationship of associated objects, covering any one of the listed objects and all possible combinations. The character " / " generally indicates that the associated objects before and after are in an "or" relationship.

[0017] In the present disclosure, unless otherwise specified, the terms "first", "second", etc. are used to distinguish similar objects without intending to limit their positional relationship, temporal relationship, or importance relationship. It should be understood that the terms used in this way can be interchangeable under appropriate circumstances, so that the embodiments of the present invention described herein can be implemented in ways other than those illustrated or described herein.

[0018] In addition, the terms "comprises," "comprising," and "having," and any variations thereof, are intended to cover non-exclusive inclusions. For example, a process, method, system, product, or apparatus that includes a series of steps or elements is not necessarily limited to those steps or elements explicitly listed, but may include other steps or elements not explicitly listed or inherent to such process, method, system, product, or apparatus.

[0019] Figure 1 is a schematic diagram illustrating a vehicle in which the various techniques disclosed herein may be implemented.

[0020] Please refer to Figure 1, the vehicle 100 may be a car, truck, motorcycle, bus, boat, airplane, helicopter, lawnmower, excavator, snowmobile, aircraft, recreational vehicle, amusement park vehicle, farm equipment, construction equipment, tram, golf cart, train, trolley bus, or other vehicle. The vehicle 100 may be operated in full or in part in an autonomous driving mode. The vehicle 100 may control itself in the autonomous driving mode, for example, the vehicle 100 may determine the current state of the vehicle and the current state of the environment in which the vehicle is located, determine the predicted behavior of at least one other vehicle in the environment, determine a confidence level corresponding to the likelihood that the at least one other vehicle will perform the predicted behavior, and control the vehicle 100 itself based on the determined information. When in the autonomous driving mode, the vehicle 100 may operate without human interaction.

[0021] The vehicle 100 may include various vehicle systems, such as a drive system 142, a sensor system 144, a control system 146, a user interface system 148, a computing system 150, and a communication system 152. The vehicle 100 may include more or fewer systems, and each system may include multiple units. Further, each system and unit of the vehicle 100 may be interconnected. For example, the computing system 150 may be able to communicate data with one or more of the drive system 142, the sensor system 144, the control system 146, the user interface system 148, and the communication system 152. Thus, one or more of the described functions of the vehicle 100 may be divided into additional functional components or physical components, or combined into a smaller number of functional components or physical components. In further examples, the additional functional components or physical components may be increased to, for example, Figure 1 In the example shown.

[0022] The drive system 142 may include a plurality of operable components (or units) that provide kinetic energy to the vehicle 100. In some embodiments, the drive system 142 may include an engine or motor, wheels, a transmission, an electronic system, and a power source (or power source). The engine or motor may be any combination of an internal combustion engine, an electric motor, a steam engine, a fuel cell engine, a propane engine, or other forms of engines or motors. In some embodiments, the engine may convert a power source into mechanical energy. In some embodiments, the drive system 142 may include a plurality of engines or motors. For example, a hybrid vehicle may include a gasoline engine and an electric motor, and may also include other situations.

[0023] The wheels of the vehicle 100 may be standard wheels. The wheels of the vehicle 100 may be wheels of various types, including single, double, triple, or quad wheels, such as quad wheels on a car or truck. Other numbers of wheels are also possible, such as six or more wheels. One or more wheels of the vehicle 100 may be operated to rotate in a different direction than the other wheels. The wheel may be at least one wheel fixedly connected to a transmission. The wheel may include a combination of metal and rubber, or a combination of other materials. The transmission may include a unit operable to transmit mechanical power from the engine to the wheels. For this purpose, the transmission may include a gearbox, a clutch, a differential gear, and a drive shaft. The transmission may also include other units. The drive shaft may include one or more axles that match the wheels. The electronic system may include a unit for transmitting or controlling electronic signals of the vehicle 100. These electronic signals may be used to activate multiple lights, multiple servo mechanisms, multiple motors, and other electronic drive or control devices in the vehicle 100. The power source may be an energy source that provides power to the engine or motor in whole or in part. That is, the engine or motor is capable of converting the power source into mechanical energy. Exemplarily, the power source may include gasoline, oil, petroleum-based fuels, propane, other compressed gas fuels, ethanol, fuel cells, solar panels, batteries, and other electrical energy sources. The power source may additionally or alternatively include any combination of a fuel tank, a battery, a capacitor, or a flywheel. The power source may also provide energy for other systems of the vehicle 100.

[0024] The sensor system 144 may include a plurality of sensors for sensing information about the environment and conditions of the vehicle 100. For example, the sensor system 144 may include an inertial measurement unit (IMU), a global navigation satellite system (GNSS) transceiver (e.g., a global positioning system (GPS) transceiver), a radio detection and ranging device (RADAR, referred to as radar), a laser detection and ranging system (LiDAR, referred to as laser radar), an acoustic sensor, an ultrasonic sensor, and an image capture device (e.g., a camera). The sensor system 144 may include a plurality of sensors for monitoring the vehicle 100 (e.g., an oxygen (O2) monitor, a fuel gauge sensor, an engine oil pressure sensor, and temperature, humidity, pressure sensors, etc.). Of course, in other embodiments, other sensors may also be configured. One or more sensors included in the sensor system 144 may be driven individually or collectively to update the position, orientation, or both of the one or more sensors.

[0025] The IMU may include a combination of sensors (e.g., accelerometers and gyroscopes) for sensing attitude changes (e.g., position changes and direction changes) of the vehicle 100 based on inertial acceleration. The GPS transceiver may be any sensor for estimating the geographic location of the vehicle 100. For this purpose, the GPS transceiver may include a receiver / transmitter to provide location information of the vehicle 100 relative to the earth. It should be noted that GPS is an example of a global navigation satellite system, so in some embodiments, the GPS transceiver may be replaced by a Beidou satellite navigation system transceiver or a Galileo satellite navigation system transceiver. The radar unit may use radio signals to sense objects in the environment where the vehicle 100 is located. In some embodiments, in addition to sensing objects, the radar unit may also be used to sense the speed and direction of objects approaching the vehicle 100. The LiDAR unit may be any sensor that uses lasers to sense objects in the environment where the vehicle 100 is located. In some embodiments, the LiDAR unit may include a laser source, a laser scanner, and a detector. The LiDAR unit is used to operate in a continuous (e.g., using heterodyne detection) or discontinuous detection mode. The image capture device may include a device for capturing multiple images of the environment where the vehicle 100 is located. An example of an image capture device is a camera, which may be a still image camera or a motion video camera.

[0026] The control system 146 is used to control the operation of the vehicle 100 and its components (or units). Accordingly, the control system 146 may include various units, such as a steering unit, a power control unit, a braking unit, and a navigation unit.

[0027] The steering unit can be a combination of machines that adjust the direction of travel of the vehicle 100. A power control unit (e.g., a throttle) can be used, for example, to control the speed of the engine, and thus the speed of the vehicle 100. The braking unit can include a combination of machines for decelerating the vehicle 100. The braking unit can use friction to decelerate the vehicle in a standard manner. In other embodiments, the braking unit can convert the kinetic energy of the wheels into electric current. The braking unit can also take other forms. The navigation unit can be any system that determines a driving path or route for the vehicle 100. The navigation unit can also dynamically update the driving path while the vehicle 100 is moving. The control system 146 can also additionally or optionally include other components (or units) that are not shown or described.

[0028] The user interface system 148 may be used to allow interaction between the vehicle 100 and external sensors, other vehicles, other computer systems, and / or users of the vehicle 100. For example, the user interface system 148 may include a standard visual display device (e.g., a plasma display, a liquid crystal display (LCD), a touch screen display, a head mounted display, or other similar display), a speaker or other audio output device, a microphone or other audio input device. For example, the user interface system 148 may also include a navigation interface and an interface for controlling the interior environment of the vehicle 100 (e.g., temperature, air conditioning, etc.).

[0029] The communication system 152 can provide a way for the vehicle 100 to communicate with one or more devices or other vehicles in the vicinity. In an exemplary embodiment, the communication system 152 can communicate with one or more devices directly or through a communication network. The communication system 152 can be, for example, a wireless communication system. For example, the communication system can use 3G cellular communication (such as CDMA, EVDO, GSM / GPRS) or 4G cellular communication (such as WiMAX or LTE), and can also use 5G cellular communication. Optionally, the communication system can communicate with a wireless local area network (WLAN) (for example, using In some embodiments, the communication system 152 can communicate directly with one or more devices or other vehicles around it, for example, using infrared, Bluetooth, or other communication methods. Or ZIGBEE. In other embodiments, other wireless protocols may also be used, such as various vehicle-mounted communication systems, which are also within the scope of the present application. For example, the communication system may include one or more dedicated short-range communication (DSRC) devices, V2V devices, or V2X devices, which may communicate data publicly or privately with vehicles and / or roadside stations.

[0030] The computing system 150 can control some or all functions of the vehicle 100. The autonomous driving control unit in the computing system 150 can be used to identify, evaluate, and avoid or overcome potential obstacles in the environment of the vehicle 100. Generally, the autonomous driving control unit can be used to control the vehicle 100 without a driver, or to assist the driver in controlling the vehicle. In some embodiments, the autonomous driving control unit is used to combine data from sensors, such as data from a GPS transceiver, radar data, LiDAR data, camera data, and data from other vehicle systems to determine the driving path or trajectory of the vehicle 100. The autonomous driving control unit can be activated to enable the vehicle 100 to be driven in an autonomous driving mode.

[0031] The computing system 150 may include at least one processor (which may include at least one microprocessor) that executes processing instructions (i.e., machine executable instructions) stored in a non-volatile computer-readable medium (e.g., a data storage device or memory). The computing system 150 may also be a plurality of computing devices that control components or systems of the vehicle 100 in a distributed manner. In some embodiments, the memory may contain processing instructions (e.g., program logic) that are executed by the processor to implement various functions of the vehicle 100. In some embodiments, the computing system 150 is capable of data communication with the drive system 142, the sensor system 144, the control system 146, the user interface system 148, and / or the communication system 152. The interface in the computing system is used to facilitate data communication between the computing system 150 and the drive system 142, the sensor system 144, the control system 146, the user interface system 148, and the communication system 152.

[0032] The memory may also include other instructions, including instructions for data transmission, instructions for data reception, instructions for interaction, or instructions for controlling the drive system 142 , the sensor system 144 , or the control system 146 or the user interface system 148 .

[0033] In addition to storing processing instructions, the memory may store a variety of information or data, such as image processing parameters, road maps, and path information, which may be used by the vehicle 100 and the computing system 150 during operation of the vehicle 100 in an autonomous, semi-autonomous, and / or manual mode.

[0034] Although the autonomous driving control unit is shown as being separate from the processor and memory, it should be understood that in some embodiments, some or all of the functions of the autonomous driving control unit may be implemented using program code instructions residing in one or more memories (or data storage devices) and executed by one or more processors, and the autonomous driving control unit may in some cases be implemented using the same processor and / or memory (or data storage device). In some embodiments, the autonomous driving control unit may be implemented at least in part using various dedicated circuit logic, various processors, various field programmable gate arrays ("FPGA"), various application specific integrated circuits ("ASIC"), various real-time controllers, and hardware.

[0035] The computing system 150 may control functions of the vehicle 100 based on input received from various vehicle systems (e.g., the drive system 142, the sensor system 144, and the control system 146), or input received from the user interface system 148. For example, the computing system 150 may use input from the control system 146 to control a steering unit to avoid an obstacle detected by the sensor system 144. In some embodiments, the computing system 150 may be used to control various aspects of the vehicle 100 and its systems.

[0036] Although Figure 1 1 shows various components (or units) integrated into the vehicle 100, one or more of which may be mounted on the vehicle 100 or separately associated with the vehicle 100. For example, the computing system 150 may exist partially or completely independently of the vehicle 100. Thus, the vehicle 100 may exist in the form of separate or integrated equipment units. The equipment units constituting the vehicle 100 may communicate with each other in the form of wired or wireless communications. In some embodiments, additional components or units may be added to each system or one or more components or units may be removed from the system (for example, Figure 1 LiDAR or radar as shown).

[0037] The disclosed embodiment proposes a method and related device for predicting collision time based on an image, which is used to improve the accuracy of the collision time TTC between the vehicle 100 and the target object (such as the vehicle in front or an obstacle). The disclosed embodiment can solve the TTC with large errors caused by directly calculating the collision time through the size ratio of the detection box / bounding box, and provide a more accurate TTC to optimize the path planning and safety of the autonomous driving system.

[0038] Figure 2 FIG. 2 is a flowchart showing a method 200 for predicting collision time based on an image according to an embodiment of the present disclosure.

[0039] Please refer to Figure 2, the method 200 for predicting collision time based on images according to the embodiment of the present disclosure can be applied to the vehicle 100. In step 202, the vehicle 100 or the computing system 150 collects a plurality of frames of images according to the target object, wherein the plurality of frames of images include a first image corresponding to the first frame and a second image corresponding to the second frame, and the second frame is after the first frame in terms of time sequence. In step 204, the vehicle 100 or the computing system 150 calculates a reference scale ratio between the two bounding boxes according to a first bounding box corresponding to the first image and a second bounding box corresponding to the second image, wherein the first bounding box and the second bounding box correspond to the target object. In step 206, the vehicle 100 or the computing system 150 generates a plurality of scale ratios associated with the reference scale ratio. In step 208, the vehicle 100 or the computing system 150 scales the second bounding box according to each scale ratio to obtain a third bounding box for extracting the first image block from the first image. In step 210, the vehicle 100 or the computing system 150 extracts the second image block from the second image through the second bounding box. In step 212, the vehicle 100 or the computing system 150 performs similarity calculation on the first image block and the second image block corresponding to each scale ratio to select at least one scale ratio. In step 214, the vehicle 100 or the computing system 150 calculates the predicted value of the collision time of the target object based on the at least one scale ratio selected.

[0040] The method 200 of the embodiment of the present disclosure first generates a plurality of different scale ratios to scale the bounding box according to the ratio between the bounding boxes of two frames of images in time sequence, so as to perform cropping in the two frames of images, performs similarity calculation on the two cropped images to select a better scale ratio, and calculates the collision time according to the selected scale ratio.

[0041] In some embodiments, for example, in step 212, the vehicle 100 or the computing system 150 performs similarity calculation on the first image block and the second image block corresponding to each scale ratio to select at least one scale ratio, which includes the following process:

[0042] The vehicle 100 or the computing system 150 calculates a mean square error of pixel values ​​between the first image block and the second image block based on each scale ratio, and selects at least one minimum mean square error (MSE) of pixel values ​​from the mean square errors of pixel values ​​calculated based on each scale ratio to obtain at least one scale ratio corresponding to at least one minimum mean square error of pixel values.

[0043] In some embodiments, the screening method for calculating the scale ratio of TTC is based on the mean squared error (MSE) of pixel values. However, in other embodiments, the screening method for calculating the scale ratio of TTC can be based on the structural similarity index (SSIM), peak signal-to-noise ratio (PSNR), normalized cross-correlation (NCC) or Manhattan distance (also called L1 norm), but is not limited to this.

[0044] In detail, Mean Squared Error (MSE) is the most common way to measure image differences. It simply calculates the average of the squares of the differences between corresponding pixels in two images.

[0045] The structural similarity index SSIM is a more complex measurement method that takes into account the structural information of the image. SSIM not only compares the pixel values ​​of two images, but also compares their brightness, contrast and structural similarity within a local window. The value of SSIM is, for example, between -1 and 1. The closer the value of SSIM is to 1, the more similar the images are.

[0046] Peak signal-to-noise ratio (PSNR) is an indicator to measure the quality of image compression and can also be used to compare the difference between two images. PSNR is usually used to measure the quality of image reconstruction. The higher the PSNR value, the better the image quality and the smaller the error.

[0047] Normalized Cross Correlation (NCC) is a metric used to compare two images. It is often used in pattern recognition and image processing to find the location of a small image (called a template) in a large image. NCC can handle image brightness and contrast changes well. The value of NCC is, for example, between -1 and 1. The closer the value of NCC is to 1, the more similar the images are.

[0048] Manhattan distance / L1 norm calculates the sum of the absolute values ​​of the corresponding pixel differences between two images.

[0049] In some embodiments, for example, in step 212, the vehicle 100 or the computing system 150 performs similarity calculation on the first image block and the second image block corresponding to each scale ratio to select at least one scale ratio, which includes the following process:

[0050] The vehicle 100 or the computing system 150 calculates the value of the structural similarity index (SSIM) between the first image block and the second image block based on each scale ratio. Then, the vehicle 100 or the computing system 150 selects at least one maximum structural similarity index value from the values ​​of the structural similarity index calculated based on each scale ratio to obtain at least one scale ratio corresponding to the value of the at least one maximum structural similarity index.

[0051] In some embodiments, for example, in step 212, the vehicle 100 or the computing system 150 performs similarity calculation on the first image block and the second image block corresponding to each scale ratio to select at least one scale ratio, which includes the following process:

[0052] The vehicle 100 or the computing system 150 calculates the value of the peak signal-to-noise ratio (PSNR) of the first image block and the second image block based on each scale ratio. Then, the vehicle 100 or the computing system 150 selects at least one maximum PSNR value from the PSNR values ​​calculated based on each scale ratio to obtain at least one scale ratio corresponding to the at least one maximum PSNR value.

[0053] In some embodiments, for example, in step 212, the vehicle 100 or the computing system 150 performs similarity calculation on the first image block and the second image block corresponding to each scale ratio to select at least one scale ratio, which includes the following process:

[0054] The vehicle 100 or the computing system 150 calculates the normalized cross correlation (NCC) value between the first image block and the second image block based on each scale ratio. Then, the vehicle 100 or the computing system 150 selects at least one maximum normalized cross correlation value from the normalized cross correlation values ​​calculated based on each scale ratio to obtain at least one scale ratio corresponding to the at least one maximum normalized cross correlation value.

[0055] In some embodiments, for example, in step 212, the vehicle 100 or the computing system 150 performs similarity calculation on the first image block and the second image block corresponding to each scale ratio to select at least one scale ratio, which includes the following process:

[0056] The vehicle 100 or the computing system 150 calculates the Manhattan distance value between the first image block and the second image block based on each scale ratio. Then, the vehicle 100 or the computing system 150 selects at least one minimum Manhattan distance value from the Manhattan distance values ​​calculated based on each scale ratio to obtain at least one scale ratio corresponding to the at least one minimum Manhattan distance value.

[0057] In some embodiments, for example, in step 214, the vehicle 100 or the computing system 150 calculates the predicted value of the collision time of the target object according to the at least one selected scale ratio, including the following process:

[0058] The vehicle 100 or the computing system 150 calculates a predicted value of the collision time according to at least one scale ratio corresponding to at least one minimum pixel value square error.

[0059] In some embodiments, for example, in step 214, the vehicle 100 or the computing system 150 calculates the predicted value of the collision time of the target object according to the at least one selected scale ratio, including the following process:

[0060] The vehicle 100 or the computing system 150 calculates a predicted value of the collision time based on at least one scale ratio corresponding to at least one minimum pixel value square error, at least one maximum structural similarity index value, at least one maximum peak signal-to-noise ratio value, at least one maximum normalized cross-correlation value, or at least one minimum Manhattan distance value.

[0061] In some embodiments, the vehicle 100 or the computing system 150 may further perform a weighted average operation (a first weighted average operation) on the collision time calculated by at least one scale ratio to obtain a predicted value of the collision time.

[0062] In some embodiments, the vehicle 100 or the computing system 150 may further perform a weighted average operation (a second weighted average operation) on at least one scale ratio to obtain a weighted average scale ratio, and then calculate a predicted value of the collision time based on the weighted average scale ratio.

[0063] In some embodiments, for example in step 206 , the vehicle 100 or the computing system 150 generates a plurality of scale ratios according to the reference scale ratio and a range parameter associated with the reference scale ratio.

[0064] In some embodiments, for example, in step 206, the vehicle 100 or the computing system 150 calculates the minimum scale ratio and the maximum scale ratio according to the reference scale ratio and the range parameter. Then, the vehicle 100 or the computing system 150 generates a plurality of scale ratios within the range of the minimum scale ratio and the maximum scale ratio according to the minimum scale ratio and the maximum scale ratio.

[0065] In some embodiments, the vehicle 100 or the computing system 150 sets the range parameter according to the distance between the imaging device used to collect the plurality of frames of images and the target object. In short, the setting of the range parameter is related to the distance between the vehicle and the target object.

[0066] In some embodiments, the reference scale ratio includes a width ratio and a height ratio between the two bounding boxes.

[0067] In some embodiments, the plurality of scale ratios include a width ratio and a height ratio associated with a base scale ratio.

[0068] In some embodiments, for example in step 208 , the vehicle 100 or the computing system 150 scales the second bounding box according to the width ratio and the height ratio in each scale ratio.

[0069] In some embodiments, for example in step 208 , the vehicle 100 or the computing system 150 extracts a first image block from the first image through a third bounding box based on the center position of the first bounding box.

[0070] In some embodiments, for example, in step 208, the vehicle 100 or the computing system 150 scales the second bounding box according to each scale ratio to obtain a third bounding box, and the step of extracting the first image block from the first image includes the following process:

[0071] The vehicle 100 or the computing system 150 extracts a central image block from the first image through a third bounding box based on the central position of the first bounding box; the vehicle 100 or the computing system 150 adjusts the central position of the first bounding box to at least one other position on the first image; for each other position, the vehicle 100 or the computing system 150 extracts other image blocks from the first image through the third bounding box to obtain at least one other image block; and the vehicle 100 or the computing system 150 respectively calculates the similarity between the central image block and at least one other image block and the second image block, so as to select from the central image block and at least one other image block to obtain the first image block. In short, in order to reduce the error caused by the inaccurate center of the first bounding box, the center of the first bounding box is offset and then other image blocks with different central positions are extracted from the first bounding box through the third bounding box. Further, the vehicle 100 or the computing system 150 respectively calculates the similarity between the central image block and other image blocks and the second image block to select an appropriate first image block.

[0072] In some embodiments, the vehicle 100 or the computing system 150 calculates the pixel value mean square error (or the above-mentioned structural similarity index, peak signal-to-noise ratio, normalized cross-correlation or Manhattan distance) of the central image block and at least one other image block respectively with the second image block; and the vehicle 100 or the computing system 150 selects the central image block or one of the at least one other image blocks corresponding to the minimum pixel value mean square error (or the above-mentioned maximum structural similarity index value, the maximum peak signal-to-noise ratio value, the maximum normalized cross-correlation value, or the minimum Manhattan distance value) from the calculated pixel value mean square error (or the above-mentioned structural similarity index, the maximum peak signal-to-noise ratio value, the maximum normalized cross-correlation value, or the minimum Manhattan distance value) as the first image block.

[0073] In some embodiments, before performing similarity calculation, the vehicle 100 or the computing system 150 adjusts the size of the first image block to be the same as that of the second image block.

[0074] In some embodiments, before performing similarity calculation, the vehicle 100 or the computing system 150 scales the width and height of the first image block to be the same as the width and height of the second image block.

[0075] The specific process of predicting TTC in the embodiment of the present disclosure is described in detail as follows.

[0076] Figure 3 is a schematic diagram showing a process of predicting collision time based on an image according to an embodiment of the present disclosure.

[0077] like Figure 3 As shown, in the two frames of images F0 and F1 at time t0 and t1, the vehicle 100 or the computing system 150 tracks the specific vehicle and marks the bounding box thereof to calculate the TTC between the vehicle 100 and the specific vehicle. The representation of the bounding box can be b i =[x i ,y i ,w i ,h i ]. Where i = 0, 1, x i ,y i is the center point coordinate of the bounding box, w i ,h i is the width and height of the bounding box. Therefore, the width ratio α of the bounding box in the two frames F0 and F1 is w and high ratio α h It can be expressed as:

[0078]

[0079] In addition, according to a preset scale ratio range r (the scale ratio range r can be set based on experience, such as 0.1 or 0.2), the vehicle 100 or the computing system 150 can obtain a minimum scale ratio and a maximum scale ratio, respectively. min It can be expressed as α min =min(α w ,α h )-r. That is, the minimum scale ratio α min You can choose a wide ratio α w and high ratio α h The smaller one and subtract the default scale ratio range r. Maximum scale ratio α max It can be expressed as α max =max(α w ,α h)+r. That is, the maximum scale ratio α max You can choose a wide ratio α w and high ratio α h The larger one and the preset scale ratio range r are added. Alternatively, an appropriate minimum scale ratio α can be set according to other criteria. min , maximum scale ratio α max And the preset scale ratio range r.

[0080] Then, the vehicle 100 or the computing system 150 may enumerate n scale ratios according to the minimum scale ratio and the maximum scale ratio (for example, linearly generate n scale ratios according to the data range between the minimum scale ratio and the maximum scale ratio. n may be set based on experience, such as 20 or 30), where n scale ratios α i It can be expressed as:

[0081]

[0082] In this embodiment, the image block C1 can be obtained by cropping the image F1 using the bounding box b1 of the image F1. i , with (x0, y0) as the center (that is, the center point coordinates of the bounding box b0 (x0, y0)), by cropping the image F0 with a width of α i w1, height is α i h1 image block (that is, according to each scale ratio α i Scale the width and height of the bounding box b1, then crop the image F0), and scale the cropped image block to an image block with a width and height of w1, h1 (That is, the cropped image block is scaled to the same size as the image block C1 cropped from the bounding box b1). The above-mentioned scaling may be accomplished, for example, by interpolation. The vehicle 100 or the computing system 150 calculates the image block and the mean square error mse of the C1 pixel values i Next, the vehicle 100 or the computing system 150 calculates the scale ratio α according to each scale ratio α. i The mean square error mse of the calculated pixel values i , take the k scales with the smallest mean square error of pixel values (k can be set based on experience, for example, 3), that is, the weighted average collision time can be obtained by weighted average operation. The weighted average collision time TTC can be expressed as:

[0083]

[0084] In short, the vehicle 100 or the computing system 150 processes each selected scale ratio. The corresponding TTC calculates its weighted average to obtain the predicted value of TTC.

[0085] Specifically, the mathematical relationship between the scale ratio and TTC is nonlinear. When the scale ratio is close to 1, a large TTC value will be calculated, which will have a greater impact on the final weighted average TTC value. Therefore, in some embodiments, the weighted average of the scale ratio can be calculated first, and then the predicted value of TTC can be calculated. Specifically, the vehicle 100 or the computing system 150 can calculate the predicted value of TTC based on the k scale ratios with the smallest mean square error of the above pixel values. The weighted average operation is performed to obtain a weighted average scale ratio. Then, the vehicle 100 or the computing system 150 can calculate the predicted value of TTC based on the weighted average scale ratio. It can be expressed, for example, as:

[0086]

[0087] According to the weighted average scale ratio The predicted value of the calculated TTC can be expressed as:

[0088]

[0089] It is worth noting that in addition to calculating the image block The mean square error of the pixel values ​​of C1 can also be calculated by The structural similarity index SSIM, peak signal-to-noise ratio PSNR, normalized cross correlation NCC or Manhattan distance / L1 norm of C1 are used to screen out the appropriate scale to calculate the collision time TTC.

[0090] In addition, in order to reduce the error caused by the inaccurate center of the bounding box b0, the center point coordinates (x0, y0) can be offset (for example, taking (x0, y0) as the reference point, the center point is offset by a predetermined number of pixels (the predetermined number is, for example, 1) in the upward, downward, left, and right directions, respectively). Then, the image blocks are cut according to the offset center point to calculate the mean square error of the pixel values ​​of each image block and the image block C1 from these image blocks. After that, the image block corresponding to the mean square error with the smallest pixel value among these image blocks is taken to determine the image block at a specific scale ratio.

[0091] For example, for each scale ratio α i , image block Before calculating the similarity with C1, the image block is first determined by the above-mentioned center point offset method (x0, y0) can be used as the reference point (center position), and the center point can be offset in the directions of up, down, left, and right to form 4 positions. And (x0, y0) can be used as the reference (center position), and the center point can be offset in the directions of upper left, upper right, lower left, and lower right to form another 4 positions. At this time, there are 9 points including the center position and 8 other positions, and these 9 points form a neat arrangement of 3x3, for example. Then, the image is cropped according to these 9 points to obtain 9 image blocks, and the mean square error of the pixel values ​​of each image block and the image block C1 is calculated from these image blocks. Afterwards, the image block corresponding to the mean square error with the smallest pixel value among these 9 image blocks is taken to determine the image block at this scale ratio.

[0092] On the other hand, as the distance between the vehicle 100 and the target object increases, the scale ratio range will theoretically decrease. Therefore, the scale ratio range r can be set to be related to the distance of the target, thereby improving the TTC prediction accuracy of the distant target.

[0093] The following Table 1 compares the scale ratio obtained by predicting the collision time based on the scale ratio of the traditional detection frame and the scale ratio obtained by predicting the collision time based on the image in the embodiment of the present disclosure. In detail, Table 1 compares the average error values ​​between the scale ratio obtained by the present disclosure based on the image prediction of the collision time and the traditional scale ratio directly using the detection frame at different distances, and the real scale ratio. In one embodiment, the real scale ratio can be obtained by the laser radar on the vehicle, for example, the real TTC is obtained by laser radar ranging, and the real scale ratio is calculated by back calculation.

[0094] Table 1

[0095]

[0096] As shown in Table 1, the scale ratio obtained according to the implementation of the present disclosure has a smaller error than the true scale ratio, so that a more accurate TTC can be obtained. In other words, compared with the traditional method of directly calculating TTC based on the scale ratio of the detection box, the implementation of the present disclosure has a smaller TTC error when the detection box / bounding box is inaccurate. Therefore, the method of predicting collision time based on an image in an embodiment of the present disclosure can achieve a higher accuracy TTC prediction, so that the path planning efficiency and safety of the autonomous driving system can be improved. In addition, because the method of predicting collision time based on an image in an embodiment of the present disclosure does not need to rely on accurate optical flow information, the computational overhead is low.

[0097] Figure 4 is a schematic diagram showing a computing device according to an embodiment of the present disclosure.

[0098] Please refer to Figure 4In some embodiments, the computing system 150 may be implemented in the form of a computing device 400 .

[0099] In some embodiments, in computing device 400, instruction set when being executed and / or processing logic when being started can make the machine perform any one or more of the methods described and / or required herein. In alternative embodiments, the machine operates as a standalone device, or can be connected (e.g., networked) to other machines. In a networked deployment, the machine can operate as a server or client machine in a server-client network environment, or as a peer machine in a peer (or distributed) network environment. The machine can be a personal computer (PC), a laptop computer, a tablet computing system, a personal digital assistant (PDA), a cellular phone, a smart phone, a network application, a set-top box (STB), a network router, a switch or a bridge or any machine capable of executing an instruction set (in succession or otherwise) or starting processing logic specifying the action to be taken by the machine. Further, although only a single machine is illustrated, the term "machine" can also be understood to include any set of machines that individually or jointly execute an instruction set (or multiple instruction sets) to perform any one or more of the methods described and / or required herein.

[0100] The computing device 400 may include a data processor 402 (e.g., a system-on-chip (SoC), a general processing core, a graphics core, and optional other processing logic) and a memory 404 (e.g., memory) that may communicate with each other via a bus 406 or other data transfer system. The computing device 400 may also include various input / output (I / O) devices and / or interfaces 410, such as a touch screen display, an audio jack, a voice interface, and an optional network interface 412. In an example embodiment, the network interface 412 may include one or more radio transceivers configured to communicate with any one or more standard wireless and / or cellular protocols or access technologies (e.g., second generation (2G), 2.5 generation, third generation (3G), fourth generation (4G) and next generation radio access of cellular systems, global system for mobile communications (GSM), general packet radio service (GPRS), enhanced data GSM environment (EDGE), wideband code division multiple access (WCDMA), LTE, CDMA2000, WLAN, wireless router (WR) mesh, etc.). The network interface 412 may also be configured to communicate with various other wired and / or wireless communication protocols including TCP / IP, UDP, SIP, SMS, RTP, WAP, CDMA, TDMA, UMTS, UWB, WiFi, WiMax, In essence, network interface 412 may include or support virtually any wired and / or wireless communication and data processing mechanism by which information / data may be transmitted between computing device 400 and another computing or communication system via network 414.

[0101] The memory 404 may represent a machine-readable medium (or computer-readable storage medium) on which one or more instruction sets, software, firmware, or other processing logic (e.g., logic 408) are stored that implement any one or more of the methods or functions described and / or required herein. During execution by the computing device 400, the logic 408 or a portion thereof may also reside completely or at least partially within the processor 402. In this way, the memory 404 and the processor 402 may also constitute a machine-readable medium (or computer-readable storage medium). The logic 408 or a portion thereof may also be configured as a processing logic or logic, at least a portion of which is partially implemented in hardware. The logic 408 or a portion thereof may also be transmitted or received over a network 414 via a network interface 412. Although the machine-readable medium (or computer-readable storage medium) of the example embodiment may be a single medium, the term "machine-readable medium" (or computer-readable storage medium) should be understood to include a single non-transitory medium or multiple non-transitory media (e.g., a centralized or distributed database and / or associated cache and computing system) storing one or more instruction sets. The term "machine-readable medium" (or computer-readable storage medium) may also be understood to include any non-transitory medium that can store, encode, or carry an instruction set for execution by a machine and cause the machine to perform any one or more of the methods of the various embodiments, or that can store, encode, or carry a data structure utilized by or associated with such an instruction set. The term "machine-readable medium" (or computer-readable storage medium) may therefore be understood to include, but is not limited to, solid-state memories, optical media, and magnetic media.

[0102] The disclosed and other embodiments, modules, and functional operations described in this document may be implemented in digital electronic circuit systems, or in computer software, firmware or hardware (including the structures disclosed in this document and their structural equivalents), or in a combination of one or more of them. The disclosed and other embodiments may be implemented as one or more computer program products, that is, one or more modules of computer program instructions encoded on a computer-readable medium to be executed by a data processing device or to control the operation of the data processing device. The computer-readable medium may be a machine-readable storage device, a machine-readable storage substrate, a memory device, a material composition that affects a machine-readable propagation signal, or a combination of one or more of them. The term "data processing device" covers all devices, equipment, and machines for processing data, including, for example, a programmable processor, a computer or multiple processors or computers. In addition to hardware, the device may also include code that creates an execution environment for the computer program in question, such as code that constitutes processor firmware, a protocol stack, a database management system, an operating system, or a combination of one or more of them. A propagation signal is an artificially generated signal, such as an electrical signal, an optical signal, or an electromagnetic signal generated by a machine, which is generated to encode information to be transmitted to a suitable receiver device.

[0103] A computer program (also referred to as a program, software, software application, script, or code) can be written in any form of programming language (including compiled or interpreted languages), and the computer program can be deployed in any form, including being deployed as an independent program or as a module, component, subroutine, or another unit suitable for use in a computing environment. A computer program does not necessarily correspond to a file in a file system. A program may be stored in a portion of a file that holds other programs or data (e.g., one or more scripts stored in a markup language document), or in a single file dedicated to the program under discussion, or in multiple collaborative files (e.g., files storing one or more modules, subroutines, or partial codes). A computer program may be deployed to be executed on one computer or on multiple computers located at one site or distributed in multiple sites and interconnected by a communication network.

[0104] The processes and logic flows described in this document may be performed by one or more programmable processors executing one or more computer programs to perform functions by operating on input data and generating output. The processes and logic flows may also be performed by a dedicated logic circuit system (e.g., FPGA (field programmable gate array) or ASIC (application-specific integrated circuit)), and the apparatus may also be implemented as a dedicated logic circuit (e.g., FPGA (field programmable gate array) or ASIC (application-specific integrated circuit)).

[0105] Processors suitable for executing computer programs include, for example, both general-purpose microprocessors and special-purpose microprocessors and any one or more processors of any kind of digital computer. Typically, the processor will receive instructions and data from a read-only memory or a random access memory or both. The necessary elements of a computer are a processor for executing instructions and one or more memory devices for storing instructions and data. Typically, a computer will also include one or more mass storage devices (such as magnetic disks, magneto-optical disks, or optical disks) for storing data, or the computer will also be operatively connected to receive data from the one or more mass storage devices or to transfer data to the one or more mass storage devices or to perform both. However, the computer does not need to have such a device. Computer-readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media, and memory devices, including, for example, semiconductor memory devices, such as EPROM, EEPROM, and flash memory devices; magnetic disks, such as internal hard disks or removable disks; magneto-optical disks; and CD-ROM and DVD-ROM disks. The processor and memory may be supplemented by a dedicated logic circuit system or may be incorporated into the dedicated logic circuit system.

[0106] Some embodiments described herein are described in the general context of a method or process, which in one embodiment can be implemented by a computer program product included in a computer-readable medium, which may include computer executable instructions (such as program code), which can be executed by a computer in a networked environment, for example. Computer-readable media may include removable and non-removable storage devices, including but not limited to read-only memory (ROM), random access memory (RAM), compact disk (CD), digital versatile disk (DVD), etc. Therefore, computer-readable media may include non-transitory storage media. Generally, program modules may include routines, programs, objects, components, data structures, etc., which perform specific tasks or implement specific abstract data types. Computer or processor executable instructions, associated data structures, and program modules represent examples of program codes for executing the steps of the methods disclosed herein. Such specific sequences of executable instructions or associated data structures represent examples of corresponding actions for implementing the functions described in these steps or processes.

[0107] Some embodiments of the disclosed embodiments can be implemented as devices or modules using hardware circuits, software or a combination thereof. For example, a hardware circuit implementation may include discrete analog and / or digital components, which may be integrated as parts of a printed circuit board, for example. Alternatively or additionally, the disclosed components or modules may be implemented as application specific integrated circuits (ASICs) and / or field programmable gate arrays (FPGAs) devices. Additionally or alternatively, some implementations may include a digital signal processor (DSP), which is a dedicated microprocessor with an architecture optimized for the operational needs of digital signal processing associated with the disclosed functionality of the present application. Similarly, various components or subassemblies within each module may be implemented with software, hardware or firmware. Any connection method and medium known in the art may be used to provide a connection between a module and / or a component within a module, including but not limited to communication via the Internet, a wired network, or a wireless network using an appropriate protocol.

[0108] Although this paper includes many details, these details should not be interpreted as limitations to the scope of the claimed invention, but are interpreted as descriptions of features peculiar to specific embodiments. Some features described in the context of different embodiments herein may also be combined in a single embodiment. On the contrary, the various features described in the context of a single embodiment may also be separated or implemented in multiple embodiments with any suitable sub-combination. In addition, although features may be described as acting with some combinations and even initially claimed above, one or more features from the claimed combination may be removed from the combination in some cases, and the claimed combination may be directed to a variation of a sub-combination or a sub-combination. Similarly, although operations are depicted in a particular order in the drawings, this should not be understood as requiring to perform such operations in the particular order shown or in a sequential order, or to perform all described operations to achieve desired results.

Claims

1. A method for predicting collision time based on an image, characterized in that: include: According to the target object, a plurality of frames of images are collected, wherein the plurality of frames of images include a first image corresponding to a first frame and a second image corresponding to a second frame, wherein the second frame is after the first frame in terms of time sequence; calculating a reference scale ratio between a first bounding box corresponding to the first image and a second bounding box corresponding to the second image, wherein the first bounding box and the second bounding box correspond to the target object; generating a plurality of scale ratios associated with the reference scale ratio; scaling the second bounding box according to each scale ratio to obtain a third bounding box for extracting a first image block from the first image; Extracting a second image block from the second image using the second bounding box; Performing similarity calculation on the first image block and the second image block corresponding to each scale ratio to screen out at least one scale ratio; as well as A predicted value of the collision time of the target object is calculated according to the at least one scale ratio that is screened out.

2. The method for predicting collision time based on an image according to claim 1, characterized in that: Calculating similarity between the first image block and the second image block corresponding to each scale ratio to select the at least one scale ratio includes: Based on each scale ratio, calculating a mean square error of pixel values ​​between the first image block and the second image block; and At least one minimum pixel value mean square error is selected from the pixel value mean square errors calculated based on each scale ratio to obtain the at least one scale ratio corresponding to the at least one minimum pixel value mean square error.

3. The method for predicting collision time based on an image according to claim 2, characterized in that: Calculating the predicted value of the collision time of the target object according to the at least one scale ratio selected includes: The predicted value of the collision time is calculated according to the at least one scale ratio corresponding to the at least one minimum pixel value square error.

4. The method for predicting collision time based on an image according to claim 3, characterized in that: Calculating the predicted value of the collision time of the target object according to the at least one scale ratio corresponding to the at least one minimum pixel value square error includes: A first weighted average operation is performed on the collision time calculated by the at least one scale ratio to obtain a predicted value of the collision time.

5. The method for predicting collision time based on an image according to claim 3, characterized in that: Calculating the predicted value of the collision time of the target object according to the at least one scale ratio corresponding to the at least one minimum pixel value square error includes: performing a second weighted average operation on the at least one scale ratio to obtain a weighted average scale ratio; and The predicted value of the collision time is calculated according to the weighted average scale ratio.

6. The method for predicting collision time based on an image according to claim 1, characterized in that: Calculating similarity between the first image block and the second image block corresponding to each scale ratio to select the at least one scale ratio includes: Based on each scale ratio, calculating a value of a structural similarity index between the first image block and the second image block; and At least one maximum structural similarity index value is selected from the values ​​of the structural similarity index calculated based on each scale ratio to obtain the at least one scale ratio corresponding to the at least one maximum structural similarity index value.

7. The method for predicting collision time based on an image according to claim 1, characterized in that: Calculating similarity between the first image block and the second image block corresponding to each scale ratio to select the at least one scale ratio includes: Based on each scale ratio, calculating a value of a peak signal-to-noise ratio of the first image block and the second image block; and At least one maximum peak signal-to-noise ratio value is selected from the peak signal-to-noise ratio values ​​calculated based on each scale ratio to obtain the at least one scale ratio corresponding to the at least one maximum peak signal-to-noise ratio value.

8. The method for predicting collision time based on an image according to claim 1, characterized in that: Calculating similarity between the first image block and the second image block corresponding to each scale ratio to select the at least one scale ratio includes: Based on each scale ratio, calculating a value of a normalized cross-correlation between the first image block and the second image block; and At least one maximum normalized cross-correlation value is selected from the normalized cross-correlation values ​​calculated based on each scale ratio to obtain the at least one scale ratio corresponding to the at least one maximum normalized cross-correlation value.

9. The method for predicting collision time based on images as claimed in claim 1, characterized in that: Calculating similarity between the first image block and the second image block corresponding to each scale ratio to select the at least one scale ratio includes: Based on each scale ratio, calculating a value of the Manhattan distance between the first image block and the second image block; and At least one minimum Manhattan distance value is selected from the Manhattan distance values ​​calculated based on each scale ratio to obtain the at least one scale ratio corresponding to the at least one minimum Manhattan distance value.

10. The method for predicting collision time based on an image according to claim 1, characterized in that: Generating the plurality of scale ratios associated with the reference scale ratio comprises: The plurality of scale ratios are generated according to the reference scale ratio and a range parameter associated with the reference scale ratio.

11. The method for predicting collision time based on an image according to claim 10, characterized in that: Generating the plurality of scale ratios according to the reference scale ratio and the range parameter associated with the reference scale ratio comprises: Calculating a minimum scale ratio and a maximum scale ratio according to the reference scale ratio and the range parameter; and According to the minimum scale ratio and the maximum scale ratio, the plurality of scale ratios within the range of the minimum scale ratio and the maximum scale ratio are generated.

12. The method for predicting collision time based on an image according to claim 10, characterized in that: The method further includes setting the range parameter according to a distance between an imaging device used to collect the plurality of frames of images and the target object.

13. The method for predicting collision time based on an image according to claim 1, characterized in that: The reference scale ratio includes a width ratio and a height ratio between the two bounding boxes.

14. The method for predicting collision time based on an image according to claim 1, characterized in that: The plurality of scale ratios include a width ratio and a height ratio associated with the base scale ratio.

15. The method for predicting collision time based on an image according to claim 14, characterized in that: Scaling the second bounding box according to each scale ratio to obtain the third bounding box for extracting the first image block from the first image includes: The second bounding box is scaled according to the width ratio and the height ratio in each scale ratio.

16. The method for predicting collision time based on an image according to claim 1, characterized in that: Scaling the second bounding box according to each scale ratio to obtain the third bounding box for extracting the first image block from the first image includes: Based on the center position of the first bounding box, the first image block is extracted from the first image through the third bounding box.

17. The method for predicting collision time based on an image according to claim 1, characterized in that: Scaling the second bounding box according to each scale ratio to obtain the third bounding box for extracting the first image block from the first image includes: Based on the center position of the first bounding box, extracting a center image block from the first image through the third bounding box; adjusting the center position of the first bounding box to at least one other position on the first image; For each other position, extracting other image blocks from the first image through the third bounding box to obtain at least one other image block; and The similarity between the central image block and the at least one other image block and the second image block is calculated respectively, so as to filter the central image block and the at least one other image block to obtain the first image block.

18. The method for predicting collision time based on an image according to claim 17, characterized in that: Calculating similarities between the central image block and the at least one other image block and the second image block respectively, so as to select the central image block and the at least one other image block to obtain the first image block, comprising: Calculate pixel value mean square errors of the central image block and the at least one other image block and the second image block respectively; and From the calculated pixel value mean square errors, one of the central image block or the at least one other image block corresponding to the minimum pixel value mean square error is selected as the first image block.

19. The method for predicting collision time based on an image according to claim 1, characterized in that: Also includes: The size of the first image block is adjusted to be the same as that of the second image block.

20. The method for predicting collision time based on an image according to claim 19, characterized in that: Adjusting the size of the first image block to be the same as that of the second image block, comprising: The width and height of the first image block are scaled to be the same as the width and height of the second image block.

21. A device for predicting collision time based on an image, characterized in that: It includes: one or more processors, and A memory storing a program, wherein the program includes instructions, and when the instructions are executed by the processor, the target object labeling device performs the method according to any one of claims 1 to 20.

22. A computer-readable storage medium storing a program, characterized in that: The program comprises instructions which, when executed by one or more processors of a computing device, cause the computing device to perform a method according to any one of claims 1 to 20.

Images