Method for predicting collision time based on image and method for training neural network

Abstract

The invention provides a method for predicting collision time based on an image, and the method comprises the steps: collecting a plurality of frames of images according to a target object, and the plurality of frames of images comprise a first image corresponding to a first frame and a second image corresponding to a second frame; obtaining a first bounding box corresponding to the first image and a second bounding box corresponding to the second image; respectively coding the first image and the second image to obtain a first feature image and a second feature image; scaling the second bounding box according to each of a plurality of scale proportions to obtain a third bounding box; capturing a first feature image block from the first feature image through a third bounding box; capturing a second feature image block from the second feature image through a second bounding box; and performing similarity calculation on the first feature image block and the second feature image block corresponding to each scale proportion to screen out at least one scale proportion so as to calculate a predicted value of the collision time of the target object. In addition, a method for training the neural network is also provided.

Description

Technical Field

[0001] The present disclosure relates to a technology for predicting collision time, and more particularly to a method for predicting collision time through image similarity and a neural network, and a method for training a neural network. Background Art

[0002] In the field of autonomous driving, high-precision time to contact (TTC) estimation is required for collision time prediction. Inaccurate TTC estimation will affect the path planning and safety of the autonomous driving system. In this paper, TTC can represent the predicted value of the collision time between the vehicle and the target object (e.g., an obstacle or a preceding vehicle).

[0003] In the related art, the TTC of the vehicle and other vehicles can be estimated based on the image by using closed curves, optical flow, brightness or light intensity. Alternatively, the detection box (or bounding box) scale ratio method can be used to predict TTC. The collision time prediction method based on optical flow is to infer the speed and direction of the object's movement by detecting the change in the position offset of the pixel points of the image over time, and then estimate the predicted value of TTC between the vehicle and the target object. However, TTC estimated by optical flow depends on accurate optical flow information, which leads to significant computational overhead. Specifically, whether it is a closed curve, optical flow, brightness or light intensity method, it requires careful adjustment of parameters or strong prior knowledge, such as constant brightness or static scenes. This limits the practical applicability of predicting TTC. In addition, deep learning-based methods can also be used for TTC estimation. Deep learning methods require a large amount of real scene flow data with annotations. However, due to the expensive annotation cost, most scene flow data sets are obtained by synthesis, resulting in a gap between them and actual application fields.

[0004] In addition to the scene flow-based method, the single object tracking (SOT)-based method can also infer the scale ratio between the template and the tracked object. However, this type of method uses a large multiple of downsampling during the neural network reasoning process, resulting in inaccurate estimation of the object bounding box and unable to meet the requirements of TTC estimation. In addition, the collision time prediction method based on the bounding box scale ratio calculates the scale ratio based on the bounding box size in the two frames of the image, and then directly calculates TTC. The formula for calculating TTC based on the scale ratio is: Where α is the scale ratio from time t0 to time t1. However, when the bounding box does not accurately mark the target object (for example, the sizes of the bounding boxes in the two frames are inconsistent or inaccurate), a large TTC error will be caused.

[0005] Therefore, in order to achieve higher-precision TTC prediction, it is necessary to propose a more optimized TTC prediction method to improve the path planning efficiency and safety of the autonomous driving system. Summary of the invention

[0006] In view of this, the present disclosure proposes a method, an apparatus and a computer-readable storage medium for predicting collision time based on an image, so as to achieve high-precision collision time prediction.

[0007] The first aspect of the present disclosure proposes a method for predicting collision time based on an image, comprising: collecting multiple frames of images according to a target object, wherein the multiple frames of images include a first image corresponding to a first frame and a second image corresponding to a second frame, and the second frame is after the first frame in time sequence; obtaining a first bounding box and a second bounding box, wherein the first bounding box corresponds to the target object in the first image, and the second bounding box corresponds to the target object in the second image; encoding the first image and the second image respectively to obtain a first feature image and a second feature image; scaling the second bounding box according to each of a plurality of scale ratios to obtain a third bounding box; extracting a first feature image block from the first feature image through the third bounding box; extracting a second feature image block from the second feature image through the second bounding box; calculating similarity between the first feature image block and the second feature image block corresponding to each scale ratio to obtain a similarity score corresponding to each scale ratio; and selecting at least one scale ratio according to the similarity score corresponding to each scale ratio to calculate a predicted value of the collision time of the target object.

[0008] The second aspect of the present disclosure proposes a method for training a neural network, comprising: obtaining a first bounding box and a second bounding box, wherein the first bounding box corresponds to a target object in a first image, and the second bounding box corresponds to a target object in a second image, the first image corresponds to a first frame, and the second image corresponds to a second frame, and the second frame is after the first frame in time sequence; inputting the first image and the second image into a neural network to obtain a first feature image and a second feature image output by a prediction model; scaling the second bounding box according to each of a plurality of scale ratios to obtain a third bounding box; extracting a first feature image block from the first feature image through the third bounding box; extracting a first feature image block from the second feature image through the second bounding box extracting a second feature image block from the image; performing similarity calculation on the first feature image block and the second feature image block corresponding to each scale ratio to obtain a similarity score corresponding to each scale ratio, wherein the similarity score is used to indicate whether the scale ratio corresponding to the similarity score is close to the true scale ratio; obtaining a sample similarity score corresponding to each scale ratio according to the true scale ratio; comparing the similarity score and the sample similarity score for each scale ratio to generate a comparison result; and updating the prediction model by the neural network according to the comparison result to output a new similarity score, wherein the updated prediction model is used to make the new similarity score more responsive to whether the corresponding scale ratio is close to the true scale ratio.

[0009] The third aspect of the present disclosure provides a device for predicting collision time based on an image, comprising one or more processors and a memory storing a program. The program comprises instructions, which, when executed by the processor, enable the device to execute the above-mentioned method for predicting collision time based on an image.

[0010] The fourth aspect of the present disclosure provides a device for training a neural network, comprising one or more processors and a memory storing a program. The program comprises instructions, and when the instructions are executed by the processor, the device executes the above-mentioned method for training a neural network.

[0011] A fifth aspect of the present disclosure proposes a computer-readable storage medium storing a program, wherein the program includes instructions, which, when executed by one or more processors of a computing device, cause the computing device to execute the above-mentioned method of predicting collision time based on an image.

[0012] Based on the above disclosure, since the above-mentioned method for predicting collision time based on an image includes encoding an image to obtain a first feature image and a second feature image, then extracting a first feature image block and a second feature image block from the first feature image and the second feature image according to a plurality of scale ratios, and performing similarity calculation on the first feature image block and the second feature image block corresponding to each scale ratio to obtain a similarity score, and then screening out at least one scale ratio according to the similarity score to calculate the predicted value of the collision time of the target object. Therefore, the method disclosed in the present invention can calculate the predicted value of the collision time by screening out an appropriate scale ratio through encoding the image and the similarity calculation result with low computational overhead, thereby improving the accuracy of the predicted value of the collision time. The method disclosed in the present invention has good practical applicability and can optimize the path planning and safety of the autonomous driving system on the basis of low computational overhead. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] The accompanying drawings exemplarily illustrate the embodiments and constitute a part of the specification, and together with the text description of the specification, are used to explain the exemplary implementation of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention, and for those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative work. In all drawings, the same reference numerals refer to similar but not necessarily identical elements.

[0014] Figure 1 is a schematic diagram showing a vehicle in which the various techniques disclosed herein may be implemented;

[0015] Figure 2 is a flowchart showing a method for predicting collision time based on an image according to an embodiment of the present disclosure;

[0016] Figure 3 is a flowchart showing a method for training a neural network according to an embodiment of the present disclosure;

[0017] Figure 4 is a schematic diagram showing a process of predicting collision time based on an image according to an embodiment of the present disclosure;

[0018] Figure 5 is a schematic diagram showing a computing device according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0019] In order to make the purpose, technical scheme and advantages of the present invention clearer, the present invention will be further described in detail below with reference to the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0020] In this disclosure, the term "plurality" means two or more, unless otherwise specified. In this disclosure, the term "and / or" describes the association relationship of associated objects, covering any one of the listed objects and all possible combinations. The character " / " generally indicates that the associated objects before and after are in an "or" relationship.

[0021] In the present disclosure, unless otherwise specified, the terms "first", "second", etc. are used to distinguish similar objects without intending to limit their positional relationship, temporal relationship, or importance relationship. It should be understood that the terms used in this way can be interchangeable under appropriate circumstances, so that the embodiments of the present invention described herein can be implemented in ways other than those illustrated or described herein.

[0022] In addition, the terms "comprises," "comprising," and "having," and any variations thereof, are intended to cover non-exclusive inclusions. For example, a process, method, system, product, or apparatus that includes a series of steps or elements is not necessarily limited to those steps or elements explicitly listed, but may include other steps or elements not explicitly listed or inherent to such process, method, system, product, or apparatus.

[0023] Figure 1 is a schematic diagram illustrating a vehicle in which the various techniques disclosed herein may be implemented.

[0024] Please refer to Figure 1 , the vehicle 100 may be a car, truck, motorcycle, bus, boat, airplane, helicopter, lawnmower, excavator, snowmobile, aircraft, recreational vehicle, amusement park vehicle, farm equipment, construction equipment, tram, golf cart, train, trolley bus, or other vehicle. The vehicle 100 may be operated in full or in part in an autonomous driving mode. The vehicle 100 may control itself in the autonomous driving mode, for example, the vehicle 100 may determine the current state of the vehicle and the current state of the environment in which the vehicle is located, determine the predicted behavior of at least one other vehicle in the environment, determine a confidence level corresponding to the likelihood that the at least one other vehicle will perform the predicted behavior, and control the vehicle 100 itself based on the determined information. When in the autonomous driving mode, the vehicle 100 may operate without human interaction.

[0025] The vehicle 100 may include various vehicle systems, such as a drive system 142, a sensor system 144, a control system 146, a user interface system 148, a computing system 150, and a communication system 152. The vehicle 100 may include more or fewer systems, and each system may include multiple units. Further, each system and unit of the vehicle 100 may be interconnected. For example, the computing system 150 may be able to communicate data with one or more of the drive system 142, the sensor system 144, the control system 146, the user interface system 148, and the communication system 152. Thus, one or more of the described functions of the vehicle 100 may be divided into additional functional components or physical components, or combined into a smaller number of functional components or physical components. In further examples, the additional functional components or physical components may be increased to, for example, Figure 1 In the example shown.

[0026] The drive system 142 may include a plurality of operable components (or units) that provide kinetic energy to the vehicle 100. In some embodiments, the drive system 142 may include an engine or motor, wheels, a transmission, an electronic system, and a power source (or power source). The engine or motor may be any combination of an internal combustion engine, an electric motor, a steam engine, a fuel cell engine, a propane engine, or other forms of engines or motors. In some embodiments, the engine may convert a power source into mechanical energy. In some embodiments, the drive system 142 may include a plurality of engines or motors. For example, a hybrid vehicle may include a gasoline engine and an electric motor, and may also include other situations.

[0027] The wheels of the vehicle 100 may be standard wheels. The wheels of the vehicle 100 may be wheels of various types, including single, double, triple, or quad wheels, such as quad wheels on a car or truck. Other numbers of wheels are also possible, such as six or more wheels. One or more wheels of the vehicle 100 may be operated to rotate in a different direction than the other wheels. The wheel may be at least one wheel fixedly connected to a transmission. The wheel may include a combination of metal and rubber, or a combination of other materials. The transmission may include a unit operable to transmit mechanical power from the engine to the wheels. For this purpose, the transmission may include a gearbox, a clutch, a differential gear, and a drive shaft. The transmission may also include other units. The drive shaft may include one or more axles that match the wheels. The electronic system may include a unit for transmitting or controlling electronic signals of the vehicle 100. These electronic signals may be used to activate multiple lights, multiple servo mechanisms, multiple motors, and other electronic drive or control devices in the vehicle 100. The power source may be an energy source that provides power to the engine or motor in whole or in part. That is, the engine or motor is capable of converting the power source into mechanical energy. Exemplarily, the power source may include gasoline, oil, petroleum-based fuels, propane, other compressed gas fuels, ethanol, fuel cells, solar panels, batteries, and other electrical energy sources. The power source may additionally or alternatively include any combination of a fuel tank, a battery, a capacitor, or a flywheel. The power source may also provide energy for other systems of the vehicle 100.

[0028] The sensor system 144 may include a plurality of sensors for sensing information about the environment and conditions of the vehicle 100. For example, the sensor system 144 may include an inertial measurement unit (IMU), a global navigation satellite system (GNSS) transceiver (e.g., a global positioning system (GPS) transceiver), a radio detection and ranging device (RADAR, referred to as radar), a laser detection and ranging system (LiDAR, referred to as laser radar), an acoustic sensor, an ultrasonic sensor, and an image capture device (e.g., a camera). The sensor system 144 may include a plurality of sensors for monitoring the vehicle 100 (e.g., an oxygen (O2) monitor, a fuel gauge sensor, an engine oil pressure sensor, and temperature, humidity, pressure sensors, etc.). Of course, in other embodiments, other sensors may also be configured. One or more sensors included in the sensor system 144 may be driven individually or collectively to update the position, orientation, or both of the one or more sensors.

[0029] The IMU may include a combination of sensors (e.g., accelerometers and gyroscopes) for sensing attitude changes (e.g., position changes and direction changes) of the vehicle 100 based on inertial acceleration. The GPS transceiver may be any sensor for estimating the geographic location of the vehicle 100. For this purpose, the GPS transceiver may include a receiver / transmitter to provide location information of the vehicle 100 relative to the earth. It should be noted that GPS is an example of a global navigation satellite system, so in some embodiments, the GPS transceiver may be replaced by a Beidou satellite navigation system transceiver or a Galileo satellite navigation system transceiver. The radar unit may use radio signals to sense objects in the environment where the vehicle 100 is located. In some embodiments, in addition to sensing objects, the radar unit may also be used to sense the speed and direction of objects approaching the vehicle 100. The LiDAR unit may be any sensor that uses lasers to sense objects in the environment where the vehicle 100 is located. In some embodiments, the LiDAR unit may include a laser source, a laser scanner, and a detector. The LiDAR unit is used to operate in a continuous (e.g., using heterodyne detection) or discontinuous detection mode. The image capture device may include a device for capturing multiple images of the environment where the vehicle 100 is located. An example of an image capture device is a camera, which may be a still image camera or a motion video camera.

[0030] The control system 146 is used to control the operation of the vehicle 100 and its components (or units). Accordingly, the control system 146 may include various units, such as a steering unit, a power control unit, a braking unit, and a navigation unit.

[0031] The steering unit can be a combination of machines that adjust the direction of travel of the vehicle 100. A power control unit (e.g., a throttle) can be used, for example, to control the speed of the engine, and thus the speed of the vehicle 100. The braking unit can include a combination of machines for decelerating the vehicle 100. The braking unit can use friction to decelerate the vehicle in a standard manner. In other embodiments, the braking unit can convert the kinetic energy of the wheels into electric current. The braking unit can also take other forms. The navigation unit can be any system that determines a driving path or route for the vehicle 100. The navigation unit can also dynamically update the driving path while the vehicle 100 is moving. The control system 146 can also additionally or optionally include other components (or units) that are not shown or described.

[0032] The user interface system 148 may be used to allow interaction between the vehicle 100 and external sensors, other vehicles, other computer systems, and / or users of the vehicle 100. For example, the user interface system 148 may include a standard visual display device (e.g., a plasma display, a liquid crystal display (LCD), a touch screen display, a head mounted display, or other similar display), a speaker or other audio output device, a microphone or other audio input device. For example, the user interface system 148 may also include a navigation interface and an interface for controlling the interior environment of the vehicle 100 (e.g., temperature, air conditioning, etc.).

[0033] The communication system 152 can provide a way for the vehicle 100 to communicate with one or more devices or other vehicles in the vicinity. In an exemplary embodiment, the communication system 152 can communicate with one or more devices directly or through a communication network. The communication system 152 can be, for example, a wireless communication system. For example, the communication system can use 3G cellular communication (such as CDMA, EVDO, GSM / GPRS) or 4G cellular communication (such as WiMAX or LTE), and can also use 5G cellular communication. Optionally, the communication system can communicate with a wireless local area network (WLAN) (for example, using In some embodiments, the communication system 152 can communicate directly with one or more devices or other surrounding vehicles, for example, using infrared. Or ZIGBEE. Other wireless protocols, such as various vehicle-mounted communication systems, are also within the scope of the present disclosure. For example, the communication system may include one or more dedicated short-range communication (DSRC) devices, V2V devices, or V2X devices that communicate data with vehicles and / or roadside stations in public or private.

[0034] The computing system 150 can control some or all functions of the vehicle 100. The autonomous driving control unit in the computing system 150 can be used to identify, evaluate, and avoid or overcome potential obstacles in the environment of the vehicle 100. Generally, the autonomous driving control unit can be used to control the vehicle 100 without a driver, or to assist the driver in controlling the vehicle. In some embodiments, the autonomous driving control unit is used to combine data from sensors, such as data from a GPS transceiver, radar data, LiDAR data, camera data, and data from other vehicle systems to determine the driving path or trajectory of the vehicle 100. The autonomous driving control unit can be activated to enable the vehicle 100 to be driven in an autonomous driving mode.

[0035] The computing system 150 may include at least one processor (which may include at least one microprocessor) that executes processing instructions (i.e., machine executable instructions) stored in a non-volatile computer-readable medium (e.g., a data storage device or memory). The computing system 150 may also be a plurality of computing devices that control components or systems of the vehicle 100 in a distributed manner. In some embodiments, the memory may contain processing instructions (e.g., program logic) that are executed by the processor to implement various functions of the vehicle 100. In some embodiments, the computing system 150 is capable of data communication with the drive system 142, the sensor system 144, the control system 146, the user interface system 148, and / or the communication system 152. The interface in the computing system is used to facilitate data communication between the computing system 150 and the drive system 142, the sensor system 144, the control system 146, the user interface system 148, and the communication system 152.

[0036] The memory may also include other instructions, including instructions for data transmission, instructions for data reception, instructions for interaction, or instructions for controlling the drive system 142 , the sensor system 144 , or the control system 146 or the user interface system 148 .

[0037] In addition to storing processing instructions, the memory may store a variety of information or data, such as image processing parameters, road maps, and path information, which may be used by the vehicle 100 and the computing system 150 during operation of the vehicle 100 in an autonomous, semi-autonomous, and / or manual mode.

[0038] Although the autonomous driving control unit is shown as being separate from the processor and memory, it should be understood that in some embodiments, some or all of the functions of the autonomous driving control unit may be implemented using program code instructions residing in one or more memories (or data storage devices) and executed by one or more processors, and the autonomous driving control unit may in some cases be implemented using the same processor and / or memory (or data storage device). In some embodiments, the autonomous driving control unit may be implemented at least in part using various dedicated circuit logic, various processors, various field programmable gate arrays ("FPGA"), various application specific integrated circuits ("ASIC"), various real-time controllers, and hardware.

[0039] The computing system 150 may control functions of the vehicle 100 based on input received from various vehicle systems (e.g., the drive system 142, the sensor system 144, and the control system 146), or input received from the user interface system 148. For example, the computing system 150 may use input from the control system 146 to control a steering unit to avoid an obstacle detected by the sensor system 144. In some embodiments, the computing system 150 may be used to control various aspects of the vehicle 100 and its systems.

[0040] Although Figure 1 1 shows various components (or units) integrated into the vehicle 100, one or more of which may be mounted on the vehicle 100 or separately associated with the vehicle 100. For example, the computing system 150 may exist partially or completely independently of the vehicle 100. Thus, the vehicle 100 may exist in the form of separate or integrated equipment units. The equipment units constituting the vehicle 100 may communicate with each other in the form of wired or wireless communications. In some embodiments, additional components or units may be added to each system or one or more components or units may be removed from the system (for example, Figure 1 LiDAR or radar as shown).

[0041] The disclosed embodiment proposes a method and related device for predicting collision time based on an image, which is used to improve the accuracy of TTC between the vehicle 100 and a target object (such as a vehicle in front or an obstacle). The disclosed embodiment can solve the TTC with large errors caused by directly calculating the collision time through the size ratio of the detection box / bounding box, and provide a more accurate TTC to optimize the path planning and safety of the autonomous driving system.

[0042] Figure 2 FIG. 2 is a flowchart showing a method 200 for predicting collision time based on an image according to an embodiment of the present disclosure.

[0043] Please refer to Figure 2, the method 200 for predicting collision time based on images of the embodiment of the present disclosure can be applied to the vehicle 100. In step 202, the vehicle 100 or the computing system 150 collects multiple frames of images according to the target object, wherein the multiple frames of images include a first image corresponding to the first frame and a second image corresponding to the second frame, and the second frame is after the first frame in time sequence. In step 204, the vehicle 100 or the computing system 150 obtains a first bounding box and a second bounding box, wherein the first bounding box corresponds to the target object in the first image, and the second bounding box corresponds to the target object in the second image. In step 206, the vehicle 100 or the computing system 150 encodes the first image and the second image respectively to obtain a first feature image and a second feature image. In step 208, the vehicle 100 or the computing system 150 scales the second bounding box according to each of a plurality of scale ratios to obtain a third bounding box. In step 210, the vehicle 100 or the computing system 150 extracts a first feature image block from the first feature image through the third bounding box. In step 212, the vehicle 100 or the computing system 150 extracts the second feature image block from the second feature image through the second bounding box. In step 214, the vehicle 100 or the computing system 150 performs similarity calculation on the first feature image block and the second feature image block corresponding to each scale ratio to obtain a similarity score corresponding to each scale ratio. In step 216, the vehicle 100 or the computing system 150 selects at least one scale ratio according to the similarity score corresponding to each scale ratio to calculate the predicted value of the collision time of the target object.

[0044] The method 200 of the embodiment of the present disclosure first encodes two frames of images in time sequence to obtain a feature image, then scales the bounding box according to a plurality of different scale ratios to crop the two frames of feature images to obtain feature image blocks of the two frames, which are used to perform similarity calculations and obtain similarity scores corresponding to each scale ratio, and finally the corresponding scale ratios can be screened out according to the higher similarity scores to calculate the collision time.

[0045] In some embodiments, for example in step 206, the vehicle 100 or the computing system 150 encodes the first image and the second image respectively to obtain the first feature image and the second feature image, including inputting the first image and the second image into a neural network to obtain the first feature image and the second feature image.

[0046] In some embodiments, for example, in step 214, the vehicle 100 or the computing system 150 performs similarity calculation on the first feature image block and the second feature image block corresponding to each scale ratio to obtain a similarity score corresponding to each scale ratio, including the following process:

[0047] The vehicle 100 or the computing system 150 performs similarity calculation on the first feature image block and the second feature image block corresponding to each scale ratio to obtain a similarity value corresponding to each scale ratio, and the vehicle 100 or the computing system 150 converts the similarity value corresponding to each scale ratio into a similarity score.

[0048] In some embodiments, the vehicle 100 or the computing system 150 performs similarity calculation by calculating the mean of the aligned cosine similarity, the mean square error (MSE) of the pixel values, the structural similarity index (SSIM), the peak signal-to-noise ratio (PSNR), the normalized cross-correlation (NCC) or the Manhattan distance (also called L1 norm) between the first feature image block and the second feature image, but is not limited to this.

[0049] Specifically, calculating the mean of the aligned cosine similarity is a common method for measuring image differences. The mean, for example, falls between -1 and 1.

[0050] MSE is also a common way to measure image differences. It simply calculates the average of the squared differences between corresponding pixels in two images.

[0051] SSIM is a more complex measurement method that takes into account the structural information of the image. SSIM not only compares the pixel values ​​of two images, but also compares their brightness, contrast and structural similarity within a local window. The value of SSIM is, for example, between -1 and 1. The closer the value of SSIM is to 1, the more similar the images are.

[0052] PSNR is an indicator to measure the quality of image compression and can also be used to compare the difference between two images. PSNR is usually used to measure the quality of image reconstruction. The higher the PSNR value, the better the image quality and the smaller the error.

[0053] NCC is a metric used to compare two images. It is often used in pattern recognition and image processing to find the location of a small image (called a template) in a large image. NCC can handle image brightness and contrast changes well. The value of NCC is between -1 and 1. The closer the value of NCC is to 1, the more similar the images are.

[0054] Manhattan distance / L1 norm calculates the sum of the absolute values ​​of the corresponding pixel differences between two images.

[0055] In some embodiments, the vehicle 100 or the computing system 150 converts the similarity value corresponding to each scale ratio into a similarity score, including inputting the similarity value corresponding to each scale ratio into a neural network to obtain a similarity score corresponding to each scale ratio.

[0056] In some embodiments, the vehicle 100 or the computing system 150 converts the similarity value corresponding to each scale ratio into a similarity score, including obtaining the similarity score corresponding to each scale ratio through a preset correspondence between the similarity value and the similarity score.

[0057] In some embodiments, for example, in step 214, the vehicle 100 or the computing system 150 performs similarity calculation on the first feature image block and the second feature image block corresponding to each scale ratio to obtain a similarity score corresponding to each scale ratio, including the following process:

[0058] For a scale ratio: the vehicle 100 or the computing system 150 calculates the similarity between the first feature image block and the second feature image block to obtain a similarity map, wherein the similarity map includes multiple regions, and each of the multiple regions has a positional similarity value; the vehicle 100 or the computing system 150 calculates the feature image block similarity value based on multiple positional similarity values ​​of the multiple regions to obtain a similarity score.

[0059] Furthermore, in some embodiments, the vehicle 100 or the computing system 150 averages a plurality of alignment similarity values ​​to obtain a feature image block similarity value.

[0060] In some embodiments, the vehicle 100 or the computing system 150 inputs the feature image block similarity value into a neural network to obtain a similarity score.

[0061] In some embodiments, in response to the target object in the first image being partially blocked, the vehicle 100 or the computing system 150 forms a mask with multiple first feature map pixels in the first feature image block, or, in response to the target object in the second image being partially blocked, the vehicle 100 or the computing system 150 forms a mask with multiple second feature map pixels in the second feature image block. Further, the vehicle 100 or the computing system 150 performs similarity calculation on the first feature image block and the second feature image block to obtain a similarity map, including the following process:

[0062] In response to the plurality of first feature map pixels or the plurality of second feature map pixels having a mask, the vehicle 100 or the computing system 150 forms a mask in a plurality of first sub-regions among a plurality of regions in the similarity map, wherein a plurality of regions in the similarity map that are not masked are a plurality of second sub-regions. Next, the vehicle 100 or the computing system 150 calculates a feature image block similarity value based on a plurality of alignment similarity values ​​of the plurality of regions to obtain a similarity score.

[0063] In some embodiments, the vehicle 100 or the computing system 150 calculates the feature image block similarity value based on the multiple alignment similarity values ​​of the multiple regions, including averaging the multiple alignment similarity values ​​of the multiple second sub-regions to obtain the feature image block similarity value. Specifically, in these embodiments, when the target object in the first image or the second image is partially blocked, the process of calculating the feature image block similarity value can avoid the blocked part and obtain a more accurate calculation result.

[0064] In some embodiments, for example, in step 216, the vehicle 100 or the computing system 150 selects at least one scale ratio according to the similarity score corresponding to each scale ratio to calculate the predicted value of the collision time of the target object, including the vehicle 100 or the computing system 150 calculating the predicted value of the collision time of the target object according to at least one scale ratio and the corresponding similarity score.

[0065] In some embodiments, for example, in step 216, the vehicle 100 or the computing system 150 selects at least one scale ratio according to the similarity score corresponding to each scale ratio to calculate the predicted value of the collision time of the target object, including the vehicle 100 or the computing system 150 performing a weighted average operation on the collision time calculated by at least one scale ratio to obtain the predicted value of the collision time.

[0066] In some embodiments, for example, in step 210, the step of the vehicle 100 or the computing system 150 extracting the first feature image block from the first feature image through the third bounding box includes the vehicle 100 or the computing system 150 extracting the first feature image block from the first feature image through the third bounding box based on the center position of the first bounding box.

[0067] In some embodiments, for example, in step 210, the vehicle 100 or the computing system 150 extracts a first feature image block from the first feature image through a third bounding box, including: based on the center position of the first bounding box, extracting a central feature image block from the first feature image through the third bounding box; adjusting the center position of the first bounding box to at least one other position on the first image; for each other position, extracting other feature image blocks from the first feature image through the third bounding box to obtain at least one other feature image block; and respectively calculating the similarity between the central feature image block and at least one other feature image block and the second feature image block to filter out the central feature image block and at least one other feature image block to obtain the first feature image block.

[0068] In some embodiments, for example, in step 210, the vehicle 100 or the computing system 150 respectively calculates the similarity between the central feature image block and at least one other feature image block and the second feature image block, so as to screen the central feature image block and the at least one other feature image block to obtain the first feature image block, including: calculating the mean of the alignment cosine similarity between the central feature image block and the at least one other feature image block and the second feature image block; and selecting the central feature image block or one of the at least one other feature image blocks corresponding to the largest mean of the alignment cosine similarity from the calculated mean of the alignment cosine similarity as the first feature image block.

[0069] In some embodiments, the vehicle 100 or the computing system 150 adjusts the size of the first feature image block to be the same as the size of the second feature image block.

[0070] In some embodiments, the vehicle 100 or the computing system 150 resizes the first feature image block to be the same as the second feature image block, including scaling the width and height of the first feature image block to be the same as the width and height of the second feature image block.

[0071] Please refer to Figure 3 The method 300 for training a neural network according to an embodiment of the present disclosure may be applied to a computing system that may be implemented in the form of a computing device 500, and the method 300 for training a neural network may be used to train at least one of the following: a neural network for encoding the first image and the second image in the above embodiment, a neural network for converting the similarity value corresponding to each scale ratio into a similarity score in the above embodiment, and other neural networks used in the implementation of the method 200 for predicting collision time based on an image in the above embodiment.

[0072] In step 302, the computing system obtains a first bounding box and a second bounding box, wherein the first bounding box corresponds to a target object in a first image, and the second bounding box corresponds to a target object in a second image, the first image corresponds to a first frame, and the second image corresponds to a second frame, and the second frame is after the first frame in time sequence. In step 304, the computing system inputs the first image and the second image into a neural network to obtain a first feature image and a second feature image output by a prediction model. In step 306, the computing system scales the second bounding box according to each of a plurality of scale ratios to obtain a third bounding box. In step 308, the computing system extracts a first feature image block from the first feature image through the third bounding box. In step 310, the computing system extracts a second feature image block from the second feature image through the second bounding box. In step 312, the computing system performs similarity calculation on the first feature image block and the second feature image block corresponding to each scale ratio to obtain a similarity score corresponding to each scale ratio, wherein the similarity score is used to indicate whether the scale ratio corresponding to the similarity score is close to the true scale ratio. In step 314, the computing system obtains the sample similarity score corresponding to each scale ratio according to the true scale ratio. In step 316, for each scale ratio, the computing system compares the similarity score and the sample similarity score to generate a comparison result. In step 318, the neural network updates the prediction model according to the comparison result to output a new similarity score, and the updated prediction model is used to make the new similarity score more responsive to whether the corresponding scale ratio is close to the true scale ratio.

[0073] The method 300 of the disclosed embodiment first uses a prediction model to encode two frames of images in time sequence to obtain a feature image, then scales the bounding box according to a plurality of different scale ratios to crop the feature image blocks of the two frames in the two feature images to perform similarity calculation and obtain a similarity score corresponding to each scale ratio, and obtains a sample similarity score corresponding to each scale ratio according to the true scale ratio, and finally compares the similarity score with the sample similarity score for each scale ratio to update the prediction model according to the comparison result, re-encodes the two frames of images with the updated prediction model to obtain an updated feature image, and repeats the above process to crop the feature image blocks of the two frames to perform similarity calculation and obtain a similarity score, thereby updating the prediction model according to the comparison result of the similarity score with the sample similarity score.

[0074] In some embodiments, for example, in step 312, the computing system performs similarity calculation on the first feature image block and the second feature image block corresponding to each scale ratio to obtain a similarity score corresponding to each scale ratio, including the following process:

[0075] The computing system performs similarity calculation on the first feature image block and the second feature image block corresponding to each scale ratio to obtain a similarity value corresponding to each scale ratio, and the computing system inputs the similarity value corresponding to each scale ratio into the neural network to obtain a similarity score corresponding to each scale ratio output by the prediction model.

[0076] In some embodiments, for example in step 318, the neural network updates the prediction model based on the comparison result to output a new similarity score, including the neural network updating the parameters in the prediction model based on the comparison result and through the cross entropy loss function so that the output value of the cross entropy loss function can be minimized.

[0077] In some embodiments, the true scale ratio is the scale ratio obtained by using a laser radar to obtain a true collision time for the target object and calculated using the true collision time.

[0078] In some embodiments, for example, in step 314, the step of obtaining the sample similarity score corresponding to each scale ratio according to the actual scale ratio by the computing system includes the following process:

[0079] The computing system obtains the initial sample similarity score corresponding to each scale ratio according to the real scale ratio; in response to the initial sample similarity score being greater than a preset threshold, the computing system sets the initial sample similarity score to a first value; and in response to the initial sample similarity score being less than or equal to the preset threshold, the computing system sets the initial sample similarity score to a second value, wherein the similarity score is a value falling within the range defined by the first value and the second value. Specifically, the first value may be 1, and the second value may be 0. Alternatively, the first value and the second value may be set to other values ​​according to actual needs.

[0080] In some embodiments, for example, in step 314, the computing system obtains the initial sample similarity score corresponding to each scale ratio based on the true scale ratio, including the computing system obtaining the initial sample similarity score corresponding to each scale ratio based on the difference between the numerical value of each scale ratio and the numerical value of the true scale ratio.

[0081] In some embodiments, for example, in step 314, the step of obtaining the sample similarity score corresponding to each scale ratio according to the actual scale ratio by the computing system includes the following process:

[0082] The computing system determines the scale ratio closest to the real scale ratio among the multiple scale ratios as the approximate scale ratio; the computing system determines the multiple scale ratios other than the approximate scale ratio among the multiple scale ratios as multiple other scale ratios; the computing system sets the sample similarity score corresponding to the approximate scale ratio to a third value; and the computing system sets the sample similarity score of each of the multiple other scale ratios to a fourth value, wherein the similarity score is a value falling within the range defined by the third value and the fourth value. Specifically, the third value may be 1, and the fourth value may be 0. Alternatively, the third value and the fourth value may be set to other values ​​according to actual needs.

[0083] The specific process of predicting TTC in the embodiment of the present disclosure is described in detail as follows.

[0084] Figure 4 is a schematic diagram showing a process of predicting collision time based on an image according to an embodiment of the present disclosure.

[0085] like Figure 4 As shown, in the two frames of images F0 and F1 at time t0 and t1, the vehicle 100 or the computing system 150 can track a specific vehicle and mark a bounding box for it to calculate the TTC between the vehicle 100 and the specific vehicle. The representation of the bounding box can be b=[x, y, w, h], where x, y represent the center coordinates of the bounding box, and w, h represent the width and height of the bounding box, respectively. In the embodiment of the present disclosure, the two frames of images F0 and F1 can be referred to as the reference image F0 and the target image F1. The bounding box of the reference image F0 is represented by b0=[x0, y0, w0, h0], where x0, y0 represent the center coordinates of the bounding box of the image F0, and w0, h0 represent the width and height of the bounding box of the image F0, respectively. The bounding box of the target image F1 is represented by b1=[x1, y1, w1, h1], where x1, y1 represent the center coordinates of the bounding box of the image F1, and w1, h1 represent the width and height of the bounding box of the image F1, respectively. Specifically, the neural network 400 can be used to predict the model (e.g. Figure 4 The backbone network and convolutional layer shown in FIG. 1 extract features from the input reference image F0 and the target image F1 to generate feature images F′0 and F′1. For the reference image F0, n different scale ratios α1, α2, ..., α n , to get a set of bounding box scaling boxes Where the bounding box scale box

[0086] For the i-th scale, the set of boxes can be scaled by the bounding box The corresponding bounding box scale box in The feature image F′0 is cropped, and the size of the cropped feature image block is adjusted to a width W and a height H (for example, adjusted to a width w1 and a height h1 of a bounding box of the target image F1, but not limited thereto). The adjusted feature image block is represented as Similarly, the feature image F′1 can be cropped by the bounding box b1 to obtain the feature image block Afterwards, the feature image blocks can be calculated and The similarity between them is obtained by calculating n similarity values ​​(i.e. the similarity value corresponding to each scale ratio). Figure 4 In the above example, the similarity between feature image blocks can be calculated by Specifically, the similarity between feature image blocks can be represented by the mean of the aligned cosine similarity. However, in addition to cosine similarity, similarity can also be calculated by mean square error (MSE), structural similarity index (SSIM), peak signal-to-noise ratio (PSNR), normalized cross correlation (NCC) or Manhattan distance / L1 norm.

[0087] It is worth noting that similarity calculation will produce a similarity graph, which contains multiple alignment similarity values ​​(such as Figure 4 As shown). In this embodiment, multiple alignment similarity values ​​can be averaged to obtain the feature image block similarity value. In addition, in some embodiments, whether the two feature image blocks need to be strictly aligned and the similarity map needs to be calculated depends on the similarity calculation method used. For example, the cosine similarity method, the MSE method and the L1 norm method require the two feature image blocks to be strictly aligned. The SSIM method and the NCC method do not require the two feature image blocks to be strictly aligned.

[0088] In addition, in order to reduce the error caused by the inaccurate center of the bounding box b0 of the reference image F0, for the i-th scale ratio, the center point coordinates (x0, y0) can be offset (for example, taking (x0, y0) as the reference point, the center point is offset by a predetermined number of pixels in the upward, downward, left, and right directions, respectively). Then, the image blocks of the feature image F′0 are cropped according to the offset center points. Then, the size of the cropped feature image blocks is adjusted to the width w1 and height h1 of the bounding box of the target image F1). Afterwards, the feature image blocks and the feature image blocks are calculated from these cropped and resized feature image blocks. The similarity between them (for example, the mean of the aligned cosine similarity can be calculated, or other similarity calculation methods can be used). Then, the feature image block corresponding to the maximum mean of aligned cosine similarity among these feature image blocks is taken to determine the feature image block at a specific scale ratio.

[0089] For example, for each scale ratio α i , perform feature image block and Before calculating the similarity of (x0, y0) can be used as the reference point (center position), and the center point can be shifted in the directions of up, down, left, and right to form 4 positions. And (x0, y0) can be used as the reference point (center position), and the center point can be shifted in the directions of upper left, upper right, lower left, and lower right to form another 4 positions. At this time, there are 9 points including the center position and 8 other positions, and these 9 points are arranged in a neat 3x3 shape, for example. Then, the feature image is cut according to these 9 points to obtain 9 feature image blocks, and the feature image blocks and the feature image blocks are calculated based on these feature image blocks. The cosine similarity between the two is used to determine the feature image blocks at this scale. Specifically, the feature image block corresponding to the maximum mean value of the cosine similarity among the nine feature image blocks can be taken to determine the feature image block at this specific scale ratio.

[0090] In this embodiment, the neural network 400 may pass the n similarity values ​​calculated from the n scale ratios through the prediction model to obtain a similarity score for each scale. An example implementation of the prediction model may be Figure 4 The fully connected layer (FC) shown.

[0091] In some embodiments, in the process of updating the prediction model, the TTC value may be calculated by taking the scale ratio corresponding to the largest one or several highest similarity scores.

[0092] For example, we can take the k scale ratios α0, α1, ..., α with the highest similarity scores. k-1 and their scores s0, s1, ..., s k-1 , the predicted TTC is obtained through weighted average operation.

[0093]

[0094] In some embodiments, the k selected scale ratios with the highest similarity scores may be weighted averaged according to the scores corresponding to the k selected scale ratios with the highest similarity scores to obtain the scale ratio after weighted average calculation. Then, TTC may be calculated according to the scale ratio after weighted average calculation.

[0095] In addition, in this embodiment, the neural network can be trained by providing a reference true value (ground truth, GT), and the prediction model of the neural network can be updated to output an updated similarity score. In the process of updating the prediction model, the similarity score of each scale can be supervised by binary classification according to the real scale obtained by the laser radar, and the prediction model can be updated by returning the binary classification cross entropy loss. Specifically, the process of returning the binary classification cross entropy loss can be by updating the parameters in the prediction model so that the output value of the cross entropy loss function can be minimized.

[0096] In the process of updating the prediction model, the training goal can be set as follows: make each scale ratio corresponding to the similarity score output by the prediction model as close as possible to the actual scale ratio obtained by the lidar.

[0097] It is worth noting that the process of updating the prediction model includes: the prediction model re-encodes the two frames of images F0 and F1 input into the neural network 400, and then re-outputs the feature images F′0 and F′1 (for example, re-captures features from the two frames of images F0 and F1 through the backbone network and the convolutional layer to generate updated feature images F′0 and F′1), and again performs the similarity calculation between the above-mentioned feature image blocks to obtain an updated similarity score, thereby achieving the above-mentioned training goal.

[0098] In this embodiment, the binary supervision means that for the similarity score corresponding to each scale ratio, the similarity score can be set to 0 or 1. Among them, 1 means that the scale ratio corresponding to the similarity score is the same as the real scale ratio calculated by the laser radar, and 0 means it is different. In detail, the binary supervision process of the embodiment of the present disclosure includes: predicting the similarity score of each scale according to the real scale ratio (for example, the initial sample similarity score mentioned above). Finally, this similarity score (such as the initial sample similarity score mentioned above) is set to 1 or 0.

[0099] An example is given below. If n=3 (i.e., three scale ratios are taken), assume that the three scale ratios are 0.8, 0.9, and 1.0. If the true scale ratio obtained by the laser radar through direct measurement of TTC conversion is 0.95. Since the true scale ratio (0.95) does not exactly match any of the above scale ratios, the similarity score used for training can be set through the following two implementations.

[0100] In one embodiment, an "approximate scale ratio" that is close to the true scale ratio is selected by rounding, and the initial sample similarity score of the "approximate scale ratio" is set to 1, and the initial sample similarity scores of other scale ratios are set to 0. For example, the "approximate scale ratio" of 0.95 is rounded to 1.0 to find a suitable scale ratio, so that the sample similarity scores of the three scale ratios are set as follows: scale ratio 0.8 is 0 points; scale ratio 0.9 is 0 points; scale ratio 1.0 is 1.0 points.

[0101] In another embodiment, the closer the scale ratio is to the true scale ratio, the higher the initial sample similarity score of the scale ratio is set. For example, if the true scale ratio is 0.95, the initial sample similarity scores of the scale ratios 0.9 and 1.0 can be set to 0.5 points respectively. Since the scale ratio 0.8 is farther, it can be set to 0 points. Afterwards, it can be determined whether the above 0.5 points are greater than the threshold, and then the sample similarity score is adjusted to 1 or 0. For example, the initial sample similarity score of 1.0 with a true scale ratio of 0.95 can be divided equally or proportionally to the scores of the scale ratios 0.9 and 1.0. Specifically, an initial sample similarity score can be generated: 0 points for a scale ratio of 0.8; 0.5 points for a scale ratio of 0.9; 0.5 points for a scale ratio of 1.0. Then, the initial sample similarity score can be reset or adjusted according to the preset threshold. Assuming the preset threshold is 0.5, the sample similarity score can be adjusted as follows: scale ratio 0.8 is 0 points; scale ratio 0.9 is 1.0 points; scale ratio 1.0 is 1.0 points. Assuming the preset threshold is 0.4, the sample similarity can be adjusted as follows: scale ratio 0.8 is 0 points; scale ratio 0.9 is 0 points; scale ratio 1.0 is 0 points.

[0102] Table 1 below is a comparison between the traditional bounding box-based scale ratio method, the traditional single target tracking (SOT) collision time prediction method, and the image-based collision time prediction method of the disclosed embodiment. The numerical values ​​corresponding to each method in Table 1 represent the relative collision time error (Relative TTC Error, RTE), that is, the ratio obtained by dividing the difference between the true TTC value and the predicted TTC value by the true TTC value. Specifically, Table 1 compares the RTE of the image prediction method of the disclosed embodiment with the traditional bounding box scale ratio-based method, and with the RTE of the traditional single target tracking method under different TTC ranges.

[0103] Specifically, when using the traditional bounding box scale ratio-based method, the scale ratio of the target object between two moments can be obtained by the ratio of the areas of the bounding boxes at the two moments. When using the traditional single target tracking method, when the bounding box of the object at time t0 is given, the bounding box tracked at time t1 can be obtained according to the target tracking model. Afterwards, the scale ratio of the target object between the two moments can also be calculated by the ratio of the areas of the bounding boxes at the two moments.

[0104] Table 1

[0105]

[0106] As shown in Table 1, according to the method for predicting collision time based on an image in an embodiment of the present disclosure, the RTE value obtained is smaller, so a more accurate TTC can be obtained. In other words, compared with the traditional method based on bounding box scale ratio or the traditional single target tracking method, the method for predicting collision time based on an image in an embodiment of the present disclosure can obtain a more accurate TTC.

[0107] Figure 5 is a schematic diagram showing a computing device according to an embodiment of the present disclosure.

[0108] Please refer to Figure 5 In some embodiments, the computing system 150 may be implemented in the form of a computing device 500 .

[0109] In some embodiments, within the computing device 500, the instruction set when executed and / or the processing logic when started can make the machine perform any one or more of the methods described and / or required herein. In alternative embodiments, the machine operates as a standalone device, or can be connected (e.g., networked) to other machines. In a networked deployment, the machine can operate as a server or client machine in a server-client network environment, or as a peer machine in a peer (or distributed) network environment. The machine can be a personal computer (PC), a laptop computer, a tablet computing system, a personal digital assistant (PDA), a cellular phone, a smart phone, a network application, a set-top box (STB), a network router, a switch or a bridge or any machine capable of executing an instruction set (in succession or otherwise) specifying the action to be taken by the machine or starting processing logic. Further, although only a single machine is illustrated, the term "machine" can also be understood to include any set of machines that individually or jointly execute an instruction set (or multiple instruction sets) to perform any one or more of the methods described and / or required herein.

[0110] The computing device 500 may include a data processor 502 (e.g., a system-on-chip (SoC), a general processing core, a graphics core, and optional other processing logic) and a memory 504 (e.g., memory) that may communicate with each other via a bus 506 or other data transmission system. The computing device 500 may also include various input / output (I / O) devices and / or interfaces 510, such as a touch screen display, an audio jack, a voice interface, and an optional network interface 512. In an example embodiment, the network interface 512 may include one or more radio transceivers configured to communicate with any one or more standard wireless and / or cellular protocols or access technologies (e.g., second generation (2G), 2.5 generation, third generation (3G), fourth generation (4G) and next generation radio access of cellular systems, global system for mobile communications (GSM), general packet radio service (GPRS), enhanced data GSM environment (EDGE), wideband code division multiple access (WCDMA), LTE, CDMA2000, WLAN, wireless router (WR) mesh, etc.). The network interface 512 may also be configured to communicate with various other wired and / or wireless communication protocols including TCP / IP, UDP, SIP, SMS, RTP, WAP, CDMA, TDMA, UMTS, UWB, WiFi, WiMax, In essence, network interface 512 may include or support virtually any wired and / or wireless communication and data processing mechanism by which information / data may be transmitted between computing device 500 and another computing or communication system via network 514.

[0111] The memory 504 may represent a machine-readable medium (or computer-readable storage medium) on which one or more instruction sets, software, firmware, or other processing logic (e.g., logic 508) are stored that implement any one or more of the methods or functions described and / or required herein. During execution by the computing device 500, the logic 508 or a portion thereof may also reside completely or at least partially within the processor 502. In this way, the memory 504 and the processor 502 may also constitute a machine-readable medium (or computer-readable storage medium). The logic 508 or a portion thereof may also be configured as a processing logic or logic, at least a portion of which is partially implemented in hardware. The logic 508 or a portion thereof may also be transmitted or received over a network 514 via a network interface 512. Although the machine-readable medium (or computer-readable storage medium) of an example embodiment may be a single medium, the term "machine-readable medium" (or computer-readable storage medium) should be understood to include a single non-transitory medium or multiple non-transitory media (e.g., a centralized or distributed database and / or associated cache and computing system) storing one or more instruction sets. The term "machine-readable medium" (or computer-readable storage medium) may also be understood to include any non-transitory medium that can store, encode, or carry an instruction set for execution by a machine and cause the machine to perform any one or more of the methods of the various embodiments, or that can store, encode, or carry a data structure utilized by or associated with such an instruction set. The term "machine-readable medium" (or computer-readable storage medium) may therefore be understood to include, but is not limited to, solid-state memories, optical media, and magnetic media.

[0112] The disclosed and other embodiments, modules, and functional operations described in this document may be implemented in digital electronic circuit systems, or in computer software, firmware or hardware (including the structures disclosed in this document and their structural equivalents), or in a combination of one or more of them. The disclosed and other embodiments may be implemented as one or more computer program products, that is, one or more modules of computer program instructions encoded on a computer-readable medium to be executed by a data processing device or to control the operation of the data processing device. The computer-readable medium may be a machine-readable storage device, a machine-readable storage substrate, a memory device, a material composition that affects a machine-readable propagation signal, or a combination of one or more of them. The term "data processing device" covers all devices, equipment, and machines for processing data, including, for example, a programmable processor, a computer or multiple processors or computers. In addition to hardware, the device may also include code that creates an execution environment for the computer program in question, such as code that constitutes processor firmware, a protocol stack, a database management system, an operating system, or a combination of one or more of them. A propagation signal is an artificially generated signal, such as an electrical signal, an optical signal, or an electromagnetic signal generated by a machine, which is generated to encode information to be transmitted to a suitable receiver device.

[0113] A computer program (also referred to as a program, software, software application, script, or code) can be written in any form of programming language (including compiled or interpreted languages), and the computer program can be deployed in any form, including being deployed as an independent program or as a module, component, subroutine, or another unit suitable for use in a computing environment. A computer program does not necessarily correspond to a file in a file system. A program may be stored in a portion of a file that holds other programs or data (e.g., one or more scripts stored in a markup language document), or in a single file dedicated to the program under discussion, or in multiple collaborative files (e.g., files storing one or more modules, subroutines, or partial codes). A computer program may be deployed to be executed on one computer or on multiple computers located at one site or distributed in multiple sites and interconnected by a communication network.

[0114] The processes and logic flows described in this document may be performed by one or more programmable processors executing one or more computer programs to perform functions by operating on input data and generating output. The processes and logic flows may also be performed by a dedicated logic circuit system (e.g., FPGA (field programmable gate array) or ASIC (application-specific integrated circuit)), and the apparatus may also be implemented as a dedicated logic circuit (e.g., FPGA (field programmable gate array) or ASIC (application-specific integrated circuit)).

[0115] Processors suitable for executing computer programs include, for example, both general-purpose microprocessors and special-purpose microprocessors and any one or more processors of any kind of digital computer. Typically, the processor will receive instructions and data from a read-only memory or a random access memory or both. The essential components of a computer are a processor for executing instructions and one or more memory devices for storing instructions and data. Typically, a computer will also include one or more mass storage devices (such as magnetic disks, magneto-optical disks, or optical disks) for storing data, or the computer will also be operatively connected to receive data from the one or more mass storage devices or transfer data to the one or more mass storage devices or both. However, the computer does not need to have such a device. Computer-readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media, and memory devices, including, for example, semiconductor memory devices, such as EPROM, EEPROM, and flash memory devices; magnetic disks, such as internal hard disks or removable disks; magneto-optical disks; and CD-ROM disks and DVD-ROM disks. The processor and memory may be supplemented by a dedicated logic circuit system or may be incorporated into the dedicated logic circuit system.

[0116] Some embodiments described herein are described in the general context of a method or process, which in one embodiment can be implemented by a computer program product included in a computer-readable medium, which may include computer executable instructions (such as program code), which can be executed by a computer in a networked environment, for example. Computer-readable media may include removable and non-removable storage devices, including but not limited to read-only memory (ROM), random access memory (RAM), compact disk (CD), digital versatile disk (DVD), etc. Therefore, computer-readable media may include non-transitory storage media. Generally, program modules may include routines, programs, objects, components, data structures, etc., which perform specific tasks or implement specific abstract data types. Computer or processor executable instructions, associated data structures, and program modules represent examples of program codes for executing the steps of the methods disclosed herein. Such specific sequences of executable instructions or associated data structures represent examples of corresponding actions for implementing the functions described in these steps or processes.

[0117] Some embodiments of the disclosed embodiments can be implemented as devices or modules using hardware circuits, software or a combination thereof. For example, a hardware circuit implementation may include discrete analog and / or digital components, which may be integrated as parts of a printed circuit board, for example. Alternatively or additionally, the disclosed components or modules may be implemented as application specific integrated circuits (ASICs) and / or field programmable gate arrays (FPGAs) devices. Additionally or alternatively, some implementations may include a digital signal processor (DSP), which is a dedicated microprocessor with an architecture optimized for the operational needs of digital signal processing associated with the disclosed functionality of the present application. Similarly, various components or subassemblies within each module may be implemented with software, hardware or firmware. Any connection method and medium known in the art may be used to provide a connection between a module and / or a component within a module, including but not limited to communication via the Internet, a wired network, or a wireless network using an appropriate protocol.

[0118] Although this paper includes many details, these details should not be interpreted as limitations to the scope of the claimed invention, but are interpreted as descriptions of features peculiar to specific embodiments. Some features described in the context of different embodiments herein may also be combined in a single embodiment. On the contrary, the various features described in the context of a single embodiment may also be separated or implemented in multiple embodiments with any suitable sub-combination. In addition, although features may be described as acting with some combinations and even initially claimed above, one or more features from the claimed combination may be removed from the combination in some cases, and the claimed combination may be directed to a variation of a sub-combination or a sub-combination. Similarly, although operations are depicted in a particular order in the drawings, this should not be understood as requiring to perform such operations in the particular order shown or in a sequential order, or to perform all described operations to achieve desired results.

Claims

1. A method for predicting collision time based on an image, characterized in that: include: According to the target object, a plurality of frames of images are collected, wherein the plurality of frames of images include a first image corresponding to a first frame and a second image corresponding to a second frame, and the second frame is after the first frame in terms of time sequence; Obtaining a first bounding box and a second bounding box, wherein the first bounding box corresponds to the target object in the first image, and the second bounding box corresponds to the target object in the second image; Encoding the first image and the second image respectively to obtain a first feature image and a second feature image; scaling the second bounding box according to each of a plurality of scale ratios to obtain a third bounding box; Extracting a first feature image block from the first feature image by using the third boundary frame; Extracting a second feature image block from the second feature image using the second boundary frame; Performing similarity calculation on the first feature image block and the second feature image block corresponding to each scale ratio of the multiple scale ratios to obtain a similarity score corresponding to each scale ratio; as well as At least one scale ratio is screened out according to the similarity score corresponding to each scale ratio to calculate a predicted value of the collision time of the target object.

2. The method for predicting collision time based on an image according to claim 1, characterized in that: Encoding the first image and the second image respectively to obtain the first feature image and the second feature image includes: The first image and the second image are input into a neural network to obtain the first feature image and the second feature image.

3. The method for predicting collision time based on an image according to claim 2, characterized in that: Calculating the similarity between the first feature image block and the second feature image block corresponding to each scale ratio to obtain the similarity score corresponding to each scale ratio includes: performing similarity calculation on the first characteristic image block and the second characteristic image block corresponding to each scale ratio to obtain a similarity value corresponding to each scale ratio; and The similarity value corresponding to each scale ratio is converted into a similarity score.

4. The method for predicting collision time based on an image according to claim 3, characterized in that: Converting the similarity value corresponding to each scale ratio into a similarity score includes: The similarity value corresponding to each scale ratio is input into the neural network to obtain the similarity score corresponding to each scale ratio.

5. The method for predicting collision time based on an image according to claim 2, characterized in that: Calculating the similarity between the first feature image block and the second feature image block corresponding to each scale ratio to obtain a similarity score corresponding to each scale ratio includes: For a scale ratio: Performing the similarity calculation on the first feature image block and the second feature image block to obtain a similarity graph, wherein the similarity graph includes a plurality of regions, and each of the plurality of regions has a positional similarity value; The similarity value of the feature image block is calculated according to the multiple alignment similarity values ​​of the multiple regions to obtain the similarity score.

6. The method for predicting collision time based on an image according to claim 5, characterized in that: Calculating the similarity value of the feature image block according to the multiple alignment similarity values ​​of the multiple regions to obtain the similarity score includes: The plurality of alignment similarity values ​​are averaged to obtain the feature image block similarity value.

7. The method for predicting collision time based on an image according to claim 5, characterized in that: Calculating the similarity value of the feature image block according to the multiple alignment similarity values ​​of the multiple regions to obtain the similarity score includes: The feature image block similarity value is input into the neural network to obtain the similarity score.

8. The method for predicting collision time based on an image according to claim 5, characterized in that: The method further comprises: In response to the target object in the first image being partially occluded, a plurality of first feature map pixels in the first feature image block form a mask, or In response to the target object in the second image being partially occluded, a plurality of second feature map pixels in the second feature image block form a mask; The similarity calculation is performed on the first feature image block and the second feature image block to obtain a similarity graph, including: In response to the plurality of first feature map pixels or the plurality of second feature map pixels having a mask, a plurality of first sub-regions among the plurality of regions in the similarity map form a mask, wherein a plurality of regions among the plurality of regions in the similarity map that do not form a mask are a plurality of second sub-regions; and Calculating the feature image block similarity value according to the multiple alignment similarity values ​​of the multiple regions to obtain the similarity score includes: The plurality of alignment similarity values ​​of the plurality of second sub-regions are averaged to obtain the characteristic image block similarity value.

9. The method for predicting collision time based on an image according to claim 1, characterized in that: According to the similarity score corresponding to each scale ratio, the at least one scale ratio is screened out to calculate the predicted value of the collision time of the target object, including: A predicted value of the collision time of the target object is calculated according to the at least one scale ratio and the corresponding similarity score.

10. The method for predicting collision time based on an image according to claim 1, characterized in that: According to the similarity score corresponding to each scale ratio, the at least one scale ratio is screened out to calculate the predicted value of the collision time of the target object, including: A weighted average operation is performed on the collision time calculated by the at least one scale ratio to obtain a predicted value of the collision time.

11. The method for predicting collision time based on an image according to claim 1, characterized in that: Extracting the first feature image block from the first feature image by using the third boundary frame includes: Based on the center position of the first bounding box, the first feature image block is extracted from the first feature image through the third bounding box.

12. The method for predicting collision time based on an image according to claim 1, characterized in that: Extracting the first feature image block from the first feature image by using the third boundary frame includes: Based on the center position of the first bounding box, extracting a central feature image block from the first feature image through the third bounding box; adjusting the center position of the first bounding box to at least one other position on the first image; For each other position, extracting other feature image blocks from the first feature image through the third bounding box to obtain at least one other feature image block; and The similarity between the central feature image block and the at least one other feature image block and the second feature image block is calculated respectively, so as to select the first feature image block from the central feature image block and the at least one other feature image block.

13. The method for predicting collision time based on an image according to claim 12, characterized in that: The method further comprises: calculating similarities between the central feature image block and the at least one other feature image block and the second feature image block respectively, so as to select the first feature image block from the central feature image block and the at least one other feature image block, and comprising: Calculating the mean of the alignment cosine similarity between the central feature image block and the at least one other feature image block and the second feature image block respectively; and From the calculated mean values ​​of the aligned cosine similarities, one of the central feature image block or the at least one other feature image block corresponding to the maximum mean value of the aligned cosine similarities is selected as the first feature image block.

14. The method for predicting collision time based on an image according to claim 1, characterized in that: Also includes: The size of the first feature image block is adjusted to be the same as that of the second feature image block.

15. The method for predicting collision time based on an image according to claim 14, characterized in that: Adjusting the size of the first feature image block to be the same as that of the second feature image block, comprising: The width and height of the first feature image block are scaled to be the same as the width and height of the second feature image block.

16. A method for training a neural network, characterized in that: include: Obtaining a first bounding box and a second bounding box, wherein the first bounding box corresponds to a target object in a first image, and the second bounding box corresponds to the target object in a second image, the first image corresponds to a first frame, and the second image corresponds to a second frame, and the second frame is after the first frame in terms of time sequence; Inputting the first image and the second image into the neural network to obtain a first feature image and a second feature image output by a prediction model; scaling the second bounding box according to each of a plurality of scale ratios to obtain a third bounding box; Extracting a first feature image block from the first feature image by using the third boundary frame; Extracting a second feature image block from the second feature image using the second boundary frame; Performing similarity calculation on the first feature image block and the second feature image block corresponding to each scale ratio of the multiple scale ratios to obtain a similarity score corresponding to each scale ratio, wherein the similarity score is used to indicate whether the scale ratio corresponding to the similarity score is close to the real scale ratio; According to the true scale ratio, obtaining a sample similarity score corresponding to each scale ratio; For each scale ratio, comparing the similarity score and the sample similarity score to generate a comparison result; as well as The neural network updates the prediction model according to the comparison result to output a new similarity score, and the updated prediction model is used to make the new similarity score more responsive to whether the corresponding scale ratio is close to the true scale ratio.

17. The method for training a neural network according to claim 16, wherein: Calculating the similarity between the first feature image block and the second feature image block corresponding to each scale ratio to obtain the similarity score corresponding to each scale ratio includes: performing similarity calculation on the first characteristic image block and the second characteristic image block corresponding to each scale ratio to obtain a similarity value corresponding to each scale ratio; and The similarity value corresponding to each scale ratio is input into the neural network to obtain the similarity score corresponding to each scale ratio output by the prediction model.

18. The method for training a neural network according to claim 16, wherein: The neural network updates the prediction model according to the comparison result to output the new similarity score, including: The neural network updates the parameters in the prediction model according to the comparison result and through the cross entropy loss function, so as to minimize the output value of the cross entropy loss function.

19. The method for training a neural network according to claim 16, wherein: The real scale ratio is a scale ratio obtained by using a laser radar and calculated based on the real collision time for the target object.

20. The method for training a neural network according to claim 16, wherein: Obtaining the sample similarity score corresponding to each scale ratio according to the true scale ratio, including: According to the true scale ratio, obtaining an initial sample similarity score corresponding to each scale ratio; In response to the initial sample similarity score being greater than a preset threshold, setting the initial sample similarity score to a first value; and In response to the initial sample similarity score being less than or equal to the preset threshold, the initial sample similarity score is set to a second value, wherein the similarity score is a value falling within a range defined by the first value and the second value.

21. The method for training a neural network according to claim 20, wherein: According to the true scale ratio, the initial sample similarity score corresponding to each scale ratio is obtained, including: According to the difference between the value of each scale ratio and the value of the true scale ratio, an initial sample similarity score corresponding to each scale ratio is obtained.

22. The method for training a neural network according to claim 16, wherein: Obtaining the sample similarity score corresponding to each scale ratio according to the true scale ratio, including: Determine a scale ratio among multiple scale ratios that is closest to the true scale ratio as an approximate scale ratio; determining a plurality of scale ratios other than the approximate scale ratio among the plurality of scale ratios as a plurality of other scale ratios; Setting the sample similarity score corresponding to the approximate scale ratio to a third value; and The sample similarity score of each of the plurality of other scale ratios is set to a fourth value, wherein the similarity score is a value falling within a range defined by the third value and the fourth value.

23. A device for predicting collision time based on an image, characterized in that: include: one or more processors, and A memory storing a program, the program comprising instructions which, when executed by the processor, cause the apparatus to perform a method according to any one of claims 1 to 15.

24. A device for training a neural network, characterized in that: include: one or more processors, and A memory storing a program, the program comprising instructions which, when executed by the processor, cause the apparatus to perform a method according to any one of claims 16 to 22.

25. A computer-readable storage medium storing a program, characterized in that: The program comprises instructions which, when executed by one or more processors of a computing device, cause the computing device to perform a method according to any one of claims 1 to 22.

Images