Method and device for determining depth of field of an object

Abstract

The present application provides a method and device for determining the depth of field of an object. The method includes: obtaining multiple historical image data, the multiple historical image data are obtained by using multiple image acquisition devices with different field of view angles to shoot historical targets at the same distance and / or different distances from the vehicle within a historical time period; using the depth of field of each historical image data and the historical target corresponding to each historical image data to train a normalized model; obtaining real-time image data, and using the normalized model to determine the calculated depth of field of the real-time target corresponding to the real-time image data; using the relevant parameters of the image acquisition device that shoots the real-time image data, denormalizing the calculated depth of field of the real-time target to obtain the real depth of field of the real-time target corresponding to the real-time image data. The normalized model of this solution is applicable to image acquisition devices with different viewing angles and targets at different distances.

Description

Technical Field

[0001] The present application relates to the field of computer vision, and more specifically, to a method, device, computer-readable storage medium, processor, vehicle, and system for determining the depth of field of an object. Background Art

[0002] Computer vision is mainly a simulation of biological vision through computers and related visual sensors. First, visual sensors are used to obtain external images, and then the external images are converted into digital signals to realize the digital processing of images.

[0003] Estimating the depth of field of an object is an important branch of computer vision. Existing solutions often use simple monocular scene depth estimation methods and binocular scene depth estimation methods. The features obtained by using a monocular camera are relatively few, and the use of a binocular camera requires stereo image matching, which is relatively complex to calculate. The use of monocular scene depth estimation methods and binocular scene depth estimation methods both result in low accuracy. Summary of the invention

[0004] The main purpose of the present application is to provide a method, device, computer-readable storage medium, processor, vehicle and system for determining the depth of field of an object, so as to at least solve the problem of low accuracy of the method for estimating the depth of field of an object.

[0005] In order to achieve the above-mentioned purpose, according to one aspect of the present application, a method for determining the depth of field of an object is provided, the method being applied to a vehicle driving system, the vehicle driving system comprising a vehicle and a plurality of image acquisition devices with different field of views installed on the vehicle, comprising: acquiring a plurality of historical image data, the plurality of historical image data being obtained by photographing historical targets at the same distance and / or different distances from the vehicle within a historical time period using a plurality of image acquisition devices with different field of views; training a normalized model using the depth of field of each of the historical image data and the historical targets corresponding to each of the historical image data; acquiring real-time image data, and determining the calculated depth of field of the real-time target corresponding to the real-time image data using the normalized model; and performing denormalization processing on the calculated depth of field of the real-time target using relevant parameters of the image acquisition device photographing the real-time image data to obtain the real depth of field of the real-time target corresponding to the real-time image data.

[0006] Furthermore, the calculated depth of field of the real-time target object is denormalized using relevant parameters of the image acquisition device that captures the real-time image data to obtain a real depth of field of the real-time target object corresponding to the real-time image data, including: determining a ratio relationship between the calculated depth of field of the real-time target object and the real depth of field of the real-time target object to be determined based on relevant parameters of the image acquisition device; and determining the real depth of field of the real-time target object based on the ratio relationship and the calculated depth of field of the real-time target object.

[0007] Furthermore, each of the historical image data and the depth of field of the historical target object corresponding to each of the historical image data are used to train a normalized model, including: filtering and threshold segmentation processing are performed on each of the historical image data to obtain processed historical image data corresponding to each of the historical image data; and each of the processed historical image data and the depth of field of the historical target object corresponding to each of the processed historical image data are used to train a normalized model.

[0008] Furthermore, each of the processed historical image data and the depth of field of the historical target object corresponding to each of the processed historical image data are used for training to obtain a normalized model, including: extracting a variety of different feature parameters of the processed historical image data, and a color channel of the processed historical image data represents one of the feature parameters; using a variety of different feature parameters of each of the processed historical image data and the depth of field of the historical target object corresponding to each of the processed historical image data to train to obtain a normalized model.

[0009] Further, in the case where there are three image acquisition devices, the historical image data and the depth of field of the historical target object corresponding to each historical image data are used for training to obtain a normalized model, including: constructing a training set, the training set including a first number of the historical image data captured by a first field of view angle image acquisition device, a second number of the historical image data captured by a second field of view angle image acquisition device, and a third number of the historical image data captured by a third field of view angle image acquisition device, wherein the first number is at least determined by the size of the first field of view angle, the relative position relationship between the first field of view angle image acquisition device and the vehicle, and the relative position relationship between the historical target object and the vehicle, the second number is at least determined by the size of the second field of view angle, the relative position relationship between the second field of view angle image acquisition device and the vehicle, and the relative position relationship between the historical target object and the vehicle, and the third number is at least determined by the size of the third field of view angle, the relative position relationship between the third field of view angle image acquisition device and the vehicle, and the relative position relationship between the historical target object and the vehicle; and using the training set for training to obtain the normalized model.

[0010] Furthermore, in the process of training using each of the historical image data and the depth of field of the historical target object corresponding to each of the historical image data, the method also includes: obtaining the error between the output result obtained by the normalized model and the depth of field of the historical target object; adjusting at least one of the first number, the second number and the third number according to the error.

[0011] Furthermore, after using relevant parameters of the image acquisition device that captures the real-time image data to perform denormalization on the calculated depth of field of the real-time target to obtain a real depth of field of the real-time target corresponding to the real-time image data, the method also includes: determining the driving speed and driving acceleration of the vehicle based on the real depth of field of the real-time target.

[0012] Furthermore, the relevant parameters of the image acquisition device include the relative posture between the coordinates of the image acquisition device and the world coordinates, the optical center position of the image acquisition device, and the distortion amount of the image acquisition device.

[0013] Furthermore, the field of view angle is one of the following: 30°, 60°, 90°, 120°.

[0014] Furthermore, the normalization model is a convolutional neural network model, and the convolutional neural network model includes an input layer, an output layer and a hidden layer.

[0015] According to another aspect of the present application, there is provided a device for determining the depth of field of an object, the device being applied to a vehicle driving system, the vehicle driving system comprising a vehicle and a plurality of image acquisition devices with different field of views installed on the vehicle, comprising: a first acquisition unit for acquiring a plurality of historical image data, the plurality of historical image data being obtained by photographing historical targets at the same distance and / or different distances from the vehicle within a historical time period using the plurality of image acquisition devices with different field of views; a training unit for training to obtain a normalized model using the depth of field of each of the historical image data and the historical targets corresponding to each of the historical image data; a second acquisition unit for acquiring real-time image data, and determining the calculated depth of field of the real-time target corresponding to the real-time image data using the normalized model; and a processing unit for performing denormalization processing on the calculated depth of field of the real-time target using relevant parameters of the image acquisition device photographing the real-time image data, to obtain the real depth of field of the real-time target corresponding to the real-time image data.

[0016] According to another aspect of the present application, a computer-readable storage medium is provided, wherein the computer-readable storage medium includes a stored program, wherein when the program is executed, the device where the computer-readable storage medium is located is controlled to execute any one of the methods described.

[0017] According to another aspect of the present application, a processor is provided, wherein the processor is used to run a program, wherein any one of the methods is executed when the program is run.

[0018] According to another aspect of the present application, a vehicle is provided, comprising one or more processors, a memory and one or more programs, wherein the one or more programs are stored in the memory and are configured to be executed by the one or more processors, and the one or more programs include methods for executing any one of the described methods.

[0019] According to another aspect of the present application, a system is provided, comprising the vehicle and a plurality of image acquisition devices with different field of view angles, wherein the image acquisition devices are mounted on the vehicle and communicate with the vehicle.

[0020] The technical solution of the present application is applied, by acquiring multiple historical image data, the multiple historical image data are obtained by using multiple image acquisition devices with different field of view to shoot historical targets at the same distance and / or different distances from the vehicle in the historical time period, using each of the historical image data and the depth of field of the historical targets corresponding to each of the historical image data to train a normalized model, acquiring real-time image data, and using the normalized model to determine the calculated depth of field of the real-time target corresponding to the real-time image data, using the relevant parameters of the image acquisition device that shoots the real-time image data, the calculated depth of field of the real-time target is denormalized to obtain the real depth of field of the real-time target corresponding to the real-time image data. Since the normalized model is obtained by using multiple image acquisition devices with different field of view to capture historical targets at the same distance and / or different distances from the vehicle, the obtained normalized model is applicable to image acquisition devices with different viewing angles and targets at different distances. That is, a universal model is trained to obtain the depth of field of the target object corresponding to the image acquired by the above-mentioned image acquisition devices with different field of view angles, and then the real depth of field of the real-time target object is obtained through denormalization processing. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] The drawings constituting part of the present application are used to provide a further understanding of the present application. The exemplary embodiments and descriptions of the present application are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0022] Figure 1 A flow chart of a method for determining the depth of field of an object according to an embodiment of the present application is shown;

[0023] Figure 2 A schematic diagram of a device for determining the depth of field of an object according to an embodiment of the present application is shown;

[0024] Figure 3 A schematic diagram of solving a normalized multiplier according to an embodiment of the present application is shown. DETAILED DESCRIPTION

[0025] It should be noted that, in the absence of conflict, the embodiments and features in the embodiments of the present application can be combined with each other. The present application will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0026] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present application.

[0027] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequential order. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present application described here. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0028] It should be understood that when an element (such as a layer, film, region, or substrate) is described as being "on" another element, the element may be directly on the other element, or there may be intermediate elements. Moreover, in the specification and claims, when an element is described as being "connected" to another element, the element may be "directly connected" to the other element, or "connected" to the other element through a third element.

[0029] As introduced in the background technology, the method of estimating the depth of field of an object in the prior art has low accuracy. In order to solve the problem of low accuracy of the method of estimating the depth of field of an object, the embodiments of the present application provide a method, device, computer-readable storage medium, processor, vehicle and system for determining the depth of field of an object.

[0030] According to an embodiment of the present application, a method for determining the depth of field of an object is provided.

[0031] Figure 1 is a flow chart of a method for determining the depth of field of an object according to an embodiment of the present application. The method is applied to a vehicle driving system, the vehicle driving system comprising a vehicle and a plurality of image acquisition devices with different field of view angles installed on the vehicle, such as Figure 1 As shown, the method comprises the following steps:

[0032] Step S101, acquiring a plurality of historical image data, wherein the plurality of historical image data are obtained by photographing historical targets at the same distance and / or different distances from the vehicle within a historical time period using a plurality of the image acquisition devices having different field of view angles;

[0033] Step S102, using each of the historical image data and the depth of field of the historical target object corresponding to each of the historical image data to perform training to obtain a normalized model;

[0034] Step S103, acquiring real-time image data, and using the normalized model to determine the real-time target object corresponding to the real-time image data to calculate the depth of field;

[0035] Step S104, using relevant parameters of the image acquisition device that shoots the real-time image data, denormalize the calculated depth of field of the real-time target object to obtain a real depth of field of the real-time target object corresponding to the real-time image data.

[0036] Specifically, the above-mentioned image acquisition device can be a camera.

[0037] In the above scheme, by acquiring multiple historical image data, the multiple historical image data are obtained by using multiple image acquisition devices with different field of view to shoot historical targets at the same distance and / or different distances from the vehicle within a historical time period, and using the depth of field of each of the historical image data and the historical targets corresponding to each of the historical image data to train a normalized model, acquire real-time image data, and use the normalized model to determine the calculated depth of field of the real-time target corresponding to the real-time image data, and use the relevant parameters of the image acquisition device that shoots the real-time image data to perform a denormalization process on the calculated depth of field of the real-time target, so as to obtain the real depth of field of the real-time target corresponding to the real-time image data. Since the normalized model is obtained by using multiple image acquisition devices with different field of view to capture historical targets at the same distance and / or different distances from the vehicle, the obtained normalized model is applicable to image acquisition devices with different viewing angles and targets at different distances. That is, a universal model is trained to obtain the depth of field of the target object corresponding to the image acquired by the above-mentioned image acquisition devices with different field of view angles, and then the real depth of field of the real-time target object is obtained through denormalization processing.

[0038] It should be noted that the steps shown in the flowcharts of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and that, although a logical order is shown in the flowcharts, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0039] In an optional embodiment, the calculated depth of field of the real-time target is denormalized by using the relevant parameters of the image acquisition device that shoots the real-time image data to obtain the real depth of field of the real-time target corresponding to the real-time image data, including: determining the ratio relationship between the calculated depth of field of the real-time target and the real depth of field of the real-time target to be determined according to the relevant parameters of the image acquisition device; determining the real depth of field of the real-time target according to the ratio relationship and the calculated depth of field of the real-time target. That is, the corresponding relationship between the calculated depth of field of the real-time target and the real depth of field of the real-time target is obtained through denormalization. Since the depth of field is the distance information, the ratio relationship between the calculated depth of field of the real-time target and the real depth of field of the real-time target is obtained, and then the real depth of field of the real-time target is determined according to the calculated depth of field of the real-time target obtained by the normalization model and the ratio relationship. Accurate determination of the real depth of field of the real-time target is achieved.

[0040] In a specific embodiment of the present application, a method for obtaining the depth of field of an object includes the following steps:

[0041] Step 1: Multiply the homogeneous coordinates of the center point of the 3D target by the camera extrinsic parameter matrix, and then multiply it by the camera intrinsic parameter matrix to obtain the two-dimensional coordinates of the center point of the 3D target in the image plane;

[0042] Step 2: Normalize the two-dimensional coordinates of the center point of the 3D target in the image plane through the internal parameter matrix to obtain the coordinates of the center point of the 3D target on the normalized image plane;

[0043] Step 3: If Figure 3 As shown, a ray is emitted from the camera center M to the two-dimensional coordinate N of the center point of the 3D target on the normalized image plane Z, and the ray is intercepted at a predetermined depth (e.g., 10 meters) to obtain a point P, a sphere B (e.g., with a diameter of 1 cm) is constructed based on the point P, and then the sphere B is projected onto the normalized image plane Z to obtain a circle C, the number of pixels occupied by the width of the circle is obtained, and the number of pixels is divided by the diameter of the sphere to obtain a normalized multiplier;

[0044] Step 4: Divide the depth of the annotated 3D target object (i.e., the true depth) by the normalization multiplier to obtain the normalized object depth, which is used as the deep learning regression target;

[0045] Step 5: Denormalization: For a 2D bounding box of an object in an image, the normalization model estimates a normalized depth. When denormalizing, take the center point of the box as the 2D coordinate, perform steps 2 and 3 above, and get the normalization multiplier. Multiply the depth estimated by the model by the normalization multiplier to get the denormalized depth, which is the true depth.

[0046] In an optional embodiment, each of the above historical image data and the depth of field of the above historical target object corresponding to each of the above historical image data are used for training to obtain a normalized model, including: filtering and threshold segmentation processing are performed on each of the above historical image data to obtain processed historical image data corresponding to each of the above historical image data; and each of the above processed historical image data and the depth of field of the above historical target object corresponding to each of the above processed historical image data are used for training to obtain a normalized model. Specifically, the filtering process is to filter out the noise of the historical image data, and the threshold segmentation process is to perform binary segmentation. The use of the processed historical image data is conducive to the training of the model and ensures the accuracy of the normalized model obtained through training.

[0047] In an optional embodiment, each of the above-mentioned processed historical image data and the depth of field of the above-mentioned historical target object corresponding to each of the above-mentioned processed historical image data are used for training to obtain a normalized model, including: extracting a variety of different feature parameters of the above-mentioned processed historical image data, and a color channel of the above-mentioned processed historical image data represents one of the above-mentioned feature parameters; using a variety of different feature parameters of each of the above-mentioned processed historical image data and the depth of field of the above-mentioned historical target object corresponding to each of the above-mentioned processed historical image data to train to obtain a normalized model. Model training is actually training parameters, extracting a variety of different feature parameters of the above-mentioned processed historical image data, and then training to obtain an accurate normalized model.

[0048] Specifically, the processed historical image data has ten color channels, each color channel can represent a feature parameter, and then images corresponding to different color channels can be used for training to obtain a normalized model.

[0049] In an optional embodiment, when there are three image acquisition devices, the historical image data and the depth of field of the historical target object corresponding to each historical image data are used for training to obtain a normalized model, including: constructing a training set, the training set including a first number of the historical image data captured by a first field of view image acquisition device, a second number of the historical image data captured by a second field of view image acquisition device, and a third number of the historical image data captured by a third field of view image acquisition device, wherein the first number is at least determined by the size of the first field of view, the relative position relationship between the first field of view image acquisition device and the vehicle, and the relative position relationship between the historical target object and the vehicle, the second number is at least determined by the size of the second field of view, the relative position relationship between the second field of view image acquisition device and the vehicle, and the relative position relationship between the historical target object and the vehicle, and the third number is at least determined by the size of the third field of view, the relative position relationship between the third field of view image acquisition device and the vehicle, and the relative position relationship between the historical target object and the vehicle; and using the training set for training to obtain the normalized model. That is, the amount of data in the training set can be determined based on parameters such as the size of the field of view angle, the relative position relationship between the image acquisition device and the above-mentioned vehicle, and the relative position relationship between the above-mentioned historical target object and the above-mentioned vehicle. Through adaptive adjustment, the normalized model obtained through training can be guaranteed to have better applicability and higher accuracy.

[0050] In an optional embodiment, in the process of training using each of the above historical image data and the depth of field of the above historical target object corresponding to each of the above historical image data, the above method further includes: obtaining the error between the output result obtained by the above normalization model and the depth of field of the above historical target object; and adjusting at least one of the above first quantity, the above second quantity and the above third quantity according to the above error. That is, in order to ensure the accuracy of the parameters of the model during the training process, at least one of the first quantity, the above second quantity and the above third quantity can be adjusted according to the error between the output result obtained by the normalization model and the depth of field of the above historical target object.

[0051] In another embodiment, the parameters in the model can be adjusted according to the error between the output result obtained by the normalized model and the depth of field of the historical target object, for example, the number of layers of the network can be adjusted.

[0052] In an optional embodiment, after the relevant parameters of the image acquisition device that shoots the real-time image data are used to perform a denormalization process on the calculated depth of field of the real-time target object to obtain the real depth of field of the real-time target object corresponding to the real-time image data, the method further includes: determining the driving speed and driving acceleration of the vehicle according to the real depth of field of the real-time target object. That is, guiding real-time navigation according to the real depth of field of the real-time target object.

[0053] In an optional embodiment, the relevant parameters of the image acquisition device include the relative position between the coordinates of the image acquisition device and the world coordinates, the optical center position of the image acquisition device, and the distortion of the image acquisition device. Of course, the relevant parameters also include other parameters besides the relative position between the coordinates of the image acquisition device and the world coordinates, the optical center position of the image acquisition device, and the distortion of the image acquisition device. Those skilled in the art can select according to actual needs.

[0054] In an optional embodiment, the above-mentioned field of view angle is one of the following: 30°, 60°, 90°, 120°. Of course, the field of view angle can also be a field of view angle other than 30°, 60°, 90°, 120°.

[0055] In an optional embodiment, the normalized model is a convolutional neural network model, which includes an input layer, an output layer and a hidden layer.

[0056] The embodiment of the present application also provides a device for determining the depth of field of an object. It should be noted that the device for determining the depth of field of an object in the embodiment of the present application can be used to execute the method for determining the depth of field of an object provided in the embodiment of the present application. The device for determining the depth of field of an object provided in the embodiment of the present application is introduced below.

[0057] Figure 2 is a schematic diagram of an apparatus for determining the depth of field of an object according to an embodiment of the present application. The apparatus is applied to a vehicle driving system, which includes a vehicle and a plurality of image acquisition devices with different field of view angles installed on the vehicle, such as Figure 2 As shown, the device comprises:

[0058] A first acquisition unit 10 is used to acquire a plurality of historical image data, wherein the plurality of historical image data are obtained by photographing historical targets at the same distance and / or different distances from the vehicle within a historical time period using a plurality of the image acquisition devices with different field of view;

[0059] A training unit 20, configured to use each of the historical image data and the depth of field of the historical target object corresponding to each of the historical image data to perform training to obtain a normalized model;

[0060] A second acquisition unit 30 is used to acquire real-time image data and determine the real-time target object calculation depth of field corresponding to the real-time image data using the normalization model;

[0061] The processing unit 40 is used to use the relevant parameters of the above-mentioned image acquisition device that shoots the above-mentioned real-time image data to perform denormalization processing on the calculated depth of field of the above-mentioned real-time target object, so as to obtain the real depth of field of the above-mentioned real-time target object corresponding to the above-mentioned real-time image data.

[0062] In the above scheme, the first acquisition unit acquires a plurality of historical image data, and the plurality of historical image data are obtained by photographing historical targets at the same distance and / or different distances from the vehicle within a historical time period using a plurality of the above image acquisition devices with different field of view angles, and the training unit uses each of the above historical image data and the depth of field of the above historical targets corresponding to each of the above historical image data to perform training to obtain a normalized model, the second acquisition unit acquires real-time image data, and uses the above normalized model to determine the calculated depth of field of the real-time target corresponding to the real-time image data, and the processing unit uses the relevant parameters of the above image acquisition device that photographed the real-time image data to perform denormalization processing on the calculated depth of field of the real-time target, and obtains the real depth of field of the real-time target corresponding to the real-time image data. Since the normalized model is obtained by photographing historical targets at the same distance and / or different distances from the vehicle using a plurality of the above image acquisition devices with different field of view angles, the obtained normalized model is applicable to image acquisition devices with different viewing angles and targets at different distances. That is, a universal model is trained to obtain the depth of field of the target object corresponding to the image acquired by the above-mentioned image acquisition devices with different field of view angles, and then the real depth of field of the real-time target object is obtained through denormalization processing.

[0063] In an optional embodiment, the processing unit includes a first determination module and a second determination module, wherein the first determination module is used to determine the ratio relationship between the calculated depth of field of the real-time target and the real depth of field of the real-time target to be determined according to the relevant parameters of the image acquisition device; the second determination module is used to determine the real depth of field of the real-time target according to the ratio relationship and the calculated depth of field of the real-time target. That is, the corresponding relationship between the calculated depth of field of the real-time target and the real depth of field of the real-time target is obtained through inverse normalization. Since the depth of field is the distance information, the ratio relationship between the calculated depth of field of the real-time target and the real depth of field of the real-time target is obtained, and then the real depth of field of the real-time target is determined according to the calculated depth of field of the real-time target obtained by the normalization model and the ratio relationship. Accurate determination of the real depth of field of the real-time target is achieved.

[0064] In an optional embodiment, the training unit includes a processing module and a first training module, the processing module is used to perform filtering processing and threshold segmentation processing on each of the above historical image data to obtain processed historical image data corresponding to each of the above historical image data; the first training module is used to use each of the above processed historical image data and the depth of field of the above historical target object corresponding to each of the above processed historical image data to perform training to obtain a normalized model. Specifically, the filtering processing is to filter out the noise of the historical image data, and the threshold segmentation processing is to perform binary segmentation. The use of the processed historical image data is conducive to the training of the model and ensures the accuracy of the normalized model obtained through training.

[0065] In an optional embodiment, the first training module includes an extraction submodule and a training submodule, the extraction submodule is used to extract a plurality of different feature parameters of the above-processed historical image data, and a color channel of the above-processed historical image data represents one of the above-mentioned feature parameters; the training submodule is used to use the above-mentioned plurality of different feature parameters of each of the above-mentioned processed historical image data, and the depth of field of the above-mentioned historical target object corresponding to each of the above-mentioned processed historical image data, to train and obtain a normalized model. The training of the model is actually to train the parameters, extract the various different feature parameters of the above-mentioned processed historical image data, and then train, so as to obtain an accurate normalized model.

[0066] In an optional embodiment, when there are three image acquisition devices, the training unit includes a construction module and a second training module, the construction module is used to construct a training set, the training set includes a first number of the above-mentioned historical image data captured by the first field of view angle image acquisition device, a second number of the above-mentioned historical image data captured by the second field of view angle image acquisition device, and a third number of the above-mentioned historical image data captured by the third field of view angle image acquisition device, wherein the first number is at least determined by the size of the above-mentioned first field of view angle, the relative position relationship between the above-mentioned first field of view angle image acquisition device and the above-mentioned vehicle, and the relative position relationship between the above-mentioned historical target object and the above-mentioned vehicle, the second number is at least determined by the size of the above-mentioned second field of view angle, the relative position relationship between the above-mentioned second field of view angle image acquisition device and the above-mentioned vehicle, and the relative position relationship between the above-mentioned historical target object and the above-mentioned vehicle, and the third number is at least determined by the size of the above-mentioned third field of view angle, the relative position relationship between the above-mentioned third field of view angle image acquisition device and the above-mentioned vehicle, and the relative position relationship between the above-mentioned historical target object and the above-mentioned vehicle; the second training module is used to use the above-mentioned training set for training to obtain the above-mentioned normalized model. That is, the amount of data in the training set can be determined based on parameters such as the size of the field of view angle, the relative position relationship between the image acquisition device and the above-mentioned vehicle, and the relative position relationship between the above-mentioned historical target object and the above-mentioned vehicle. Through adaptive adjustment, the normalized model obtained through training can be guaranteed to have better applicability and higher accuracy.

[0067] In an optional embodiment, the device further includes a third acquisition unit and an adjustment unit, wherein the third acquisition unit is used to acquire the error between the output result obtained by the normalized model and the depth of field of the historical target object during the training process using each of the historical image data and the depth of field of the historical target object corresponding to each of the historical image data; and the adjustment unit is used to adjust at least one of the first quantity, the second quantity and the third quantity according to the error. That is, in order to ensure the accuracy of the parameters of the model during the training process, at least one of the first quantity, the second quantity and the third quantity can be adjusted according to the error between the output result obtained by the normalized model and the depth of field of the historical target object.

[0068] In an optional embodiment, the device further includes a determination unit, which is used to perform a denormalization process on the depth of field calculated by the real-time target object using the relevant parameters of the image acquisition device that shoots the real-time image data, and obtain the real depth of field of the real-time target object corresponding to the real-time image data, and then determine the driving speed and driving acceleration of the vehicle according to the real depth of field of the real-time target object. That is, the real-time navigation is guided according to the real depth of field of the real-time target object.

[0069] The device for determining the depth of field of an object includes a processor and a memory. The first acquisition unit, training unit, second acquisition unit and processing unit are all stored in the memory as program units, and the processor executes the program units stored in the memory to implement corresponding functions.

[0070] The processor includes a kernel, which calls the corresponding program unit from the memory. One or more kernels can be set, and the depth of field of the object can be accurately determined by adjusting the kernel parameters.

[0071] The memory may include non-permanent memory in a computer-readable medium, random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM, and the memory includes at least one memory chip.

[0072] An embodiment of the present invention provides a computer-readable storage medium, wherein the computer-readable storage medium includes a stored program, wherein when the program is executed, the device where the computer-readable storage medium is located is controlled to execute the method for determining the depth of field of an object.

[0073] An embodiment of the present invention provides a processor, which is used to run a program, wherein the method for determining the depth of field of an object is executed when the program is run.

[0074] An embodiment of the present invention provides a vehicle, comprising one or more processors, a memory and one or more programs, wherein the one or more programs are stored in the memory and are configured to be executed by the one or more processors, and the one or more programs include methods for executing any one of the methods described above.

[0075] An embodiment of the present invention provides a system, comprising the above-mentioned vehicle and a plurality of image acquisition devices with different field of view angles, wherein the image acquisition devices are installed on the above-mentioned vehicle and communicate with the above-mentioned vehicle.

[0076] An embodiment of the present invention provides a device, the device including a processor, a memory, and a program stored in the memory and executable on the processor, and when the processor executes the program, at least the following steps are implemented:

[0077] Step S101, acquiring a plurality of historical image data, wherein the plurality of historical image data are obtained by photographing historical targets at the same distance and / or different distances from the vehicle within a historical time period using a plurality of the image acquisition devices having different field of view angles;

[0078] Step S102, using each of the historical image data and the depth of field of the historical target object corresponding to each of the historical image data to perform training to obtain a normalized model;

[0079] Step S103, acquiring real-time image data, and using the normalized model to determine the real-time target object corresponding to the real-time image data to calculate the depth of field;

[0080] Step S104, using relevant parameters of the image acquisition device that shoots the real-time image data, denormalize the calculated depth of field of the real-time target object to obtain a real depth of field of the real-time target object corresponding to the real-time image data.

[0081] The devices in this article can be servers, PCs, PADs, mobile phones, etc.

[0082] The present application also provides a computer program product, which, when executed on a data processing device, is suitable for executing a program for initializing at least the following method steps:

[0083] Step S101, acquiring a plurality of historical image data, wherein the plurality of historical image data are obtained by photographing historical targets at the same distance and / or different distances from the vehicle within a historical time period using a plurality of the image acquisition devices having different field of view angles;

[0084] Step S102, using each of the historical image data and the depth of field of the historical target object corresponding to each of the historical image data to perform training to obtain a normalized model;

[0085] Step S103, acquiring real-time image data, and using the normalized model to determine the real-time target object corresponding to the real-time image data to calculate the depth of field;

[0086] Step S104, using relevant parameters of the image acquisition device that shoots the real-time image data, denormalize the calculated depth of field of the real-time target object to obtain a real depth of field of the real-time target object corresponding to the real-time image data.

[0087] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that include computer-usable program code.

[0088] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0089] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0090] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process in the computer or other programmable device. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0091] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0092] The memory may include non-permanent memory in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.

[0093] Computer readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.

[0094] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0095] From the above description, it can be seen that the above embodiments of the present application achieve the following technical effects:

[0096] 1) The method for determining the depth of field of an object of the present application is to obtain multiple historical image data, wherein the multiple historical image data are obtained by using multiple image acquisition devices with different field of view to shoot historical targets at the same distance and / or different distances from the vehicle within a historical time period, and the depth of field of each historical image data and the historical targets corresponding to each historical image data are used to train a normalized model, obtain real-time image data, and use the normalized model to determine the calculated depth of field of the real-time target corresponding to the real-time image data, and use the relevant parameters of the image acquisition device that shoots the real-time image data to perform a denormalization process on the calculated depth of field of the real-time target to obtain the real depth of field of the real-time target corresponding to the real-time image data. Since the normalized model is obtained by using multiple image acquisition devices with different field of view to capture historical targets at the same distance and / or different distances from the vehicle, the obtained normalized model is applicable to image acquisition devices with different viewing angles and targets at different distances. That is, a universal model is trained to obtain the depth of field of the target object corresponding to the image acquired by the above-mentioned image acquisition devices with different field of view angles, and then the real depth of field of the real-time target object is obtained through denormalization processing.

[0097] 2) The device for determining the depth of field of an object of the present application comprises a first acquisition unit that acquires a plurality of historical image data, wherein the plurality of the historical image data are obtained by photographing historical targets at the same distance and / or different distances from the vehicle within a historical time period using a plurality of the above image acquisition devices with different field of view angles, and a training unit that uses each of the above historical image data and the depth of field of the above historical targets corresponding to each of the above historical image data to perform training to obtain a normalized model, a second acquisition unit that acquires real-time image data and uses the above normalized model to determine the calculated depth of field of the real-time target corresponding to the real-time image data, and a processing unit that uses the relevant parameters of the above image acquisition device that photographed the above real-time image data to perform a denormalization process on the calculated depth of field of the above real-time target to obtain the real depth of field of the above real-time target corresponding to the real-time image data. Since the normalized model is obtained by photographing historical targets at the same distance and / or different distances from the vehicle using a plurality of the above image acquisition devices with different field of view angles, the obtained normalized model is applicable to image acquisition devices with different viewing angles and targets at different distances. That is, a universal model is trained to obtain the depth of field of the target object corresponding to the image acquired by the above-mentioned image acquisition devices with different field of view angles, and then the real depth of field of the real-time target object is obtained through denormalization processing.

[0098] The above description is only the preferred embodiment of the present application and is not intended to limit the present application. For those skilled in the art, the present application may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. A method for determining the depth of field of an object, characterized in that: The method is applied to a vehicle driving system, which includes a vehicle and a plurality of image acquisition devices with different field of view angles installed on the vehicle, including: Acquire a plurality of historical image data, wherein the plurality of historical image data are obtained by photographing historical targets at the same distance and / or different distances from the vehicle within a historical time period using a plurality of image acquisition devices having different field of view angles; Using each of the historical image data and the depth of field of the historical target object corresponding to each of the historical image data, training is performed to obtain a normalized model; Acquire real-time image data, and use the normalized model to determine a real-time target object calculation depth of field corresponding to the real-time image data; Using relevant parameters of the image acquisition device that photographs the real-time image data, denormalize the calculated depth of field of the real-time target object to obtain a real depth of field of the real-time target object corresponding to the real-time image data; The method of using the relevant parameters of the image acquisition device that shoots the real-time image data to perform denormalization processing on the calculated depth of field of the real-time target object to obtain the real depth of field of the real-time target object corresponding to the real-time image data includes: Determine, based on the relevant parameters of the image acquisition device, the ratio relationship between the calculated depth of field of the real-time target object and the real depth of field of the real-time target object to be determined; Calculating the depth of field of the real-time target object based on the ratio relationship and the real-time target object to determine the real depth of field of the real-time target object; Wherein, when there are three image acquisition devices, each of the historical image data and the depth of field of the historical target object corresponding to each of the historical image data are used for training to obtain a normalized model, including: Constructing a training set, the training set comprising a first number of the historical image data captured by a first field of view angle image acquisition device, a second number of the historical image data captured by a second field of view angle image acquisition device, and a third number of the historical image data captured by a third field of view angle image acquisition device, wherein the first number is determined by the size of the first field of view angle, the relative position relationship between the first field of view angle image acquisition device and the vehicle, and the relative position relationship between the historical target object and the vehicle, the second number is determined by the size of the second field of view angle, the relative position relationship between the second field of view angle image acquisition device and the vehicle, and the relative position relationship between the historical target object and the vehicle, and the third number is determined by the size of the third field of view angle, the relative position relationship between the third field of view angle image acquisition device and the vehicle, and the relative position relationship between the historical target object and the vehicle; Using the training set for training to obtain the normalized model; In the process of using each of the historical image data and the depth of field of the historical target object corresponding to each of the historical image data for training, the method further includes: Obtaining an error between an output result obtained by the normalized model and a depth of field of the historical target object; At least one of the first amount, the second amount, and the third amount is adjusted based on the error.

2. The method according to claim 1, characterized in that Using each of the historical image data and the depth of field of the historical target object corresponding to each of the historical image data to train and obtain a normalized model, including: Performing filtering and threshold segmentation processing on each of the historical image data to obtain processed historical image data corresponding to each of the historical image data; The processed historical image data and the depth of field of the historical target object corresponding to the processed historical image data are used to perform training to obtain a normalized model.

3. The method according to claim 2, characterized in that Using each of the processed historical image data and the depth of field of the historical target object corresponding to each of the processed historical image data to train and obtain a normalized model, including: Extracting a plurality of different characteristic parameters of the processed historical image data, wherein one color channel of the processed historical image data represents one of the characteristic parameters; A normalized model is obtained by training using a plurality of different feature parameters of each of the processed historical image data and a depth of field of the historical target object corresponding to each of the processed historical image data.

4. The method according to any one of claims 1 to 3, characterized in that After performing denormalization processing on the calculated depth of field of the real-time target object using relevant parameters of the image acquisition device that photographs the real-time image data to obtain the real depth of field of the real-time target object corresponding to the real-time image data, the method further includes: The driving speed and driving acceleration of the vehicle are determined according to the real depth of field of the real-time target object.

5. The method according to any one of claims 1 to 3, characterized in that The relevant parameters of the image acquisition device include the relative position between the coordinates of the image acquisition device and the world coordinates, the optical center position of the image acquisition device, and the distortion amount of the image acquisition device.

6. The method according to any one of claims 1 to 3, characterized in that The field of view angle is one of the following: 30°、60°、90°、120°。 7. The method according to any one of claims 1 to 3, characterized in that The normalization model is a convolutional neural network model, which includes an input layer, an output layer and a hidden layer.

8. A device for determining the depth of field of an object, characterized in that: The device is applied to a vehicle driving system, which includes a vehicle and a plurality of image acquisition devices with different viewing angles installed on the vehicle, including: A first acquisition unit is used to acquire a plurality of historical image data, wherein the plurality of historical image data are obtained by photographing historical targets at the same distance and / or different distances from the vehicle within a historical time period using a plurality of image acquisition devices with different field of view angles; A training unit, configured to use each of the historical image data and the depth of field of the historical target object corresponding to each of the historical image data to perform training to obtain a normalized model; A second acquisition unit is used to acquire real-time image data and determine the real-time target object calculation depth of field corresponding to the real-time image data using the normalized model; A processing unit, configured to perform a denormalization process on the calculated depth of field of the real-time target object by using relevant parameters of the image acquisition device that photographs the real-time image data, so as to obtain a real depth of field of the real-time target object corresponding to the real-time image data; The processing unit includes a first determining module and a second determining module, wherein the first determining module is used to determine the ratio relationship between the calculated depth of field of the real-time target object and the real depth of field of the real-time target object to be determined according to the relevant parameters of the image acquisition device; the second determining module is used to determine the real depth of field of the real-time target object according to the ratio relationship and the calculated depth of field of the real-time target object; Wherein, in the case where there are three image acquisition devices, the training unit includes a construction module and a second training module, the construction module is used to construct a training set, the training set includes a first number of the historical image data captured by the first field of view angle image acquisition device, a second number of the historical image data captured by the second field of view angle image acquisition device, and a third number of the historical image data captured by the third field of view angle image acquisition device, wherein the first number is determined by the size of the first field of view angle, the relative position relationship between the first field of view angle image acquisition device and the vehicle, and the relative position relationship between the historical target object and the vehicle, the second number is determined by the size of the second field of view angle, the relative position relationship between the second field of view angle image acquisition device and the vehicle, and the relative position relationship between the historical target object and the vehicle, the third number is determined by the size of the third field of view angle, the relative position relationship between the third field of view angle image acquisition device and the vehicle, and the relative position relationship between the historical target object and the vehicle, and the second training module is used to use the training set for training to obtain the normalized model; The device also includes a third acquisition unit and an adjustment unit. The third acquisition unit is used to obtain the error between the output result obtained by the normalized model and the depth of field of the historical target object during the training process using each of the historical image data and the depth of field of the historical target object corresponding to each of the historical image data; the adjustment unit is used to adjust at least one of the first number, the second number and the third number according to the error.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium includes a stored program, wherein when the program is executed, the device where the computer-readable storage medium is located is controlled to execute the method according to any one of claims 1 to 7.

10. A processor, characterized in that: The processor is used to run a program, wherein the program executes the method according to any one of claims 1 to 7 when running.

11. A vehicle, characterized in that: The method comprises one or more processors, a memory and one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, and the one or more programs include methods for executing any one of claims 1 to 7.

12. A vehicle driving system, characterized in that: The invention comprises the vehicle according to claim 11 and a plurality of image acquisition devices with different field of view angles, wherein the image acquisition devices are mounted on the vehicle and communicate with the vehicle.

Images