Image generation method, device, electronic device and computer storage medium

Abstract

The embodiment of the present application provides an image generation method, device, electronic device and computer storage medium. The image generation method includes: obtaining the pixel area position range covered by the object of interest in the kth frame image; based on the pixel area position range, obtaining the sub-image to be processed from the kth frame image; restoring the clarity of the sub-image to be processed to obtain the target sub-image; synthesizing the target sub-image with the kth frame image to obtain the output image of the kth frame. Through the scheme provided by the present application, the detection accuracy of the visual perception system can be improved, thereby improving the safety of unmanned driving and making it easier to promote unmanned vehicles.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of computer technology, and in particular to an image generation method, device, electronic device, and computer storage medium. Background Art

[0002] Since driverless vehicles (such as unmanned logistics trucks, logistics robots, unmanned passenger cars, etc.) need to drive autonomously on the road, driverless vehicles usually need to have visual perception capabilities (equivalent to the eyes of human drivers). Visual perception requires the ability to perceive both nearby road environment information and distant road environment information, such as traffic light status, traffic sign information, whether there are obstacles, etc.

[0003] At present, the visual perception system of unmanned vehicles generally perceives the road environment based on road environment images taken by visual sensors (such as variable-focus cameras or fixed-focus cameras). However, due to the performance of the visual sensors themselves or the manufacturing cost of the unmanned vehicles, it is impossible to achieve clear imaging of both near and distant objects in the same frame through low-cost visual sensors. This makes the visual perception ability of unmanned vehicles insufficient, which in turn limits the application and promotion of unmanned vehicles. Summary of the invention

[0004] In view of this, an embodiment of the present application provides an image generation solution to at least partially solve the above-mentioned problem.

[0005] According to a first aspect of an embodiment of the present application, an image generation method is provided, comprising: obtaining a pixel area position range covered by an object of interest in a k-th frame image; based on the pixel area position range, obtaining a sub-image to be processed from the k-th frame image; restoring the clarity of the sub-image to be processed to obtain a target sub-image; and synthesizing the target sub-image with the k-th frame image to obtain an output image of the k-th frame.

[0006] According to a second aspect of an embodiment of the present application, an image generating device is provided, comprising: a position information acquisition module, used to acquire a pixel area position range covered by an object of interest in the kth frame image; a first sub-image acquisition module, used to acquire a sub-image to be processed from the kth frame image based on the pixel area position range; a second sub-image acquisition module, used to restore the clarity of the sub-image to be processed to obtain a target sub-image; and a synthesis module, used to synthesize the target sub-image with the kth frame image to obtain an output image of the kth frame.

[0007] According to a third aspect of an embodiment of the present application, there is provided an electronic device, comprising: a processor, a memory, a communication interface and a communication bus, wherein the processor, the memory and the communication interface communicate with each other via the communication bus; the memory is used to store at least one executable instruction, and the executable instruction enables the processor to perform operations corresponding to the image generation method described in the first aspect.

[0008] According to a fourth aspect of the embodiments of the present application, a computer storage medium is provided, on which a computer program is stored. When the program is executed by a processor, the image generating method as described in the first aspect is implemented.

[0009] According to the image generation scheme provided by the embodiment of the present application, the scheme provided by the embodiment obtains the pixel area position range covered by the object of interest in the kth frame image; based on the pixel area position range, obtains the sub-image to be processed from the kth image frame; restores the clarity of the sub-image to be processed to obtain the target sub-image; synthesizes the target sub-image with the kth frame image to obtain the output image of the kth frame. Since the sub-image to be processed corresponding to the object of interest in the output image of the kth frame is processed for clarity restoration, the clarity of the object of interest in the output image can be improved without increasing the image acquisition cost. The visual perception system of the unmanned vehicle can accurately obtain the distance and position of the object of interest (obstacles, traffic lights) in front of the vehicle based on the output image. Therefore, based on this scheme, the detection accuracy of the visual perception system can be further improved, thereby improving the safety of unmanned vehicles and facilitating the promotion of unmanned vehicles. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in the embodiments of the present application. For ordinary technicians in this field, other drawings can also be obtained based on these drawings.

[0011] Figure 1A This is a flowchart of the steps of an image generation method according to Embodiment 1 of the present application;

[0012] Figure 1B for Figure 1A A schematic diagram of an example scenario in the illustrated embodiment;

[0013] Figure 2 This is a flowchart of the steps of an image generation method according to Embodiment 2 of the present application;

[0014] Figure 3A This is a flowchart of the steps of an image generation method in Embodiment 3 of the present application;

[0015] Figure 3B A trend graph of FOV and viewing distance of a lens;

[0016] Figure 3C This is a structural block diagram of an image generation device for a usage scenario of the third embodiment of the present application;

[0017] Figure 3D A schematic diagram of a flow chart of steps executed by an image generating device for a usage scenario of the third embodiment of the present application;

[0018] Figure 4 This is a structural block diagram of an image generating device according to the fourth embodiment of the present application;

[0019] Figure 5 This is a schematic diagram of the structure of an electronic device according to the fifth embodiment of the present application. DETAILED DESCRIPTION

[0020] In order to enable those skilled in the art to better understand the technical solutions in the embodiments of the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments in the embodiments of the present application should fall within the scope of protection of the embodiments of the present application.

[0021] The specific implementation of the embodiment of the present application is further explained below in conjunction with the accompanying drawings of the embodiment of the present application.

[0022] Embodiment 1

[0023] Figure 1A This is a flowchart of a method for generating an image according to the first embodiment of the present application. As shown in the figure, the method includes:

[0024] Step S101: Obtain the pixel region position range covered by the object of interest in the kth frame image.

[0025] In order to more clearly illustrate the solution of this application, the solution is explained in conjunction with the unmanned driving scenario below. It should be noted that the example of the unmanned driving scenario should not be regarded as a limitation on the application scenario of this application. The solution provided in this application can be applied to any scenario requiring image processing, including but not limited to the processing of images collected by devices such as mobile phones and driving recorders.

[0026] In this embodiment, images are collected by a visual sensor carried by the unmanned vehicle, where k is an integer and the carried visual sensor is a camera.

[0027] In the unmanned driving scenario, the sight distance is the distance of obstacles, traffic facilities, road markings, etc. in front of the vehicle that can be clearly seen from the driving position (or camera lens position) during normal driving. For the images captured by the camera, one frame of image can usually capture multiple objects. In this case, the sight distance refers to the distance from the camera lens to the corresponding object when the camera captures the frame of image. Since the distances from different objects to the lens are different, the sight distances that can clearly capture the object are also different.

[0028] Take the fixed-focus lens of a camera as an example. The inherent property of a fixed-focus lens is that it has a limited viewing distance. Affected by the focal length, the closer the viewing distance corresponding to the object is to the focal length, the higher the clarity of the image. Due to this characteristic of the fixed-focus lens, when using a fixed-focus lens to capture images, some objects in the same frame of image will be clearly imaged, while others will be unclear.

[0029] For example, a traffic light that is far away from the lens has a visual distance outside the focal length of the fixed-focus lens, so the image of the traffic light in the k-th frame image is very blurry and lacks clarity. If the subsequent decision-making system of the unmanned vehicle directly uses the k-th frame image to make driving decisions, it will be difficult to accurately obtain (perceive) the state of the traffic light (such as whether it is a red light) from the k-th frame image, which will result in an inability to correctly judge the driving environment.

[0030] In order to solve the problem that the visual perception ability of unmanned vehicles using fixed-focus lenses is insufficient, resulting in reduced driving safety of unmanned vehicles, in this embodiment, the pixel area position range covered by the object of interest in the kth frame image is obtained, so that the image corresponding to the pixel area position range covered by the object of interest can be subsequently processed, thereby making the image corresponding to the object of interest clearer to facilitate subsequent decision-making.

[0031] For ease of description, the pixel area covered by the object of interest may be referred to as a region of interest (RIO) including the object of interest in the k-th frame image. The object of interest may be a traffic light, a pedestrian, an obstacle, and the like.

[0032] The position range of the region of interest in the k-th frame image can be obtained by means of a high-precision map. For example, based on the information of the high-precision map, the object of interest in front of the vehicle and the spatial distance between the object of interest and the vehicle are determined. Based on the spatial distance and the focal length of the fixed-focus lens installed on the vehicle, the imaging position of the object of interest in the k-th frame image can be determined. The imaging position is the position range of the region of interest corresponding to the object of interest, that is, the position range of the pixel area covered by the object of interest in the k-th frame image.

[0033] Alternatively, the region of interest may also be determined based on the region of interest corresponding to the same object of interest in the output image of the k-1th frame, which is not limited in this embodiment.

[0034] It should be noted that the pixel area position range can be expressed by its coordinates in the kth frame image, such as the starting point coordinates are (x1, y1) and the end point coordinates are (x2, y2), or it can be expressed by the pixel ID of the starting pixel and the pixel ID of the end point pixel it contains.

[0035] Step S102: based on the pixel region position range, obtaining a sub-image to be processed from the k-th frame image.

[0036] According to the pixel region position range, an image corresponding to the pixel region position range may be cut out from the k-th frame image as a sub-image to be processed.

[0037] For example, the pixel area position range can be expressed as: the upper left corner coordinate is (10, 10), and the lower right corner coordinate is (20, 20). Based on this coordinate, the corresponding image can be cut out from the k-th frame image as the sub-image to be processed, or, based on this coordinate, the pixels included in the pixel area position range are determined, and then the information of these pixels (such as the color information of the pixels) is read from the camera, and the sub-image to be processed is determined based on the read pixel information.

[0038] Step S103: restore the clarity of the sub-image to be processed to obtain a target sub-image.

[0039] The clarity of the sub-image to be processed may be restored by a super-resolution restoration algorithm such as a bicubic interpolation algorithm or a deep learning method based on a neural network, and this embodiment does not limit this.

[0040] It should be noted that, in a feasible manner, step S102 and step S103 may be executed in series, that is, after obtaining a complete sub-image to be processed, the clarity thereof is restored.

[0041] In another feasible embodiment, step S102 and step S103 can be executed in parallel, that is, in the process of reading out the pixel information to obtain the sub-image to be processed in step S102, clarity restoration is performed based on the read pixel information. In this way, processing in pixels can be achieved, thereby improving the processing rate, reducing the time required to restore the clarity of the sub-image to be processed to obtain the target sub-image, and greatly improving the image processing efficiency.

[0042] For example, a parallel first process and a second process may be set in the autonomous driving system, wherein the first process is used to read pixel information in the sub-image to be processed, and the second process obtains the pixel information read from the first process, performs definition restoration on the pixel information, and obtains the target sub-image. The definition of the object of interest in the target sub-image is better than the definition of the object of interest in the sub-image to be processed.

[0043] Step S104: synthesize the target sub-image with the k-th frame image to obtain an output image of the k-th frame.

[0044] For example, the target sub-image is synthesized with the remaining image in the k-th frame image to obtain the output image of the k-th frame, wherein the remaining image refers to the image of the k-th frame image excluding the pixel area covered by the object of interest.

[0045] In the output image of the kth frame, objects at different viewing distances have clear images, so that when a camera with a fixed-focus lens is used to capture images, objects at different viewing distances in the same exposure frame can be clearly imaged in real time, which saves costs and ensures imaging effects.

[0046] The solution of this embodiment is exemplified below through a specific usage scenario.

[0047] For example, Figure 1B As shown, during the driving process of vehicle A, it is assumed that at the kth moment, there are objects such as traffic lights and roadblocks in front of vehicle A. The camera with a fixed-focus lens installed on vehicle A can collect the scene in front of vehicle A, and obtain the kth frame image at the kth moment, wherein the image of the roadblock in the kth frame image is clear because the roadblock is at the focal length of the fixed-focus lens, while the image of the traffic light in the kth frame image is blurred and lacks clarity because the traffic light is too far away from the fixed-focus lens.

[0048] In this usage scenario, the object of interest is the traffic light. Based on the positioning of vehicle A, the distance between the traffic light and vehicle A can be determined through high-precision maps, that is, the distance between the traffic light and the fixed-focus lens on vehicle A can be determined, and then the pixel area position range covered by the traffic light in the k-th frame image can be determined (i.e. Figure 1B The area enclosed by the dashed box).

[0049] Based on the pixel area position range, the sub-image to be processed is obtained from the k-th frame image. It should be noted that one way to obtain the sub-image to be processed is: after the camera completes exposure, in the process of reading the information of each pixel of the k-th frame image from the camera, when the pixels included in the pixel area position range are read, it is considered that the sub-image to be processed begins to be obtained. In the process of obtaining the sub-image to be processed, the clarity restoration can be performed synchronously, that is, each time a pixel in the sub-image to be processed is read, the clarity of the pixel is restored. In this way, when all the pixels of the sub-image to be processed are read, the clarity restoration can also be completed synchronously, and the target sub-image is obtained. Simultaneously here should be understood as a delay less than a set value (such as 1ms, 10ms, etc.). The clarity of the traffic lights in the target sub-image obtained in this way is better.

[0050] After the definition is restored to obtain the target sub-image, the target sub-image is synthesized with the remaining image in the k-th frame image to obtain the output image at the k-th moment, and the output image at the k-th moment includes clear imaging of the roadblock and the traffic light.

[0051] The solution provided in this embodiment obtains the pixel area position range covered by the object of interest in the kth frame image; based on the pixel area position range, obtains the sub-image to be processed from the kth image frame; restores the clarity of the sub-image to be processed to obtain the target sub-image; synthesizes the target sub-image with the kth frame image to obtain the output image of the kth frame. Since the sub-image to be processed corresponding to the object of interest in the output image of the kth frame is processed with clarity restoration, the clarity of the object of interest in the output image can be improved without increasing the image acquisition cost. The visual perception system of the unmanned vehicle can accurately obtain the distance and position of the object of interest (obstacles, traffic lights, etc.) in front of the vehicle based on the output image. Therefore, based on this solution, the detection accuracy of the visual perception system can be further improved, thereby improving the safety of unmanned vehicles and facilitating the promotion of unmanned vehicles.

[0052] The solution provided in this embodiment can be executed by any electronic device suitable for unmanned vehicles, such as a chip installed inside the unmanned vehicle, a mobile phone or computer connected to the unmanned vehicle for communication, etc.

[0053] Embodiment 2

[0054] Figure 2 This is a flowchart of the steps of an image generation method according to the second embodiment of the present application. The image generation method includes the aforementioned steps S101 to S104.

[0055] In this embodiment, an example is taken in which an image is collected by a visual sensor, that is, the kth frame of image is collected by a visual sensor.

[0056] Preferably, the visual sensor may include a fixed-focus lens, such as a camera including a fixed-focus lens. Compared with a camera equipped with a mechanical zoom lens, the fixed-focus lens configured in such a camera has lower requirements on the vehicle body structure, high adaptability to the vehicle vibration environment, and a lower price, which can reduce the cost of the autonomous driving vehicle.

[0057] In the case where the k-th frame image is collected by a visual sensor, if the object of interest is a geographical element, then in the first implementation method, step S101 can be implemented as follows: based on the longitude and latitude coordinates of the geographical element recorded in the electronic map data, the posture and positioning position of the visual sensor when the k-th frame image is collected are obtained, and the position range of the pixel area covered by the geographical element in the k-th frame image is determined, and the accuracy of the longitude and latitude coordinates of the geographical element meets the preset accuracy conditions.

[0058] Geographic elements can be elements with relatively fixed locations, such as buildings, traffic facilities along the road (such as traffic lights or road signs, etc.), etc.

[0059] The electronic map data records the longitude and latitude coordinates of the geographical elements, so that the longitude and latitude coordinates of the corresponding geographical elements can be directly found according to the electronic map.

[0060] The preset accuracy condition that the latitude and longitude coordinates of the geographic elements meet may be meter level, decimeter level, centimeter level, etc. In order to ensure the positioning accuracy of the pixel area position range, in this embodiment, the electronic map data may be high-precision map data, which can achieve centimeter level or below positioning, that is, the latitude and longitude coordinates of the geographic elements can meet the centimeter level accuracy condition.

[0061] In addition to road information, high-precision maps also include information about surrounding geographical elements related to traffic. Geographical elements may include obstacles on the road, traffic lights, signs, etc. The information about geographical elements may be their latitude and longitude coordinates, etc.

[0062] In this embodiment, the positioning position of the visual sensor can be determined according to the positioning of the vehicle, or can be determined by using a high-precision map.

[0063] The posture of the visual sensor can be determined based on a gyroscope or the like configured thereon.

[0064] Based on the positioning position of the visual sensor and the longitude and latitude coordinates of the geographical elements recorded in the electronic map data, the geographical elements within the shooting range of the visual sensor can be determined, and then the distance between the geographical elements and the visual sensor can also be determined. Combined with the posture of the visual sensor and the focal length of the fixed-focus lens in the visual sensor, the pixel area position range covered by the geographical elements in the kth frame image can be estimated.

[0065] In this process, since the accuracy of the latitude and longitude coordinates of the geographic elements in the electronic map data is high enough, the accuracy of the pixel area position range covered by the geographic elements in the k-th frame image estimated based on the coordinates will also be high.

[0066] In the second implementation, in order to further improve the accuracy of the pixel region position range, step S101 may further include the following sub-steps:

[0067] Sub-step S1011, based on the pixel area position range obtained by detecting the object of interest in the output image of the k-1th frame, predict the predicted pixel area position range and prediction confidence of the object of interest in the kth frame image.

[0068] For example, the trained neural network model is used to detect the output image of the k-1th frame to obtain the pixel area position range corresponding to the object of interest. At the same time, the neural network model can output the confidence (denoted as tg) corresponding to the detected pixel area position range, which can be used as the prediction confidence.

[0069] Based on the detected pixel area position range, the predicted pixel area position range covered by the object of interest in the k-th frame image can be predicted (recorded as the upper left corner point (xg1, yg1) and the lower right corner point (xg2, yg2)). Since the object of interest in the output image of the k-1th frame has been restored in clarity, the detection accuracy is higher, thereby ensuring the accuracy of the predicted pixel area position range of the object of interest in the k-th frame image.

[0070] Specifically, the method for obtaining the predicted pixel area position range can be: summing the pixel area position range detected in the output image of the k-1th frame and the inter-frame displacement correction value (also called inter-frame displacement vector) between the k-1th frame to the kth frame to obtain the predicted pixel area position range corresponding to the kth frame image.

[0071] The inter-frame displacement correction value between the k-1th frame and the kth frame is determined by the difference between the pixel area position range covered by the object of interest in the output image of the k-2th frame and the pixel area position range covered by the object of interest in the output image of the k-1th frame.

[0072] Sub-step S1012, based on the latitude and longitude coordinates of the geographic element recorded in the electronic map data, obtain the posture and positioning position of the visual sensor when collecting the k-th frame image, and determine the target pixel area position range and target confidence covered by the geographic element in the k-th frame image.

[0073] Specifically, the electronic map may be a high-precision map. When determining the target pixel area position range and target confidence, a matching model may be used, and the matching model may be obtained by training data such as the latitude and longitude coordinates of geographic elements in the electronic map data.

[0074] For example, the k-th frame image and the positioning position, posture and other data of the visual sensor are input into the matching model as input data, so that the matching model matches the target pixel area position range covered by the object of interest in the k-th frame image according to the input data (denoted as the upper left corner point (xd1, yd1) and the lower right corner point (xd2, yd2)), and outputs the target confidence (denoted as td).

[0075] Sub-step S1013, determining the pixel area position range covered by the object of interest in the k-th frame image from the predicted pixel area position range and the target pixel area position range according to the prediction confidence and the target confidence.

[0076] In a specific implementation, a pixel region position range corresponding to a higher confidence level between the prediction confidence level and the target confidence level is selected as the pixel region position range covered by the object of interest in the k-th frame image.

[0077] For example, if the prediction confidence is higher than the target confidence, the predicted pixel region position range is used as the pixel region position range covered by the object of interest in the kth frame image, and vice versa.

[0078] In this way, when the object of interest is a geographic feature, the position range of the pixel area covered by the object of interest in the kth frame can be accurately determined, thereby ensuring the accuracy of the obtained sub-image to be processed, and then ensuring the effect of clarity restoration, so that the output image of the kth frame can contain a clear image of the object of interest, achieving the effect of clearly imaging objects at different viewing distances in the same exposure frame.

[0079] Embodiment 3

[0080] Figure 3A The flowchart of the steps of the image generation method of the third embodiment of the present application is shown. The image generation method comprises:

[0081] Step S301: Obtain the pixel region position range covered by the object of interest in the kth frame image.

[0082] When the object of interest is a non-geographical element, step S301 may be implemented as: obtaining the pixel area position range covered by the preset non-geographical element in the kth frame image.

[0083] Non-geographic features can be features with unfixed locations, such as pedestrians and vehicles on the road.

[0084] When the object of interest is a non-geographical element, since its position is not fixed, the pixel area position range covered by the non-geographical element in the kth frame image can be estimated based on the pixel area position range covered by the non-geographical element in different frame images.

[0085] For example, the method in sub-step S1011 in the aforementioned second embodiment is used for determination. Since the object of interest in the output image of the k-1th frame has been restored in definition, the accuracy of the detected pixel area position range is guaranteed, thereby ensuring the accuracy of the prediction of the pixel area position range covered by the object of interest in the kth frame image.

[0086] Alternatively, in this embodiment, it is not necessary to distinguish whether the object of interest is a geographic element or a non-geographic element, and step S301 may include the following sub-steps:

[0087] Sub-step S3011: determining an inter-frame displacement correction value of the object of interest based on the output image of the k-1th frame and the output image of the k-2th frame.

[0088] Since the motion trajectory of the vehicle is continuous, the position change of the object of interest in the multiple frames of images collected by the vehicle's visual sensor is also continuous. Based on this principle, the inter-frame displacement correction value of the object of interest can be determined based on the output image of the k-1th frame and the output image of the k-2th frame.

[0089] The inter-frame displacement correction value is used to indicate the change in the position range of the pixel area covered by the object of interest in two adjacent frames of images caused by the displacement of the visual sensor relative to the object of interest.

[0090] The inter-frame displacement correction value may be the difference between the pixel area position range covered by the object of interest in the output image of the k-2th frame and the pixel area position range covered by the object of interest in the output image of the k-1th frame. Since both pixel area position ranges are known, the inter-frame displacement correction value can be calculated very conveniently. Since the time interval between two adjacent frames of images collected by the visual sensor is very short, the relative speed between the object of interest and the visual sensor has little effect on the inter-frame displacement correction value, thereby ensuring the accuracy of the inter-frame displacement correction value.

[0091] Sub-step S3012: based on the pixel area position range covered by the object of interest in the output image of the k-1th frame and the inter-frame displacement correction value, obtain the pixel area position range covered in the kth frame image of the object of interest.

[0092] The method for determining the position range of the pixel area covered by the object of interest in the output image of the k-1th frame has been described in the aforementioned second embodiment, so it will not be repeated here.

[0093] The pixel area position range covered by the object of interest in the k-th frame image can be obtained by summing the pixel area position range covered by the object of interest in the output image of the k-1th frame and the inter-frame displacement correction value.

[0094] For example, the object of interest is detected in the output image of the k-1th frame, and the upper left corner point in the pixel area position range covered by the traffic light is determined to be (x, y); according to the pixel area position range covered by the traffic light in the output image of the k-1 frame and the output image of the k-2th frame, the inter-frame displacement correction value of the imaging area of ​​the traffic light is determined to increase the horizontal coordinate by 10 pixels and decrease the vertical coordinate by 10 pixels, then it can be determined that the upper left corner point of the pixel area position range covered by the object of interest in the kth frame image is represented as (x+10, y-10).

[0095] In this way, no matter the object of interest is a geographic feature or a non-geographic feature, the location range of the pixel area it covers can be accurately determined, so it has better adaptability.

[0096] Step S302: Based on the pixel region position range, obtain a sub-image to be processed from the k-th frame image.

[0097] In a feasible manner, after the complete k-th frame image is read out from the visual sensor, a portion of the image indicated by the pixel area position range may be cut out from the k-th frame image as a sub-image to be processed.

[0098] Alternatively, in another feasible manner, in the process of reading pixel information of the kth frame image from the visual sensor, when pixels included in the pixel area position range are read, it is determined to start obtaining the sub-image to be processed, and at the same time, step S303 can be executed.

[0099] Step S303: Perform pixel-by-pixel clarity restoration processing on the sub-image to be processed to obtain a target sub-image.

[0100] Specifically, each pixel of the k-th frame image can be read out in sequence to determine the digital image corresponding to the k-th frame image. For example, if the k-th frame image is a 300*500 pixel picture, the upper left corner can be used as the starting pixel and the pixels can be read one by one until the 300*500 pixels are read out.

[0101] In this embodiment, when pixels located within the pixel area position range are read, clarity restoration processing is performed on the read pixels, which ensures that clarity restoration processing can be performed synchronously during the pixel reading process, reduces the time occupied by processing, and improves time utilization.

[0102] After completing the definition restoration process of each pixel in the sub-image to be processed, the target sub-image is obtained.

[0103] For example, the pixel region position range can be: a rectangular coordinate system is established with the lower left corner pixel of the kth frame image as the coordinate origin, the lower left corner coordinate of the pixel region position range is (200, 100), and the upper right corner coordinate is (230, 160).

[0104] When reading out each pixel of the k-th frame image from the visual sensor, it can be read row by row starting from the lower left corner; when the 200th pixel of the 100th row is read, that is, the pixel corresponding to the sub-image to be processed is read, the pixel is restored to clarity, and after reading the pixel, the next pixel can be read until the target sub-image is obtained.

[0105] Step S304: synthesize the target sub-image with the k-th frame image to obtain an output image of the k-th frame.

[0106] The specific implementation method of this step can refer to the above embodiment and will not be repeated here.

[0107] In this way, the clarity restoration process can be performed during the process of reading out the k-th frame image, so that the output image of the k-th frame including clear imaging of objects at different viewing distances can be quickly obtained.

[0108] The solution provided in this embodiment can be executed by any electronic device suitable for unmanned vehicles, such as a chip installed inside the vehicle, a mobile phone connected to the vehicle for communication, a computer, etc.

[0109] The following describes the implementation process of the method in combination with a specific usage scenario:

[0110] In this usage scenario, the method may be performed by an image generation device installed on a vehicle, and the image generation device may be a chip or circuit capable of implementing corresponding functions. In this usage scenario, the output image of the kth frame is obtained through the following steps.

[0111] In this usage scenario, image acquisition is performed through a visual sensor including a fixed-focus lens to obtain the kth frame image.

[0112] Those skilled in the art can select a fixed-focus lens with a suitable focal length according to the clear imaging distance requirement of autonomous driving and the pixel resolution of the visual sensor.

[0113] When the pixel resolution is determined, the relationship between the lens field of view (FOV) and the viewing distance is as follows: Figure 3B As shown in the figure, when the FOV is f1, the visible distance is d1, and when the FOV increases to f2, the visible distance decreases to d2. The horizontal FOV of the lens refers to the maximum field of view that can be captured in the horizontal direction. Figure 3BIt can be seen that the higher the ROI clear imaging distance requirement, the greater the visible distance, and the smaller the horizontal FOV of the lens.

[0114] The pixel information of the k-th frame image is stored in the visual sensor (i.e. Figure 3C The process of reading pixel information from the visual sensor and forming a frame of image output is the exposure process. In this usage scenario, in order to make the output image of the exposed k-th frame contain clear images of objects at different viewing distances, as shown in FIG. Figure 3D As shown, the image generating device performs the following steps:

[0115] S1. Based on the latitude and longitude coordinates of the geographic element recorded in the electronic map data, the posture and positioning position of the visual sensor when collecting the k-th frame image are obtained, and the target pixel area position range and target confidence td covered by the geographic element in the k-th frame image are determined.

[0116] The first candidate acquisition unit of the image generating device executes step S1 to obtain the target pixel region position range (expressed as the coordinates of the upper left corner point (xd, yd) for ease of description) and the corresponding target confidence td.

[0117] Specifically, the electronic map can be a high-precision map, and the matching model can be trained based on the latitude and longitude coordinates of the geographical elements in the high-precision map. By inputting the posture and positioning position of the visual sensor when collecting the k-th frame image as input data into the matching model, the matching model outputs the target pixel area position range (xd, yd) covered by the object of interest in the k-th frame image and the corresponding target confidence td based on the input data.

[0118] The first candidate acquisition unit outputs the target pixel area position range and the target confidence to the position information acquisition subunit of the image generation device.

[0119] S2. Based on the pixel area position range obtained by detecting the object of interest in the output image of the k-1th frame, predict the predicted pixel area position range and prediction confidence of the object of interest in the kth frame image.

[0120] The second candidate acquisition unit of the image generating device executes step S3 to obtain the predicted pixel region position range and the prediction confidence.

[0121] The predicted pixel region position range is determined based on the inter-frame displacement correction value and the pixel region position range covered by the object of interest in the output image of the k-1th frame.

[0122] The inter-frame displacement correction value is determined based on the output image of the k-1th frame and the output image of the k-2th frame.

[0123] Specifically, through deep learning, machine learning or traditional feature detection methods, object detection of interest (for example, detection of traffic lights, obstacles, etc.) is performed on the output image of the k-1th frame and the output image of the k-2th frame, and the pixel area position range covered by the object of interest in the output image of the k-1th frame and the pixel area position range covered by the object of interest in the output image of the k-2th frame are obtained, and then the difference between the two is taken to obtain the inter-frame displacement correction value.

[0124] Alternatively, information fusion may be performed on the object of interest in the output image of the k-1th frame and the output image of the k-2th frame by methods such as optical flow matching or motion estimation-based matching to predict the inter-frame displacement correction value of the object of interest.

[0125] The pixel area position range covered by the object of interest in the output image of the k-1th frame can be obtained by performing object of interest detection on it. At the same time, the corresponding confidence level will also be output during the object of interest detection, which can be used as the prediction confidence level.

[0126] The pixel area position range is predicted based on the inter-frame displacement correction value and the pixel area position range covered by the object of interest in the output image of the k-1th frame (expressed as the coordinates of its upper left corner point (xg, yg) for ease of description).

[0127] The specific correction formula can be as follows:

[0128] (x d,k ,y d,k )=(x d,k-1 ,y d,k-1 )+(Δx k,k-1 ,Δy k,k-1 )

[0129] Among them, (x d,k ,y d,k ) represents the coordinates of the upper left corner of the predicted pixel region in the k-th frame image, (x d,k-1 ,y d,k-1 ) represents the coordinate of the upper left corner of the pixel area covered by the object of interest in the output image of the k-1th frame, (Δx k,k-1 ,Δy k,k-1 ) represents the inter-frame displacement correction value of the object of interest from the k-1th frame to the kth frame.

[0130] The predicted pixel region position range and prediction confidence obtained by the second candidate acquisition unit may be output to the position information acquisition subunit.

[0131] S3. Determine the pixel area position range covered by the object of interest in the kth frame image from the predicted pixel area position range and the target area position range according to the prediction confidence and the target confidence.

[0132] The position information acquisition subunit determines the position range of the pixel area based on the prediction confidence and the target confidence.

[0133] The specific process can be expressed as the comparison process of the above two confidence levels, as follows:

[0134]

[0135] Among them, (x, y) represents the coordinates of the upper left corner point of the pixel area position range covered by the object of interest, (x g ,y g ) represents the coordinates of the upper left corner of the predicted pixel area position range, (x d ,y d ) represents the coordinates of the upper left corner of the target pixel area, t g represents the prediction confidence, t d The confidence level can be determined by a person skilled in the art according to the requirements, and this embodiment does not limit this.

[0136] After comparison, a corresponding pixel region position range with a higher confidence level is selected as the determined pixel region position range and output to the pixel determination unit.

[0137] S4. Obtain a sub-image to be processed from the read k-th frame image based on the pixel region position information, and perform definition restoration processing to obtain an output image of the k-th frame.

[0138] In this usage scenario, the sensor imaging input unit can read out the pixel information of the k-th frame image collected by the visual sensor one by one. The sensor imaging input unit can perform image processing on the read pixel information and then output it to the pixel determination unit and the image frame acquisition unit. Image processing can be determined as needed, but does not include operations such as scaling, watermark superposition, etc. that cause changes in the content of the k-th frame image.

[0139] The pixel determination unit determines the corresponding pixel ID according to the pixel area position range. When the sensor imaging input unit reads the required pixels, it transmits these pixels to the pixel determination unit. The pixel determination unit determines to obtain the sub-image to be processed, and outputs the pixels in the sub-image to be processed to the restoration processing sub-unit for clarity restoration processing. The restoration processing sub-unit obtains the target sub-image through clarity restoration processing and outputs it to the image frame acquisition unit.

[0140] The pixels required by the non-pixel determination unit read by the sensor imaging input unit can be directly output to the image frame acquisition unit, which merges these pixels with the pixels in the target sub-image to form an output image of the kth frame, and transmits it to the image output unit, which outputs it to the video encoder.

[0141] Specifically, in units of pixels, each pixel of the k-th frame image is read in sequence, and at time T0, the first pixel in the sub-image to be processed, i.e., Pixel start, is read out, and pixel definition restoration processing is performed on it at this time, and pixel reading operations can be performed in parallel, for example, while pixel definition restoration processing is performed on Pixel start, an operation of reading pixel Pixel start+1 is performed in parallel. Pixel definition restoration processing can be an image processing method such as super-resolution restoration.

[0142] After all pixels in the sub-image to be processed are processed for clarity restoration and a target sub-image is obtained, the target sub-image can be merged with the remaining parts.

[0143] For example, the merge can be done as follows:

[0144] 1) Overlap the target sub-image after the definition is restored with the sub-image to be processed before the definition is restored in the k-th frame image;

[0145] 2) The upper left corner point of the target sub-image after the definition is restored is overlapped with the upper left corner point of the sub-image to be processed before the definition is restored;

[0146] 3) The upper right corner point of the target sub-image after the definition is restored is overlapped with the upper right corner point of the sub-image to be processed before the definition is restored;

[0147] 4) The lower left corner point of the target sub-image after the definition is restored is overlapped with the lower left corner point of the sub-image to be processed before the definition is restored;

[0148] 5) The lower right corner point of the target sub-image after the definition is restored is overlapped with the lower right corner point of the sub-image to be processed before the definition is restored, and so on.

[0149] In addition, the definition-restored target sub-image can be output as valid image data during the blanking period of the k-th frame image.

[0150] In this usage scenario, a fixed-focus lens is used to achieve clear imaging of both distant (the viewing distance range where the object of interest is located) and near (the viewing distance range where the fixed-focus lens can clearly image) fields of view at the same exposure time (corresponding to the k-th frame output image), thereby improving the target detection accuracy of the perception system at a low cost and further enhancing the safety of autonomous driving.

[0151] The solution provided in this embodiment can be executed by any electronic device suitable for unmanned vehicles, such as a chip installed inside an unmanned vehicle, a mobile phone connected to the unmanned vehicle for communication, a computer, etc. In addition, the image generation device used in the unmanned vehicle for executing the above-mentioned image generation method can be an FPGA, an ASIC, a GPU, etc., or a multi-chip device synthesized by two or more of the above chips.

[0152] Embodiment 4

[0153] Figure 4 This is a structural block diagram of an image generating device according to the fourth embodiment of the present application, as shown in the figure, which includes:

[0154] The position information acquisition module 401 is used to obtain the pixel area position range covered by the object of interest in the kth frame image;

[0155] A first sub-image acquisition module 402, configured to acquire a sub-image to be processed from the k-th frame image based on the pixel region position range;

[0156] The second sub-image acquisition module 403 is used to restore the clarity of the sub-image to be processed to obtain a target sub-image;

[0157] The synthesis module 404 is used to synthesize the target sub-image with the k-th frame image to obtain the k-th frame output image.

[0158] Optionally, the object of interest is a geographic element, and the location information acquisition module 401 is used to determine the position range of the pixel area covered by the geographic element in the kth frame image based on the latitude and longitude coordinates of the geographic element recorded in the electronic map data and the posture and positioning position of the visual sensor when the kth frame image is acquired, and the accuracy of the latitude and longitude coordinates of the geographic element meets the preset accuracy conditions.

[0159] Optionally, the object of interest is a non-geographic element, and the position information acquisition module 401 is used to obtain the position range of the pixel area covered by the preset non-geographic element in the kth frame image.

[0160] Optionally, the position information acquisition module 401 is used to determine the inter-frame displacement correction value of the object of interest based on the output image of the k-1th frame and the output image of the k-2th frame; based on the pixel area position range covered by the object of interest in the output image of the k-1th frame and the inter-frame displacement correction value, obtain the pixel area position range covered by the object of interest in the k-th frame image.

[0161] Optionally, the object of interest is a geographic element, and the location information acquisition module 401 includes:

[0162] The first acquisition module 4011 is used to predict the predicted pixel area position range and prediction confidence of the object of interest in the k-th frame image based on the pixel area position range obtained by performing object of interest detection on the output image of the k-1-th frame;

[0163] The second acquisition module 4012 is used to obtain the attitude and positioning position of the visual sensor when collecting the k-th frame of image based on the latitude and longitude coordinates of the geographical element recorded in the electronic map data, and determine the target pixel area position range and target confidence covered by the geographical element in the k-th frame of image;

[0164] The third acquisition module 4013 is used to determine the pixel area position range covered by the object of interest in the kth frame image from the predicted pixel area position range and the target pixel area position range according to the prediction confidence and the target confidence.

[0165] Optionally, the second sub-image acquisition module 403 is used to perform pixel-by-pixel definition restoration processing on the sub-image to be processed to obtain a target sub-image.

[0166] Optionally, the image is collected by a visual sensor, and the visual sensor includes a fixed-focus lens.

[0167] The image generation device of the embodiment of the present application is used to implement the corresponding image generation method in the aforementioned multiple method embodiments, and has the beneficial effects of the corresponding method embodiments, which will not be described in detail here. In addition, the functional implementation of each module in the image generation device of this embodiment can refer to the description of the corresponding part in the aforementioned method embodiments, which will not be described in detail here.

[0168] Reference Figure 5 , shows a schematic diagram of the structure of an electronic device according to Embodiment 5 of the present application. The specific embodiments of the present application do not limit the specific implementation of the electronic device.

[0169] like Figure 5 As shown, the electronic device may include: a processor (processor) 502 , a communication interface (Communications Interface) 504 , a memory (memory) 506 , and a communication bus 508 .

[0170] in:

[0171] The processor 502 , the communication interface 504 , and the memory 506 communicate with each other via a communication bus 508 .

[0172] The communication interface 504 is used to communicate with other electronic devices or servers.

[0173] The processor 502 is used to execute the program 510, and specifically can execute the relevant steps in the above-mentioned image generation method embodiment.

[0174] Specifically, the program 510 may include program codes, which include computer operation instructions.

[0175] The processor 502 may be a central processing unit (CPU), or an application-specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of the present application. The one or more processors included in the smart device may be processors of the same type, such as one or more CPUs; or processors of different types, such as one or more CPUs and one or more ASICs.

[0176] The memory 506 is used to store the program 510. The memory 506 may include a high-speed RAM memory, and may also include a non-volatile memory (non-volatile memory), such as at least one disk memory.

[0177] The specific implementation of each step in program 510 can refer to the corresponding description of the corresponding steps and units in the above-mentioned image generation method embodiment, which will not be repeated here. Those skilled in the art can clearly understand that for the convenience and simplicity of description, the specific working process of the above-described devices and modules can refer to the corresponding process description in the above-mentioned method embodiment, which will not be repeated here.

[0178] It should be pointed out that, according to the needs of implementation, the various components / steps described in the embodiments of the present application can be split into more components / steps, or two or more components / steps or partial operations of components / steps can be combined into new components / steps to achieve the purpose of the embodiments of the present application.

[0179] The above-mentioned method according to the embodiment of the present application can be implemented in hardware, firmware, or implemented as software or computer code that can be stored in a recording medium (such as a CD ROM, RAM, floppy disk, hard disk or magneto-optical disk), or implemented as a computer code originally stored in a remote recording medium or a non-temporary machine-readable medium downloaded through a network and to be stored in a local recording medium, so that the method described herein can be stored in such software processing on a recording medium using a general-purpose computer, a dedicated processor or programmable or dedicated hardware (such as an ASIC or FPGA). It can be understood that a computer, a processor, a microprocessor controller or programmable hardware includes a storage component (e.g., RAM, ROM, flash memory, etc.) that can store or receive software or computer code, and when the software or computer code is accessed and executed by a computer, a processor or hardware, the image generation method described herein is implemented. In addition, when a general-purpose computer accesses the code for implementing the image generation method shown here, the execution of the code converts the general-purpose computer into a dedicated computer for executing the image generation method shown here.

[0180] Those of ordinary skill in the art will appreciate that the units and method steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the embodiments of the present application.

[0181] The above implementation methods are only used to illustrate the embodiments of the present application, and are not limitations on the embodiments of the present application. Ordinary technicians in the relevant technical field can make various changes and modifications without departing from the spirit and scope of the embodiments of the present application. Therefore, all equivalent technical solutions also belong to the scope of the embodiments of the present application. The scope of patent protection of the embodiments of the present application should be limited by the claims.

Claims

1. A method for generating an image, comprising: Obtain the pixel area position range covered by the object of interest in the kth frame image; Based on the pixel region position range, obtaining a sub-image to be processed from the k-th frame image; Restoring the clarity of the sub-image to be processed to obtain a target sub-image; Synthesize the target sub-image with the k-th frame image to obtain an output image of the k-th frame; Wherein, if the object of interest is a geographic element, the step of obtaining the pixel area position range covered by the object of interest in the k-th frame image includes: Based on the pixel area position range obtained by detecting the object of interest on the output image of the k-1th frame, the predicted pixel area position range and the prediction confidence of the object of interest in the kth frame image are predicted; based on the latitude and longitude coordinates of the geographical element recorded in the electronic map data, the posture and positioning position of the visual sensor when the kth frame image is acquired are obtained, and the target pixel area position range and the target confidence covered by the geographical element in the kth frame image are determined; according to the prediction confidence and the target confidence, the pixel area position range covered by the object of interest in the kth frame image is determined from the predicted pixel area position range and the target pixel area position range.

2. The method according to claim 1, wherein: The accuracy of the latitude and longitude coordinates of the geographic element meets the preset accuracy conditions.

3. The method according to claim 1, wherein: If the object of interest is a non-geographical element, the step of obtaining the pixel area position range covered by the object of interest in the k-th frame image includes: Get the pixel area position range covered by the preset non-geographical feature in the k-th frame image.

4. The method according to claim 1, wherein: The step of obtaining the pixel region position range covered by the object of interest in the kth frame image includes: Determine an inter-frame displacement correction value of the object of interest based on an output image of the k-1th frame and an output image of the k-2th frame; Based on the pixel area position range covered by the object of interest in the output image of the k-1th frame and the inter-frame displacement correction value, the pixel area position range covered by the object of interest in the kth frame image is obtained.

5. The method according to any one of claims 1 to 4, wherein: The step of restoring the clarity of the sub-image to be processed to obtain a target sub-image includes: The sub-image to be processed is processed pixel by pixel to restore the clarity of the sub-image, so as to obtain a target sub-image.

6. The method according to any one of claims 1 to 4, wherein: The image is collected by a visual sensor, and the visual sensor includes a fixed-focus lens.

7. An image generating device, comprising: A position information acquisition module is used to obtain the pixel area position range covered by the object of interest in the kth frame image; A first sub-image acquisition module, configured to acquire a sub-image to be processed from the k-th frame image based on the pixel region position range; A second sub-image acquisition module is used to restore the clarity of the sub-image to be processed to obtain a target sub-image; A synthesis module is used to synthesize the target sub-image with the k-th frame image to obtain an output image of the k-th frame, Among them, the position information acquisition module is specifically used to, if the object of interest is a geographic element, predict the predicted pixel area position range and prediction confidence of the object of interest in the k-th frame image based on the pixel area position range obtained by detecting the object of interest on the output image of the k-1th frame; determine the target pixel area position range and target confidence covered by the geographic element in the k-th frame image based on the latitude and longitude coordinates of the geographic element recorded in the electronic map data and the posture and positioning position of the visual sensor when the k-th frame image is acquired; and determine the pixel area position range covered by the object of interest in the k-th frame image from the predicted pixel area position range and the target pixel area position range according to the prediction confidence and the target confidence.

8. An electronic device, comprising: A processor, a memory, a communication interface and a communication bus, wherein the processor, the memory and the communication interface communicate with each other via the communication bus; The memory is used to store at least one executable instruction, and the executable instruction enables the processor to perform an operation corresponding to the image generation method according to any one of claims 1 to 6.

9. A computer storage medium having a computer program stored thereon, wherein when the program is executed by a processor, the image generating method according to any one of claims 1 to 6 is implemented.

Images