Blind area filling method and device, electronic equipment, storage medium and product

Abstract

The invention discloses a blind area filling method and device, electronic equipment, a storage medium and a product, and relates to the technical field of automobile auxiliary driving. The blind area filling method comprises the following steps: when a historical reference point exceeds a preset range, determining a target reference point according to a preset distance; determining the relative position of the vehicle and the target reference point, and determining the real-time position of the vehicle according to the relative position and the position information of the target reference point in the scene map; acquiring a vehicle surrounding original image of the vehicle at the real-time position, and determining an all-round spliced top view according to the vehicle surrounding original image; and determining a target area image in the scene map according to the look-around splicing top view, and filling a blind area position in the look-around splicing top view according to the target area image. According to the embodiment of the invention, the image data for blind area filling is obtained from the scene map, and the efficiency and accuracy of blind area filling are improved.

Description

Technical Field

[0001] The present invention relates to the field of automobile assisted driving technology, and in particular to a blind spot filling method, device, electronic equipment, storage medium and product. Background Art

[0002] Blind spot filling is the process of filling in blind spots (areas below the vehicle that are not captured by the camera) in the surround view top-down view to obtain a complete top-down view. This allows the driver to clearly understand the conditions under and around the vehicle, which were previously invisible, and to proactively detect potential hazards such as potholes and rocks, thus preventing damage to vehicle components. It also allows the driver to promptly detect pedestrians and small animals in the blind spot when reversing, parking, or driving at low speeds, reducing the risk of collision.

[0003] Existing blind spot filling methods rely on keyframes and transformation matrices to map the pixel coordinates of the blind spot to the corresponding positions in the keyframe. These mapped pixels are then filled into the current frame. This requires a high level of computation and consumes significant hardware resources. Furthermore, existing methods rely heavily on the accuracy of the transformation matrix, and errors in the transformation matrix are directly reflected in the filling effect, making it difficult to achieve high-quality filling results. Therefore, finding an accurate and convenient way to fill blind spots has become a pressing issue. Summary of the Invention

[0004] The present invention provides a blind spot filling method, device, electronic device, storage medium and product to solve the problem of inaccurate and inconvenient blind spot filling in the prior art.

[0005] According to one aspect of the present invention, a blind spot filling method is provided, characterized by comprising:

[0006] When the historical reference point exceeds the preset range, the target reference point is determined according to the preset distance;

[0007] Determining a relative position between the vehicle and the target reference point, and determining a real-time position of the vehicle based on the relative position and position information of the target reference point in a scene map;

[0008] Acquire original images of the vehicle surrounding the vehicle at the real-time position, and determine a surround stitching top view based on the original images of the vehicle surrounding the vehicle;

[0009] A target area image in the scene map is determined according to the surround-view spliced top view, and a blind area position in the surround-view spliced top view is filled according to the target area image.

[0010] According to another aspect of the present invention, a blind spot filling device is provided, characterized by comprising:

[0011] A reference point determination module is used to determine a target reference point according to a preset distance when a historical reference point exceeds a preset range;

[0012] a position determination module, configured to determine a relative position between the vehicle and the target reference point, and determine a real-time position of the vehicle based on the relative position and position information of the target reference point in a scene map;

[0013] A top-view position determination module is used to obtain an original image of the vehicle surrounding the vehicle at the real-time position, and determine a surround stitching top view based on the original image of the vehicle surrounding;

[0014] The blind spot filling module is used to determine the target area image in the scene map according to the surround-view spliced top view, and fill the blind spot position in the surround-view spliced top view according to the target area image.

[0015] According to another aspect of the present invention, an electronic device is provided, comprising:

[0016] at least one processor; and

[0017] a memory communicatively connected to the at least one processor; wherein,

[0018] The memory stores a computer program that can be executed by the at least one processor. The computer program is executed by the at least one processor to enable the at least one processor to perform a blind spot filling method described in any embodiment of the present invention.

[0019] According to another aspect of the present invention, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement a blind spot filling method according to any embodiment of the present invention when executed.

[0020] According to another aspect of the present invention, an embodiment of the present invention further provides a computer program product, which includes a computer program. When the computer program is executed by a processor, it implements a blind spot filling method of any embodiment of the present invention.

[0021] The technical solution of the embodiment of the present invention improves the vehicle position accuracy by determining the target reference point according to a preset distance when the historical reference point exceeds a preset range, determining the relative position of the vehicle and the target reference point, and determining the real-time position of the vehicle based on the relative position and the position information of the target reference point in the scene map; obtaining the original image of the vehicle surrounding the vehicle at the real-time position, determining a surround view stitching top view based on the original image of the vehicle surrounding, determining the target area image in the scene map based on the surround view stitching top view, filling the blind spot position in the surround view stitching top view according to the target area image, and realizing that the image data for blind spot filling is obtained from the scene map, thereby improving the efficiency and accuracy of blind spot filling.

[0022] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present invention, nor is it intended to limit the scope of the present invention. Other features of the present invention will become readily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0024] Figure 1 This is a flowchart of a blind spot filling method provided according to the first embodiment of the present invention;

[0025] Figure 2 This is a flow chart of a blind spot filling method provided according to the second embodiment of the present invention;

[0026] Figure 3 is a schematic diagram of a panoramic splicing top view provided according to the second embodiment of the present invention;

[0027] Figure 4 This is a flowchart of a blind area filling method provided according to the third embodiment of the present invention;

[0028] Figure 5 is a schematic diagram of a blind spot filling method provided according to the third embodiment of the present invention;

[0029] Figure 6 This is a schematic diagram of a process for filling a blind spot position according to a third embodiment of the present invention;

[0030] Figure 7 This is a structural diagram of a blind spot filling device provided according to a fourth embodiment of the present invention;

[0031] Figure 8It is a structural diagram of an electronic device for implementing a blind area filling method according to an embodiment of the present invention. DETAILED DESCRIPTION

[0032] In order to enable those skilled in the art to better understand the solutions of the present invention, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the embodiments described are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts should fall within the scope of protection of the present invention.

[0033] It should be noted that the terms "first", "second", etc. in the description and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that the numbers used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0034] Example 1

[0035] Figure 1 This is a flow chart of a blind spot filling method provided according to the first embodiment of the present invention. This embodiment is applicable to filling blind spots during vehicle driving. The method can be executed by a blind spot filling device. The blind spot filling device can be implemented in the form of hardware and / or software. The blind spot filling device can be configured in an electronic device, such as a vehicle. Figure 1 As shown, the method includes:

[0036] S110 : When the historical reference point exceeds a preset range, determine a target reference point according to a preset distance.

[0037] Among them, the historical reference point refers to the position reference point of the vehicle at a historical moment, and the target reference point refers to the position reference point of the vehicle at the current moment. Generally speaking, the historical reference point and the target reference point are determined based on the preset distance between them and the vehicle. The historical reference point and the target reference point can be specific signs in the vehicle's road, such as parking spaces, lane lines, etc. The preset range refers to a preset critical value used to determine whether the target reference point needs to be replaced. The preset distance refers to the preset distance between the vehicle and the reference point. The target reference point can be set according to the preset distance. In actual operation, the preset distance can be greater than the preset range. That is to say, when the historical reference point exceeds the preset range, the historical reference point can still be used to determine the real-time position of the vehicle.

[0038] In an embodiment, it is possible to determine whether the historical reference points are outside a preset range. If so, the target reference point can be determined by filtering based on a preset distance. In actual operation, all reference points within the preset distance can be determined, with the closest reference point being the target reference point. Alternatively, a priority can be determined for each reference point, with the reference point with the highest priority being the target reference point.

[0039] S120: Determine the relative position of the vehicle and the target reference point, and determine the real-time position of the vehicle based on the relative position and the position information of the target reference point in the scene map.

[0040] Among them, relative position refers to the positional relationship between the selected reference point and the vehicle, and is the orientation and distance of the target reference point relative to the vehicle. The scene map still refers to a top view of the large scene of vehicle operation. In one embodiment, the position when the blind spot filling function is turned on can be used as the origin. At this time, the direction of the front of the vehicle is the positive x direction, and the direction of the left side of the vehicle perpendicular to the body is the positive y direction. The position information refers to the position of the target reference point in the scene map. During actual operation, the position information can be the coordinate information in the coordinate system of the scene map. The real-time position refers to the driving position of the vehicle.

[0041] In an embodiment, the relative position of the vehicle and the target reference point can be obtained by a preset sensor. Exemplarily, the preset sensor may include but is not limited to a depth camera, a laser, a radar, etc. The preset sensor can be installed in any unobstructed position of the vehicle, such as the front of the vehicle, the side door of the vehicle, etc. In the actual operation process, the distance between the target reference point and the vehicle and the angle with the direction of travel of the vehicle can be determined, and the relative position of the vehicle and the target reference point can be determined according to the distance and the angle. In one embodiment, the relative position of the target reference point and the vehicle can be determined as (Δx1, Δy1) with the vehicle as the origin, the vehicle position (x0, y0) as (0, 0), and the vehicle heading angle as angle0 = 0°. After determining the relative position, the position information of the target reference point in the scene map can be extracted, and the real-time position of the vehicle can be calculated according to the position information and the relative position. Generally speaking, the horizontal and vertical coordinates of the position information and the relative position can be added together as the real-time position of the vehicle.

[0042] S130: Acquire original images of the surroundings of the vehicle at the real-time position of the vehicle, and determine a surround stitching top view based on the original images of the surroundings of the vehicle.

[0043] The "vehicle surrounding raw image" refers to the raw image of the vehicle's surroundings. In actual operation, multiple cameras can capture images of the vehicle's surroundings as the raw image. The "surround view stitching overhead view" is a method of stitching raw images of the vehicle's surroundings captured by multiple cameras to generate an overhead view of the vehicle's surroundings.

[0044] In an embodiment, multiple cameras can be used to capture raw images of the surroundings of the vehicle at its real-time location as raw images of the surroundings of the vehicle. These raw images of the surroundings of the vehicle can then be stitched together to produce a surround stitched top view. In actual operation, four cameras are typically mounted around the vehicle, capturing raw images of the surroundings of the vehicle from the front, rear, left, and right directions. These images are then stitched together to produce a surround stitched top view based on the positions of the raw images of the surroundings of the vehicle. Since the surround stitched top view is a stitching of images of the surroundings of the vehicle, the center of the surround stitched top view is at the vehicle's position, i.e., the blind spot. In other words, the surround stitched top view includes the position.

[0045] S140: Determine a target area image in the scene map according to the surround-view stitched top view, and fill in a blind spot position in the surround-view stitched top view according to the target area image.

[0046] The target area image refers to the image corresponding to the surround stitching top view in the scene map. The blind spot position refers to the blind spot portion of the surround stitching top view, that is, the location of the vehicle.

[0047] In an embodiment, the position area of the surround stitching top view in the scene map is determined. In actual operation, it can be determined by the markers in the surround stitching top view; or it can be determined by the coordinate position of the surround stitching top view in the scene map, and the image corresponding to the position area is determined as the target area image. The position in the target area image corresponding to the blind spot position in the surround stitching top view is determined as the target area, and the image of the target area is extracted to fill the blind spot position. In actual operation, the relative position of all pixels in the partial image covered by the vehicle body projection in the scene map and the vehicle body can be calculated and filled into the corresponding position of the blind spot in the surround stitching top view. Generally speaking, the blind spot position can be filled in each frame separately.

[0048] In an embodiment of the present invention, when a historical reference point exceeds a preset range, a target reference point is determined according to a preset distance, the relative position of the vehicle and the target reference point is determined, and the real-time position of the vehicle is determined according to the relative position and the position information of the target reference point in the scene map, thereby improving the vehicle position accuracy; by obtaining original images of the vehicle's surroundings at the real-time position, determining a surround view stitching top view according to the original images of the vehicle's surroundings, determining a target area image in the scene map according to the surround view stitching top view, filling the blind spot position in the surround view stitching top view according to the target area image, and realizing that the image data for blind spot filling is obtained from the scene map, thereby improving the efficiency and accuracy of blind spot filling.

[0049] In one embodiment, before determining the target reference point according to the preset distance, the method further includes:

[0050] Determining a relative distance between the vehicle and the historical reference point, and determining that the historical reference point is out of the preset range when the relative distance is determined to be greater than a preset range;

[0051] When it is determined that the relative distance is less than or equal to the preset range, it is determined that the historical reference point does not exceed the preset range, and the historical reference point is used as the target reference point.

[0052] In an embodiment, the relative distance between the vehicle and the historical reference point can be determined. When the relative distance is greater than a preset range, the historical reference point can be considered to be outside the preset range. When the relative distance is less than or equal to the preset range, it can be determined that the historical reference point is within the preset range. When the historical reference point is within the preset range, the historical reference point can be used as the target reference point.

[0053] In one embodiment, after filling the blind spot position in the surround view stitched top view according to the target area image, the method further includes:

[0054] Stores the association between the surround stitching top view, the image to be filled in the blind spot position, and the real-time position.

[0055] In an embodiment, the surround view stitching top view and the image to be filled in the blind spot position can be associated with the real-time position and stored, so that when the vehicle arrives at the real-time position, the corresponding surround view stitching top view and the image to be filled in the blind spot position can be extracted in time, avoiding frequent acquisition of original images around the vehicle and stitching of the surround view stitching top view, thereby reducing calculation time and occupied hardware resources.

[0056] In one embodiment, before obtaining the original images of the surroundings of the vehicle at the real-time position and determining the surround stitching top view based on the original images of the surroundings of the vehicle, the method further includes:

[0057] Determining whether there is a look-around stitched overhead view associated with the real-time location;

[0058] If it exists, the image to be filled associated with the real-time position is extracted, and the image to be filled is filled into the surround stitching top view;

[0059] If it does not exist, the original image of the vehicle surrounding the vehicle at the real-time position is obtained, and the surround stitching top view is determined according to the original image of the vehicle surrounding.

[0060] In an embodiment, before obtaining the original images around the vehicle at the real-time position of the vehicle and determining the surround-view stitching top view based on the original images around the vehicle, it is possible to query whether there is a surround-view stitching top view associated with the real-time position according to the real-time position. If so, the image to be filled associated with the real-time position can be directly extracted and filled into the surround-view stitching top view with the image to be filled. If not, the original images around the vehicle can be obtained again according to the real-time position, and the original images around the vehicle can be stitched to determine the surround-view stitching top view to fill in the blind spots, thereby improving the efficiency of blind spot filling and reducing the occupation of computing resources.

[0061] Example 2

[0062] Figure 2 This is a flow chart of a blind spot filling method provided according to the second embodiment of the present invention. This embodiment is based on the above embodiment to further optimize and expand, and can be combined with various optional technical solutions in the above embodiment. Figure 2 As shown, the method includes:

[0063] S210: When the historical reference points are outside the preset range, determine reference points of the vehicle within the preset distance, and determine the distance between each reference point and the vehicle.

[0064] In an embodiment, when a historical reference point exceeds a preset range, all reference points within a preset distance of the vehicle can be determined, and the distance between each reference point and the vehicle can be determined in sequence. In actual operation, the distance between the reference point and the vehicle can be directly determined using sensors such as depth cameras and lidar.

[0065] S220: Use the reference point corresponding to the minimum distance as the target reference point.

[0066] In an embodiment, a reference point corresponding to a minimum distance value may be determined as the target reference point, that is, the reference point closest to the vehicle may be determined as the target reference point.

[0067] S230: Establish a physical coordinate system with the vehicle position as a reference, determine the distance and angle between the vehicle and the target reference point through a preset vehicle sensor, and determine the relative position of the vehicle and the target reference point according to the distance and angle.

[0068] In an embodiment, a vehicle coordinate system can be established with the vehicle position as the origin and the vehicle orientation angle as angle0=0°. At this time, the coordinates of the vehicle are (0,0). The distance and angle between the vehicle and the target reference point can be determined by the preset vehicle sensors, and then the relative position of the vehicle and the target reference point can be determined based on the distance and angle. In actual operation, the preset vehicle sensors may include but are not limited to depth cameras, lidars and other sensors. The preset vehicle sensors can be installed in any unobstructed position of the vehicle, such as the front and side of the vehicle. Based on the distance and angle between the vehicle and the target reference point, the relative coordinates (Δx1, Δy1) of the vehicle and the target reference point can be determined, and the relative coordinates can be used as the relative position of the vehicle and the target reference point.

[0069] S240: Extract location information of the target reference point in the scene map.

[0070] In an embodiment, a coordinate system may be pre-established for the scene map, and corresponding coordinate information may exist at any point in the scene map. The coordinate information of the target reference point in the scene map may be extracted as position information.

[0071] S250: Calculate the position of the vehicle in the scene map as the real-time position according to the position information and the relative position.

[0072] In an embodiment, the position of the vehicle in the scene map can be calculated using the position information and the relative position, and the calculated position of the vehicle in the scene map is used as the real-time position. For example, when the position information is (x1, y1) and the relative position is (Δx1, Δy1), the real-time position is (x', y'), where x'=x1+Δx1 and y'=y1+Δy1.

[0073] S260: Collecting an original image around the vehicle at the real-time position through a preset visual sensor as an original image around the vehicle.

[0074] In an embodiment, a preset visual sensor can be used to capture raw images of the vehicle's surroundings, and the raw images can be used as the vehicle's surrounding raw images. In actual operation, the preset visual sensor can be a camera, and the number of preset visual sensors is not limited, as long as all the raw images can be stitched together to include the entire vehicle's surroundings. For example, preset visual sensors can be installed on the front, back, left, and right sides of the vehicle, and the preset visual sensors can capture raw images of the vehicle's surroundings at real-time locations to obtain the vehicle's surrounding raw images.

[0075] S270: Splice the original images around the vehicle according to the image positions to obtain a surround stitching top view.

[0076] Among them, the blank position in the surround stitching top view is the blind spot position.

[0077] In an embodiment, the image position of each original image around the vehicle can be determined, and the original images around the vehicle can be spliced according to the image position to obtain a surround view spliced top view. For example, when the original images around the vehicle are collected by the front, back, left, and right visual sensors of the vehicle respectively, the original images around the vehicle can be spliced in the front, back, left, and right manner to obtain a surround view spliced top view. In one embodiment, Figure 3 3 is a schematic diagram of a surround-view splicing top view provided according to the second embodiment of the present invention, wherein the black blank positions are blind spots.

[0078] S280: Determine the location area of the surround stitching top view in the scene map, and determine an image corresponding to the location area as a target area image.

[0079] In an embodiment, the location area corresponding to the surround view stitched top view can be determined based on the vehicle's real-time location, and the image corresponding to the location area can be extracted as the target area image. In actual operation, the location coordinates of the edges of the surround view stitched top view can be calculated based on the vehicle's real-time location, and the area consisting of the corresponding location coordinates in the scene map can be determined as the location area. Alternatively, landmarks can be identified in the surround view stitched top view, and the location area of the surround view stitched top view in the scene map can be searched based on the landmarks.

[0080] S290: Determine the target position in the target area image according to the blind spot position in the surround stitching top view, use the image corresponding to the target position as the image to be filled, and calculate the blind spot position of the image to be filled.

[0081] In an embodiment, the position information of the blind spot position in the surround stitched top view can be determined, and the target position in the target area image can be determined according to the position information of the blind spot position. The image corresponding to the target position can be used as the image to be filled to fill the blind spot position. In actual operation, the surround stitched top view and the target area image can be overlapped, and the area corresponding to the portion of the image covered by the blind spot position can be extracted as the target position, and the corresponding image can be used as the image to be filled, and the blind spot position of the image to be filled can be filled.

[0082] In an embodiment of the present invention, when a historical reference point exceeds a preset range, a reference point of the vehicle is determined within a preset distance, the distance between each reference point and the vehicle is determined, and the reference point corresponding to the minimum value of the distance is used as the target reference point. A physical coordinate system with the vehicle position as a reference is established, the distance and angle between the vehicle and the target reference point are determined by a preset vehicle sensor, the relative position of the vehicle and the target reference point is determined according to the distance and angle, the position information of the target reference point in the scene map is extracted, and the position of the vehicle in the scene map is calculated as the real-time position according to the position information and the relative position, thereby improving the accuracy of determining the real-time position of the vehicle; collecting original images around the vehicle by a preset visual sensor at the real-time position as the original images around the vehicle, splicing the original images around the vehicle according to the image positions to obtain a surround stitching top view, determining the position area of the surround stitching top view in the scene map, determining the image corresponding to the position area as the target area image, determining the target position in the target area image according to the blind spot position in the surround stitching top view, using the image corresponding to the target position as the image to be filled, and improving the accuracy of blind spot filling according to the blind spot position of the image to be filled.

[0083] Example 3

[0084] Figure 4 FIG4 is a flowchart of a blind spot filling method provided according to Embodiment 3 of the present invention. This embodiment further illustrates a blind spot filling method based on the above embodiment. As shown in FIG4, the method includes:

[0085] Step 1: For a specific image frame, determine whether the relative position of the selected historical reference point and the vehicle exceeds a preset range. If so, set a new reference point, the target reference point. The preset range is set based on the specific situation. For example, if the detection range of a sensor such as a camera or radar installed on the vehicle is a fixed value, the set range needs to be smaller than the sensor's detection range to prevent the relative position of the reference point and the vehicle from exceeding the detection range.

[0086] In one embodiment, a method for setting a target reference point is as follows: let n be the number of times the target reference point is set. When blind spot filling is set from "off" to "on", the target reference point needs to be set. At this time, n = 0, the vehicle position (x0, y0) is (0, 0), and the vehicle orientation angle when setting the target reference point is angle0 = 0°. The relative position of the target reference point and the vehicle is (Δx1, Δy1), then the position of the target reference point (x1, y1) in the scene map is (Δx1, Δy1). Subsequently, let the relative position of the real-time position of the vehicle during movement and the target reference point be (Δx2, Δy2), then the position of the vehicle in the reference map during movement is (x', y'), and the orientation of the vehicle in the map is angle', where: x' = Δx2–Δx1; y' = Δy2–Δy1.

[0087] When setting the target reference point, assume that the relative position between the target reference point detected by the sensor and the vehicle is (Δx1, Δy1), the vehicle's position in the scene map is (x', y'), and the angle is angle'. The calculation method for the new reference point's position in the map (x1', y1') and the vehicle's orientation angle angle0(n) when setting the reference point is:

[0088] x1'=x'+Δx1;

[0089] y1'=y'+Δy1;

[0090] angle0'=angle'–angle0.

[0091] Step 2: Calculate the current vehicle's position in the scene map based on the relative position of the target reference point and the vehicle.

[0092] In actual operation, the preset sensor data can be used to obtain the relative position of the reference point and the vehicle as (Δx1, Δy1), and the current vehicle angle and the vehicle heading angle Δangle when the historical reference point was last set. Assuming that the current target reference point is located at (x1, y1) on the map, and the vehicle heading angle when the historical reference point was last set is angle0, then the current vehicle position (x', y') and the vehicle heading angle angle' are calculated as follows:

[0093] x'=x1+Δx1;

[0094] y'=y1+Δy1;

[0095] angle'=angle0+Δangle.

[0096] Step 3: Determine whether the scene map at the current location is complete (i.e., whether it is already stored). If not, obtain the original images of the vehicle's surroundings from the surround view camera (preset visual sensor) and stitch them into a surround view stitched top view. The stitched surround view stitched top view is then inserted into the corresponding position on the scene map.

[0097] In one embodiment, Figure 5 This is a schematic diagram of a blind spot position filling method provided in accordance with the third embodiment of the present invention. The conditions for determining whether the scene map at the current position is completed are that there is a portion of the scene map that has not been stitched in the range covered by the surround view stitching top view formed by stitching the camera images of the current vehicle, and the stitching method of stitching the surround view stitching top view to the corresponding position of the scene map is to map all pixels in the currently stitched surround view stitching top view to the corresponding position of the scene map. In one embodiment, Figure 6 FIG. 1 is a schematic diagram of a process of filling a blind spot position according to a third embodiment of the present invention. Figure 6 As shown, the vehicle position after changing the reference point is (x1, y1), the vehicle head direction is angle1, and the reference point is (x0, y0); the vehicle position at the time of filling is (x2, y2), the vehicle head direction is angle2, and the blind spot position is filled according to the vehicle position at the time of filling being (x2, y2) and the vehicle head direction being angle2.

[0098] Step 4: Select a portion of the scene map image at the current vehicle position and populate it into the blind spot of the surround view stitching top view. This is done by calculating the relative position of all pixels in the portion of the scene map image covered by the vehicle body's projection and the vehicle body, and populating the image into the corresponding position in the blind spot of the surround view stitching top view. This embodiment of the present invention improves the accuracy of calculating the vehicle's position by utilizing multiple sensors to acquire data and calculate the vehicle's position. Saving the populated scene map avoids frequent acquisition of camera images and stitching of the surround view stitching top view, reducing computational time and hardware resources.

[0099] Example 4

[0100] Figure 7 Schematic diagram of a blind spot filling device according to the fourth embodiment of the present invention. Figure 7 As shown, the device includes: a reference point determination module 71, a position determination module 72, a top view position determination module 73 and a blind area filling module 74.

[0101] The reference point determination module 71 is configured to determine a target reference point according to a preset distance when the historical reference point exceeds a preset range;

[0102] A position determination module 72 is used to determine the relative position of the vehicle and the target reference point, and determine the real-time position of the vehicle based on the relative position and the position information of the target reference point in the scene map;

[0103] A top view position determination module 73 is configured to obtain an original image of the vehicle's surroundings at the vehicle's real-time position and determine a surround stitching top view based on the original image of the vehicle's surroundings;

[0104] The blind area filling module 74 is used to determine the target area image in the scene map according to the surround-view spliced top view, and fill the blind area position in the surround-view spliced top view according to the target area image.

[0105] In an embodiment of the present invention, when a historical reference point exceeds a preset range, a target reference point is determined according to a preset distance through a reference point determination module, a position determination module determines the relative position of the vehicle and the target reference point, and the real-time position of the vehicle is determined based on the relative position and the position information of the target reference point in the scene map, thereby improving the accuracy of the vehicle position; an original image of the vehicle's surroundings at the real-time position is obtained through a bird's-eye view position determination module, a blind spot filling module determines a surround-view spliced top view based on the original image of the vehicle's surroundings, determines a target area image in the scene map based on the surround-view spliced top view, and fills the blind spot position in the surround-view spliced top view according to the target area image, thereby realizing that image data for blind spot filling is obtained from the scene map, thereby improving the efficiency and accuracy of blind spot filling.

[0106] In one embodiment, the reference point determination module 71 includes:

[0107] a distance determination unit, configured to determine reference points within a preset distance of the vehicle and determine the distance between each reference point and the vehicle;

[0108] The reference point determination unit is used to take the reference point corresponding to the minimum value of the distance as the target reference point.

[0109] In one embodiment, the location determination module 72 includes:

[0110] A relative position determination unit is used to establish a physical coordinate system with the vehicle position as a reference, determine the distance and angle between the vehicle and the target reference point through a preset vehicle sensor, and determine the relative position of the vehicle and the target reference point according to the distance and angle;

[0111] a position information determination unit, configured to extract position information of a target reference point in a scene map;

[0112] The real-time position determination unit is used to calculate the position of the vehicle in the scene map as the real-time position according to the position information and the relative position.

[0113] In one embodiment, the top view position determination module 73 includes:

[0114] an original image determining unit, configured to collect an original image around the vehicle at a real-time position through a preset visual sensor as an original image around the vehicle;

[0115] The blind spot position determination unit is used to splice the original images around the vehicle according to the image positions to obtain a surround-view spliced top view; wherein the blank position in the surround-view spliced top view is the blind spot position.

[0116] In one embodiment, the blind spot filling module 74 includes:

[0117] A target image determination unit is used to determine a location area of the surround stitching top view in the scene map, and determine an image corresponding to the location area as a target area image;

[0118] The blind spot filling unit is used to determine the target position in the target area image according to the blind spot position in the surround stitching top view, and use the image corresponding to the target position as the image to be filled, and fill the blind spot position of the image to be filled.

[0119] In one embodiment, a blind spot filling device further includes:

[0120] a first range determination module, configured to determine a relative distance between the vehicle and the historical reference point, and determine that the historical reference point exceeds the preset range when the relative distance is determined to be greater than a preset range;

[0121] The second range determination module is configured to determine that the historical reference point does not exceed the preset range when it is determined that the relative distance is less than or equal to the preset range, and use the historical reference point as the target reference point.

[0122] In one embodiment, a blind spot filling device further includes:

[0123] The associated storage module is used to store the associated relationship between the surround stitching top view, the image to be filled in the blind spot position and the real-time position.

[0124] In one embodiment, a blind spot filling device further includes:

[0125] An image acquisition module, configured to determine whether there is a surround stitching overhead view associated with the real-time position;

[0126] A first extraction module is configured to extract an image to be filled associated with the real-time position, if any, and fill the image to be filled into the surround stitching top view;

[0127] The second extraction module is used to obtain the original image of the vehicle surrounding the vehicle at the real-time position if it does not exist, and determine the surround stitching top view according to the original image of the vehicle surrounding.

[0128] A blind spot filling device provided by an embodiment of the present invention can execute a blind spot filling method provided by any embodiment of the present invention, and has functional modules and beneficial effects corresponding to the execution method.

[0129] Example 5

[0130] Figure 8 1 is a structural diagram of an electronic device for implementing a blind spot filling method of an embodiment of the present invention. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices (such as helmets, glasses, watches, etc.) and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present invention described and / or required herein.

[0131] like Figure 8 As shown, the electronic device 10 includes at least one processor 11 and a memory, such as a read-only memory (ROM) 12, a random access memory (RAM) 13, etc., which is communicatively connected to the at least one processor 11. The memory stores a computer program that can be executed by the at least one processor. The processor 11 can perform various appropriate actions and processes according to the computer program stored in the read-only memory (ROM) 12 or the computer program loaded from the storage unit 18 into the random access memory (RAM) 13. Various programs and data required for the operation of the electronic device 10 can also be stored in the RAM 13. The processor 11, ROM 12, and RAM 13 are connected to each other via a bus 14. An input / output (I / O) interface 15 is also connected to the bus 14.

[0132] Multiple components in the electronic device 10 are connected to the I / O interface 15, including an input unit 16, such as a keyboard, a mouse, etc.; an output unit 17, such as various types of displays, speakers, etc.; a storage unit 18, such as a magnetic disk, an optical disk, etc.; and a communication unit 19, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 19 allows the electronic device 10 to exchange information / data with other devices via a computer network such as the Internet and / or various telecommunication networks.

[0133] The processor 11 can be any general-purpose and / or specialized processing component with processing and computing capabilities. Some examples of the processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various specialized artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The processor 11 executes the various methods and processes described above, such as a blind spot filling method.

[0134] In some embodiments, a blind spot filling method may be implemented as a computer program, which is tangibly contained in a computer-readable storage medium, such as a storage unit 18. In some embodiments, part or all of the computer program may be loaded and / or installed on the electronic device 10 via the ROM 12 and / or the communication unit 19. When the computer program is loaded into the RAM 13 and executed by the processor 11, one or more steps of the blind spot filling method described above may be performed. Alternatively, in other embodiments, the processor 11 may be configured to perform a blind spot filling method in any other appropriate manner (e.g., by means of firmware).

[0135] Various embodiments of the systems and techniques described herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), system-on-chip systems (SOCs), programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments can include being implemented in one or more computer programs that are executable and / or interpreted on a programmable system comprising at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0136] Computer programs for implementing the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, such that when the computer program is executed by the processor, the functions / operations specified in the flowcharts and / or block diagrams are implemented. The computer program may be executed entirely on the machine, partially on the machine, as a stand-alone software package, partially on the machine and partially on a remote machine, or entirely on a remote machine or server.

[0137] In the context of the present invention, computer-readable storage media can be tangible media that can contain or store a computer program for use with an instruction execution system, device or equipment or used in combination with an instruction execution system, device or equipment. Computer-readable storage media can include but are not limited to electronic, magnetic, optical, electromagnetic, infrared or semiconductor systems, devices or equipment, or any suitable combination of the foregoing. Alternatively, computer-readable storage media can be machine-readable signal media. More specific examples of machine-readable storage media can include electrical connections based on one or more lines, portable computer disks, hard disks, random access memories (RAM), read-only memories (ROM), erasable programmable read-only memories (EPROM or flash memory), optical fibers, portable compact disk read-only memories (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0138] To provide interaction with a user, the systems and techniques described herein can be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the electronic device. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0139] The systems and techniques described herein can be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), a blockchain network, and the Internet.

[0140] A computing system may include clients and servers. The clients and servers are typically remote from each other and typically interact via a communication network. This client-server relationship arises through computer programs running on the respective computers, creating a client-server relationship. The server may be a cloud server, also known as a cloud computing server or cloud host. This server is a hosting product within the cloud computing service ecosystem that addresses the management difficulties and limited scalability of traditional physical hosting and VPS services.

[0141] In one embodiment, the present invention further includes a computer program product, which includes a computer program. When the computer program is executed by a processor, it implements a blind spot filling method of any embodiment of the present invention.

[0142] The computer program product may be implemented by writing computer program code for performing the operations of the present invention in one or more programming languages, or a combination thereof, including object-oriented programming languages such as Java, Smalltalk, C++, and conventional procedural programming languages such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a stand-alone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0143] It should be understood that the various forms of the processes shown above can be used to reorder, add, or delete steps. For example, the steps described in the present invention can be performed in parallel, sequentially, or in a different order, as long as the desired results of the technical solution of the present invention can be achieved. This is not limited herein.

[0144] The above specific embodiments do not limit the scope of protection of the present invention. Those skilled in the art will appreciate that various modifications, combinations, sub-combinations, and substitutions may be made based on design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention are intended to be included within the scope of protection of the present invention.

Claims

1. A blind area filling method, characterized in that: include: When the historical reference point exceeds the preset range, the target reference point is determined according to the preset distance; Determining a relative position between the vehicle and the target reference point, and determining a real-time position of the vehicle based on the relative position and position information of the target reference point in a scene map; Acquire an original image of the vehicle surrounding the vehicle at the real-time position, and determine a surround stitching top view based on the original image of the vehicle surrounding; A target area image in the scene map is determined according to the surround-view spliced top view, and a blind area position in the surround-view spliced top view is filled according to the target area image.

2. The method according to claim 1, characterized in that Determining the target reference point according to the preset distance includes: determining reference points within a preset distance of the vehicle, and determining the distance between each reference point and the vehicle; The reference point corresponding to the minimum distance is used as the target reference point.

3. The method according to claim 1, characterized in that The determining of the relative position of the vehicle and the target reference point, and determining the real-time position of the vehicle according to the relative position and position information of the target reference point in the scene map, includes: Establishing a physical coordinate system with the vehicle position as a reference, determining the distance and angle between the vehicle and the target reference point through a preset vehicle sensor, and determining the relative position of the vehicle and the target reference point according to the distance and angle; Extracting the position information of the target reference point in the scene map; The position of the vehicle in the scene map is calculated as a real-time position according to the position information and the relative position.

4. The method according to claim 1, wherein The acquiring of original images of the surroundings of the vehicle at the real-time position and determining a surround stitching top view according to the original images of the surroundings of the vehicle include: At the real-time position, a preset visual sensor is used to collect an original image of the surroundings of the vehicle as an original image of the surroundings of the vehicle; The original images around the vehicle are spliced according to the image positions to obtain a surround-view spliced top view; wherein the blank positions in the surround-view spliced top view are blind spot positions.

5. The method according to claim 1, wherein Determining a target area image in the scene map according to the surround-view stitched top view, and filling a blind spot position in the surround-view stitched top view according to the target area image, includes: Determine a location area of the surround stitching top view in the scene map, and determine an image corresponding to the location area as a target area image; The target position is determined in the target area image according to the blind spot position in the surround stitching top view, and the image corresponding to the target position is used as the image to be filled, and the blind spot position is determined according to the image to be filled.

6. The method according to claim 1, characterized in that Before determining the target reference point according to the preset distance, the method further includes: determining a relative distance between the vehicle and the historical reference point, and determining that the historical reference point is out of the preset range when it is determined that the relative distance is greater than a preset range; When it is determined that the relative distance is less than or equal to the preset range, it is determined that the historical reference point does not exceed the preset range, and the historical reference point is used as the target reference point.

7. The method according to claim 1, characterized in that After filling the blind spot position in the surround view stitching top view according to the target area image, the method further includes: The association relationship between the surround stitching top view, the image to be filled in the blind spot position and the real-time position is stored.

8. The method according to claim 1, characterized in that Before acquiring the original image of the surroundings of the vehicle at the real-time position and determining the surround stitching top view according to the original image of the surroundings of the vehicle, the method further includes: Determining whether there is a surround stitching bird's-eye view associated with the real-time position; If so, extracting the image to be filled associated with the real-time position, and filling the image to be filled into the surround stitching top view; If not, an original image of the vehicle surrounding the vehicle at the real-time position is obtained, and a surround stitching top view is determined based on the original image of the vehicle surrounding.

9. A blind spot filling device, characterized in that: include: A reference point determination module is used to determine a target reference point according to a preset distance when a historical reference point exceeds a preset range; a position determination module, configured to determine a relative position between the vehicle and the target reference point, and determine a real-time position of the vehicle based on the relative position and position information of the target reference point in a scene map; A top-view position determination module is used to obtain an original image of the vehicle surrounding the vehicle at the real-time position, and determine a surround stitching top view based on the original image of the vehicle surrounding; The blind spot filling module is used to determine the target area image in the scene map according to the surround-view spliced top view, and fill the blind spot position in the surround-view spliced top view according to the target area image.

10. An electronic device, characterized in that: The electronic device comprises: at least one processor; and a memory communicatively connected to the at least one processor; wherein, The memory stores a computer program that can be executed by the at least one processor. The computer program is executed by the at least one processor to enable the at least one processor to perform a blind spot filling method according to any one of claims 1 to 8.

11. A computer-readable storage medium, characterized in that The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement a blind spot filling method according to any one of claims 1 to 8 when executed.

12. A computer program product, characterized in that The computer program product comprises a computer program, and when the computer program is executed by a processor, the computer program implements a blind spot filling method according to any one of claims 1 to 8.

Images