Environmental image generation methods, electronic devices, vehicles and storage media

Abstract

This application relates to the field of vehicle technology, specifically to an environmental image generation method, electronic device, vehicle, and storage medium. The method includes: responding to a first environmental image and an image acquisition command sent from a terminal device; acquiring external environmental images using multiple cameras of the vehicle; the image acquisition command includes shooting posture information corresponding to the terminal device capturing the first environmental image; establishing a first pixel-based mapping relationship between the first environmental image and each external environmental image based on the shooting posture information and calibration parameters of the multiple cameras; fusing the pixel features of the multiple external environmental images based on the first mapping relationship to obtain a fused environmental image; repairing occluded areas in the first environmental image based on the fused environmental image to obtain a second environmental image; and sending the second environmental image to the terminal device. This application improves the utilization rate of vehicle-mounted camera resources.

Description

Technical Field

[0001] This application relates to the field of vehicle technology, specifically to an environmental image generation method, electronic device, vehicle, and storage medium. Background Technology

[0002] As the level of automotive intelligence continues to improve, the number of in-vehicle cameras being installed is constantly increasing. Currently, in-vehicle cameras are mainly used in the field of driver assistance to improve vehicle driving safety and driving convenience. This makes the application scenarios of in-vehicle cameras relatively limited, reducing the utilization rate of in-vehicle camera resources. Summary of the Invention

[0003] In view of the above, it is necessary to propose an environmental image generation method, electronic device, vehicle, and storage medium to solve the technical problem of low utilization rate of vehicle camera resources due to the relatively limited application scenarios of vehicle cameras in the prior art.

[0004] In a first aspect, this application provides an environmental image generation method, the method comprising: responding to a first environmental image and an image acquisition command sent from a terminal device; acquiring external environmental images using multiple cameras of a vehicle, the image acquisition command including shooting posture information corresponding to the terminal device capturing the first environmental image; establishing a first pixel-based mapping relationship between the first environmental image and each external environmental image based on the shooting posture information and calibration parameters of the multiple cameras; fusing pixel features of the multiple external environmental images based on the first mapping relationship to obtain a fused environmental image; repairing occluded areas in the first environmental image based on the fused environmental image to obtain a second environmental image; and sending the second environmental image to the terminal device.

[0005] Optionally, in the above-described environmental image generation method, establishing a pixel-based first mapping relationship between the first environmental image and each external environmental image based on the shooting posture information and the calibration parameters of the plurality of cameras includes: determining a first pose parameter of the terminal device in the world coordinate system based on the shooting posture information; determining a second pose parameter of the plurality of cameras in the world coordinate system based on the calibration parameters of the plurality of cameras; and establishing a pixel-based first mapping relationship between the first environmental image and each external environmental image based on the first pose parameter and the second pose parameter of the plurality of cameras in the world coordinate system.

[0006] Optionally, in the above-described environmental image generation method, the calibration parameters of the plurality of cameras include first position parameters and camera pose parameters of the plurality of cameras in the vehicle coordinate system. The step of determining second pose parameters of the plurality of cameras in the world coordinate system based on the calibration parameters of the plurality of cameras includes: determining camera pose parameters of the plurality of cameras in the vehicle coordinate system based on the first position parameters and camera pose parameters of the plurality of cameras in the vehicle coordinate system; determining vehicle pose parameters of the vehicle in the world coordinate system based on the second position parameters and vehicle body pose parameters of the vehicle in the world coordinate system; and determining the second pose parameters based on the camera pose parameters and the vehicle pose parameters.

[0007] Optionally, in the above-described environmental image generation method, the step of repairing the occluded region in the first environmental image based on the fused environmental image to obtain a second environmental image includes: establishing a second mapping relationship between the fused environmental image and the first environmental image based on pixels; calculating pixel feature differences between pixels in the fused environmental image and the first environmental image based on the second mapping relationship; determining the occluded region from the first environmental image based on the pixel feature differences; and mapping the pixel features of the pixels corresponding to the occluded region in the fused environmental image to the first environmental image based on the second mapping relationship to generate the second environmental image.

[0008] Optionally, before sending the second environmental image to the terminal device, the above-described environmental image generation method further includes: acquiring adjacent frame images, ambient lighting information, and vehicle driving information of each external environmental image; and based on the adjacent frame images, the ambient lighting information, and the vehicle driving information, eliminating repair traces in the second environmental image to obtain an updated second environmental image.

[0009] Optionally, in the above-described environmental image generation method, sending the second environmental image to the terminal device includes: performing quality verification on the updated second environmental image based on the adjacent frame images, wherein the quality verification includes at least one of scene continuity verification, lighting consistency verification, and occlusion removal integrity verification; if the updated second environmental image passes the verification, sending the updated second environmental image to the terminal device; if the updated second environmental image fails the verification, regenerating the updated second environmental image until the regenerated second environmental image passes the verification, and then sending the verified second environmental image to the terminal device.

[0010] Optionally, in the above-described environmental image generation method, the image generation instruction includes the acquisition time of the first environmental image acquired by the terminal device, and the acquisition of external environmental images using multiple cameras of the vehicle includes: acquiring historical environmental images acquired by the multiple cameras; and extracting the external environmental image from the historical environmental images based on the acquisition time.

[0011] Secondly, this application provides an electronic device, which includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the computer program is executed by the processor, it implements the environmental image generation method described above.

[0012] Thirdly, this application provides a vehicle that includes the electronic device of claim 8.

[0013] Fourthly, this application provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the environmental image generation method described above.

[0014] Based on the above, this application repairs occluded areas in a first environmental image captured by a terminal device using external images collected by multiple cameras on a vehicle, generating a second environmental image without occlusion. This enriches the application scenarios of vehicle cameras and improves the utilization rate of vehicle camera resources. Attached Figure Description

[0015] Figure 1 This is a schematic diagram illustrating an application scenario of the environmental image generation method provided in an embodiment of this application.

[0016] Figure 2 This is a flowchart of an environmental image generation method provided in an embodiment of this application.

[0017] Figure 3 This is a flowchart of repairing a first environmental image provided in one embodiment of this application.

[0018] Figure 4 This is a schematic diagram of an electronic device provided in an embodiment of this application. Detailed Implementation

[0019] The embodiments of this application are described in detail below. Examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain this application, and should not be construed as limiting this application.

[0020] In the embodiments of this application, it should be noted that, unless otherwise expressly specified and limited, the word "for example" is used to indicate an example, illustration, or description. Any embodiment or design scheme described as "for example" in the embodiments of this application should not be construed as being better or more advantageous than other embodiments or design schemes. Specifically, the use of the word "for example" is intended to present the relevant concepts in a specific manner.

[0021] In the description of this application, it should be noted that, unless otherwise expressly specified and limited, the terms "installation," "connection," and "linking" should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral connection; they can refer to a mechanical connection, an electrical connection, or a connection that allows communication between them; they can refer to a direct connection or an indirect connection through an intermediate medium; they can refer to the internal communication between two components or the interaction between two components. Those skilled in the art can understand the specific meaning of the above terms in this application according to the specific circumstances.

[0022] In the description of this application, it should be noted that, unless otherwise expressly specified and limited, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Therefore, features defined as "first" or "second" may explicitly or implicitly include one or more features. Furthermore, in the description of this application, "multiple" means two or more, unless otherwise expressly and specifically limited.

[0023] To better understand the above-mentioned objectives, features, and advantages of the present invention, the present invention will be described in detail below with reference to the accompanying drawings and specific embodiments. It should be noted that, unless otherwise specified, the embodiments and features described in these embodiments can be combined with each other.

[0024] Please see Figure 1 This is a schematic diagram illustrating an application scenario of the environmental image generation method provided in an embodiment of this application. The occupants of vehicle 1 use a handheld terminal device 2 to capture images of the environment outside the vehicle. The occupants can be the driver, front passenger, or rear-seat passengers.

[0025] In related technologies, due to the limited area of ​​the car window, the field of view of the terminal device 2 is inevitably blocked by the vehicle 1, such as by the front pillar (A pillar) or the middle pillar (B pillar) of the vehicle 1, resulting in an image of the blocked environment.

[0026] To address the aforementioned issues, this application provides an environmental image generation method. By using an unobstructed environmental image captured by an in-vehicle camera to repair the obstructed areas of an occluded environmental image captured by a terminal device, a repaired environmental image is generated, thereby improving the utilization rate of in-vehicle camera resources.

[0027] Please see Figure 2 This is a flowchart of an environmental image generation method provided in an embodiment of this application.

[0028] The environmental image generation method provided in this embodiment can be applied to... Figure 1 Vehicle 1 is shown.

[0029] Specifically, the environmental image generation method includes the following steps. Depending on different needs, the order of some steps in the flowchart can be changed, and some steps can be omitted.

[0030] S201: In response to the first environmental image and image acquisition command sent from the terminal device, the vehicle uses multiple cameras to acquire external environmental images. The image acquisition command includes the shooting posture information corresponding to the terminal device when capturing the first environmental image.

[0031] In some embodiments of this application, the terminal device can be an electronic device with both photographing and communication functions, such as any one of a smart digital camera, smartphone, tablet computer, or action camera. The terminal device responds to a shooting command, captures a first environmental image, and acquires the shooting posture information of the terminal device. The shooting posture information may include any one or more of shooting parameters such as pitch angle, yaw angle, and roll angle. The shooting command can be generated based on user control operations. For example, a shooting command is generated after the user presses or clicks the "shutter" space on the terminal device.

[0032] In some embodiments of this application, after obtaining the first environmental image, the terminal device can transmit the first environmental image to the vehicle through a preset transmission method or a preset interactive interface. Furthermore, in other embodiments, the terminal device can trigger a preset command to send an image acquisition command to the vehicle, and also send the first environmental image captured by the terminal device to the vehicle. The preset command can be triggered by clicking a function control in a preset interactive interface, or it can be a preset user command. Upon receiving the first environmental image and the image acquisition command from the terminal device, the vehicle responds to the image acquisition command and uses multiple cameras on the vehicle to acquire external environmental images. Specifically, the vehicle can respond to the image acquisition command and control its multiple cameras to acquire external environmental images respectively.

[0033] In another embodiment of this application, the image generation instruction includes the acquisition time of the terminal device acquiring the first environmental image, and the acquisition of external environmental images using multiple cameras of the vehicle, including: acquiring historical environmental images acquired by multiple cameras; and extracting external environmental images from the historical environmental images based on the acquisition time.

[0034] In some embodiments of this application, the vehicle's camera can continuously acquire environmental images at a preset sampling frequency, and then temporarily store the environmental images acquired within a preset time. When the vehicle receives an image acquisition command, it obtains the acquisition time of the first environmental image from the image acquisition command, and then obtains the external environmental image closest to the acquisition time from the temporarily stored environmental images. For example, the vehicle can store environmental images acquired by the camera within 5 minutes, and discard environmental images stored for more than 5 minutes.

[0035] S202: Based on the shooting posture information and the calibration parameters of multiple cameras, establish a first mapping relationship based on pixels between the first environmental image and each external environmental image.

[0036] In some embodiments of this application, establishing a first mapping relationship based on pixels between a first environmental image and each external environmental image, based on shooting posture information and calibration parameters of multiple cameras, includes: determining a first pose parameter of the terminal device in the world coordinate system based on shooting posture information; determining a second pose parameter of the multiple cameras in the world coordinate system based on calibration parameters of multiple cameras; and establishing a first mapping relationship based on pixels between the first environmental image and each external environmental image, based on the first pose parameter and the second pose parameter of the multiple cameras in the world coordinate system.

[0037] In some embodiments of this application, the shooting attitude information may include translation parameters and rotation parameters. Based on the translation parameters and rotation parameters, the first attitude parameter is determined. The translation parameters may include the horizontal axis translation, vertical axis translation, and lateral axis translation of the camera. The rotation parameters include pitch angle, yaw angle, and roll angle. The translation parameters and rotation parameters are converted into vector form and combined to form the complete first attitude parameter.

[0038] In some embodiments of this application, the rotation and translation parameters of the terminal device can be dynamically acquired using sensors integrated within the terminal device. When a shooting command is generated based on the user's control operation of the terminal device, a first environmental image is acquired in response to the shooting command, and shooting posture parameters are obtained. The rotation parameters of the terminal device can be acquired using an Inertial Measurement Unit (IMU). The IMU first acquires the three-axis angular velocity and acceleration of the terminal device, and then calculates the rotation parameters based on the three-axis angular velocity and acceleration. The translation parameters of the terminal device can be obtained by acquiring absolute geographic coordinates using a Global Positioning System (GPS), and then calculated based on the three-axis angular velocity and three-axis linear acceleration collected by the IMU. The absolute geographic coordinates can include geodetic longitude, geodetic latitude, and altitude, which together determine the absolute three-dimensional geographic reference position of the terminal device.

[0039] In some embodiments of this application, the calibration parameters of the multiple cameras include first position parameters and camera pose parameters of the multiple cameras in the vehicle coordinate system. Based on the calibration parameters of the multiple cameras, determining the second pose parameters of the multiple cameras in the world coordinate system includes: determining the camera pose parameters of the multiple cameras in the vehicle coordinate system based on the first position parameters and camera pose parameters of the multiple cameras in the vehicle coordinate system; determining the vehicle pose parameters of the vehicle in the world coordinate system based on the second position parameters and vehicle body pose parameters of the vehicle in the world coordinate system; and determining the second pose parameters based on the camera pose parameters and vehicle pose parameters.

[0040] In some embodiments of this application, the first position parameter of the camera in the vehicle coordinate system can be the three-dimensional spatial offset of the camera's mounting origin relative to the origin of the vehicle coordinate system. Specifically, it can include the forward / backward offset of the camera relative to the vehicle's horizontal axis, the forward / backward offset of the camera relative to the vehicle's vertical axis, and the forward / backward offset of the camera relative to the vehicle's horizontal axis. The camera's attitude parameter in the vehicle coordinate system can be the deflection angle of the camera's optical axis relative to the vehicle coordinate system's axis. Specifically, it can include the camera's pitch angle relative to the vehicle's horizontal axis, roll angle relative to the vehicle's vertical axis, and yaw angle relative to the vehicle's vertical axis. The first position parameter and camera attitude parameter are different for different cameras. The calibration parameters of the camera relative to the vehicle coordinate system can be uniformly and centrally stored in a designated storage area of ​​the vehicle system. When the system runs and calls relevant vision algorithms and coordinate transformation programs, it directly reads the fixed calibration parameters to complete the calculations.

[0041] In some embodiments of this application, based on the first pose parameter and the second pose parameters of multiple cameras in the world coordinate system, a first mapping relationship based on pixels is established between the first environmental image and each external environmental image, including: calculating the relative pose parameters of the first pose parameter and each second pose parameter respectively; and establishing the first mapping relationship based on pixels between the first environmental image and each external environmental image based on the relative pose parameters.

[0042] In some embodiments of this application, the first pose parameter refers to the pose parameter when the terminal device acquires the first environmental image. The second pose parameter refers to the pose parameter when the camera on the vehicle acquires the external environment image. The relative pose parameter characterizes the spatial transformation relationship between the first pose parameter and the second pose parameter, and is used to characterize the relative position and relative attitude between two different poses. The first mapping relationship is based on the relative pose parameter to establish the correspondence between each pixel in the first environmental image and each pixel in the corresponding external environment image. Specifically, the spatial position angle difference between the terminal device and the camera can be calculated first through the relative pose parameter, and then the correspondence between the pixels in the first environmental image and the pixels in the external environment can be calculated based on the spatial position angle difference, thus forming the first mapping relationship.

[0043] S203: Based on the first mapping relationship, the pixel features of multiple external environment images are fused to obtain a fused environment image.

[0044] In some embodiments of this application, pixel features of multiple external environment images are fused based on a first mapping relationship to obtain a fused environment image. This can be achieved by mapping each external environment image to the world coordinate system corresponding to the first environment image, based on the first mapping relationship, thus completing the fusion in the world coordinate system corresponding to the first environment image. Using the shooting viewpoint direction of the first environment image as the reference direction, and relying on the pre-constructed first mapping relationship, each external environment image is uniformly mapped and transformed from its original shooting viewpoint direction to the reference viewpoint, completing the viewpoint alignment and viewpoint normalization processing of multi-view images.

[0045] S204: Based on the fused environment image, the occluded areas in the first environment image are repaired to obtain the second environment image.

[0046] In some embodiments of this application, the occlusion area can be determined from the first environmental image, and the information missing area corresponding to the occlusion area can be determined from the fused environmental image. The pixel features of the information missing area in the fused environmental image can be mapped to the occlusion area in the first environmental image to complete the repair of the first environmental image and obtain the second environmental image.

[0047] In some embodiments of this application, the field of view corresponding to the first image captured by the mobile phone is first determined, and then, based on the field of view, an unobstructed image matching the viewpoint and field of view of the first image is extracted from the fused image. Specifically, the imaging field of view angle can be calculated using the intrinsic parameters of the terminal device, and combined with the real-time translation and rotation parameters of the terminal device, the spatial field of view of the first image is jointly determined. The field of view can be the three-dimensional spatial visible range that the terminal device can image and cover under the current spatial position and shooting posture, and is also the scene area that the corresponding image can present.

[0048] S205: Send the second environmental image to the terminal device.

[0049] In some embodiments of this application, after generating a second environmental image, the vehicle sends the second image to the terminal device via a communication connection, and the terminal device displays the second image to the user. This allows the user to see an unobstructed environmental image through the terminal device, enabling the in-vehicle shooting environment to achieve a shooting effect comparable to that of the out-of-vehicle shooting environment.

[0050] Based on the above, this application repairs occluded areas in a first environmental image captured by a terminal device using external images collected by multiple cameras on a vehicle, generating a second environmental image without occlusion. This enriches the application scenarios of vehicle cameras and improves the utilization rate of vehicle camera resources.

[0051] Please see Figure 3 This is a flowchart illustrating the restoration of a first environmental image according to an embodiment of this application. Based on the fused environmental image, occluded areas in the first environmental image are restored to obtain a second environmental image, including: S301: Establish a second mapping relationship between the fused environment image and the first environment image based on pixels.

[0052] In some embodiments of this application, the fused environment image and the first environment image are mapped to images with the same viewpoint. Therefore, the pixel coordinates of each pixel in the fused environment image and the first environment image can be obtained. Based on the pixels with the same pixel coordinates in the fused environment image and the first environment image, a second mapping relationship based on pixels is established between the fused environment image and the first environment image. For example, if the pixel coordinates of the first pixel in the fused environment image and the second pixel in the first environment image are both (199, 199), a correspondence between the first pixel and the second pixel is established.

[0053] S302: Based on the second mapping relationship, calculate the pixel feature difference between pixels in the fused environment image and the first environment image.

[0054] In some embodiments of this application, the pixel difference between two pixels that have a mapping relationship can be calculated based on a second mapping relationship. The pixel difference between pixels can be the difference of any pixel feature value, such as grayscale value, brightness value, or color value.

[0055] S303: Determine the occlusion region from the first environment image based on pixel feature differences.

[0056] In some embodiments of this application, if the pixel difference exceeds a preset difference threshold, it is determined that the pixel in the first environmental image is in the occlusion area. If the pixel difference does not exceed the preset difference threshold, the pixel difference of the pixel features between the next group of pixels with a mapping relationship is calculated until all pixels with pixel differences exceeding the difference threshold are found. Based on all pixels with pixel differences exceeding the difference threshold, the occlusion area is determined from the first environmental image.

[0057] Furthermore, in this embodiment, pixel connectivity filtering can be used to remove scattered noisy pixels from all pixels whose pixel differences exceed the difference threshold, and only continuous patches of pixels exceeding the threshold can be defined as occluded areas, which can improve the recognition accuracy of occluded areas.

[0058] S304: Based on the second mapping relationship, the pixel features of the pixels corresponding to the occluded area in the fused environment image are mapped to the first environment image to generate the second environment image.

[0059] In some embodiments of this application, pixels corresponding to the occluded area in the fused environment image can be matched according to the second mapping relationship, pixel features of the pixels in the fused environment image can be extracted, and the pixel features of the pixels can be mapped and filled into the occluded area of ​​the first environment image. After pixel feature replacement and integration, the second environment image is generated.

[0060] Based on the above, this embodiment repairs the occluded areas in the first environmental image based on the fused environmental image. This can effectively avoid image distortion and incoordination of light and shadow after image repair, reduce the visual gap between the repaired image and the real-life scene outside the vehicle, and improve the realism and natural coordination of the occluded area repair effect.

[0061] In some embodiments of this application, before sending the second environmental image to the terminal device, the method further includes: acquiring adjacent frame images, ambient lighting information, and vehicle driving information of each external environmental image; and eliminating repair traces in the second environmental image based on the adjacent frame images, ambient lighting information, and vehicle driving information to obtain an updated second environmental image.

[0062] In some embodiments of this application, adjacent frame images can be one or more frames immediately preceding or following the current external environment image captured by the vehicle's camera in continuous temporal shooting. These frames contain the same scene at different times with slight changes in viewing angle, serving as a reference for scene dynamics, object positions, and image continuity. Ambient lighting information can be real-time light-related data within the shooting scene, including any one or more of the following: ambient light intensity, light direction, natural light level, sunny / cloudy lighting conditions, color temperature, shadow distribution, and backlight / frontlight conditions. This information is used to unify image brightness and lighting style. Vehicle driving information can include operational status data collected during vehicle movement, including any one or more driving motion parameters such as vehicle speed, driving direction, vehicle posture changes, steering angle, driving position, and acceleration / deceleration status. This information can be used to predict scene changes and correct shooting angle deviations.

[0063] In some embodiments of this application, environmental feature vectors can be extracted based on adjacent frame images of each external environment image, ambient lighting information, and vehicle driving information. The environmental feature vectors and the generated second environment image are then input into a preset AI generation model, which enables the AI ​​generation model to remove repair traces from the second environment image based on the environmental feature vectors and outputs the second environment image after the repair traces have been removed.

[0064] In some embodiments of this application, sending the second environmental image to the terminal device includes: performing quality verification on the updated second environmental image based on adjacent frame images, wherein the quality verification includes at least one of scene continuity verification, lighting consistency verification, and occlusion removal integrity verification; if the updated second environmental image passes the verification, the updated second environmental image is sent to the terminal device; if the updated second environmental image fails the verification, the updated second environmental image is regenerated until the regenerated second environmental image passes the verification, and the verified second environmental image is sent to the terminal device.

[0065] In some embodiments of this application, scene continuity verification can be performed to check whether the position, outline, spatial layout, and edge connection of objects in the repaired image are consistent and unified with the surrounding real scene, ensuring that the second environmental image is free of discontinuities, misalignments, and scene fragmentation. For example, if a road in the second environmental image is obscured, after repairing and adding the road, it is verified whether the road surface lines and road edges are smoothly connected with the preceding and following roads, without any road surface bending, misalignment, or scene splicing breaks.

[0066] In some embodiments of this application, the light and shadow consistency verification can be performed to check whether the brightness, darkness, shadow, color temperature, and reflective effect of the repaired area are consistent with the overall lighting atmosphere of the entire image, without any contrast in brightness or abrupt color tones. For example, if the whole vehicle scene is shot in daylight with direct sunlight, the verification can be performed to check whether the brightness, shadow depth, and color warmth of the occluded repaired area are consistent with the whole vehicle image, and whether there are any phenomena such as the repaired area being too dark, color cast, or backlighting incongruity.

[0067] In some embodiments of this application, the occlusion removal integrity check can be to verify whether all occluded areas in the original image have been completely filled and repaired with pixels, with no residual occluded pixels, no local blank areas, and no unfilled blind spots. For example, if there are occlusions at both the car window and the front of the car in the image, the check can be made to ensure that the scene at both occluded locations has been completely filled, with no small occluded shadows or unrepaired blank areas.

[0068] Please see Figure 4 This is a schematic diagram of an electronic device provided in an embodiment of this application.

[0069] This application provides an environmental image generation method, which can be applied to one or more electronic devices 10. The electronic devices 10 can be configured in a vehicle 1. For example, the electronic devices 10 can be on-board devices of the vehicle 1, such as a Body Control Module (BCM) or a Vehicle Control Unit (VCU). In addition, the electronic devices 10 can also be computing devices or terminal devices that communicate with the vehicle 1.

[0070] Electronic device 10 is a device that can automatically perform numerical calculations and / or information processing according to pre-set or stored instructions. Its hardware includes, but is not limited to, microprocessors, application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), digital signal processors (DSPs), embedded devices, etc. Electronic device 10 includes, but is not limited to, mobile phones, tablet computers, laptops, personal computers, and servers.

[0071] Specifically, the electronic device 10 is used to repair the occluded areas of the occluded environmental images collected by the terminal device from the unobstructed environmental images captured by the vehicle-mounted camera, thereby generating repaired environmental images and improving the utilization rate of vehicle-mounted camera resources.

[0072] In some embodiments of this application, the electronic device 10 can be communicatively connected to devices such as desktop computers, laptops, handheld computers, and cloud servers.

[0073] In some embodiments of this application, the electronic device 10 can interact with the user via a keyboard, mouse, remote control, touchpad, or voice control device.

[0074] Combination Figure 4 As shown, in some embodiments of this application, the electronic device 10 includes, but is not limited to, a memory 11, a processor 12, and a computer program stored in the memory 11 and executable on the processor 12, such as an environmental image generation program. When the computer program is executed by the processor 12, it implements the environmental image generation method as described in the above embodiments.

[0075] The memory 11 in the electronic device 10 stores multiple computer-readable instructions to implement an environmental image generation method. The processor 12 can execute multiple instructions to: repair occluded areas in an occluded environmental image captured by a terminal device using an unobstructed environmental image captured by an in-vehicle camera, and generate a repaired environmental image, thereby improving the utilization rate of the in-vehicle camera resources. Specifically, the specific implementation method of the processor 12 for the above instructions can be found in [reference needed]. Figure 2 The descriptions of the relevant steps in the corresponding embodiments are not repeated here.

[0076] Those skilled in the art will understand that the schematic diagram is merely an example of the electronic device 10 and does not constitute a limitation on the electronic device 10. The electronic device 10 can be a bus topology or a star topology. The electronic device 10 may also include more or fewer other hardware or software than shown in the diagram, or different component arrangements. For example, the electronic device 10 may also include input / output devices, network access devices, etc.

[0077] The memory 11 includes at least one type of computer-readable storage medium, which can be non-volatile or volatile. Computer-readable storage media include flash memory, portable hard drives, multimedia cards, card-type memories (e.g., SD memory, DX memory, etc.), magnetic storage, magnetic disks, optical disks, etc. In some embodiments, the memory 11 can be an internal storage unit of the electronic device 10, such as the portable hard drive of the electronic device 10. In other embodiments, the memory 11 can also be an external storage device of the electronic device 10, such as a plug-in portable hard drive, smart media card (SMC), secure digital (SD) card, flash card, etc., equipped on the electronic device 10. The memory 11 can be used not only to store application software and various types of data installed on the electronic device 10, such as the code of an environmental image generation program, but also to temporarily store data that has been output or will be output.

[0078] In some embodiments, the processor 12 may be composed of integrated circuits, such as a single packaged integrated circuit or multiple integrated circuits packaged with the same or different functions, including combinations of one or more central processing units (CPUs), microprocessors, digital processing chips, graphics processors, and various control chips. The processor 12 is the control unit of the electronic device 10, connecting various components of the electronic device 10 via various interfaces and lines. It executes programs or modules stored in the memory 11 (e.g., executing an environmental image generation program) and calls data stored in the memory 11 to perform various functions and process data for the electronic device 10.

[0079] Processor 12 executes the operating system of electronic device 10 and various installed applications. Processor 12 executes these applications to implement the steps described in each of the above-described embodiments of the environmental image generation method, for example... Figure 2 The steps are shown.

[0080] For example, a computer program may be divided into one or more modules / units, one or more of which are stored in memory 11 and executed by processor 12 to complete this application. One or more modules / units may be a series of computer-readable instruction segments capable of performing a specific function, which describe the execution process of the computer program in electronic device 10.

[0081] The integrated unit implemented as a software functional module described above can be stored in a computer-readable storage medium. This software functional module, stored in a storage medium, includes several instructions to cause a computer device (which may be a personal computer, computer equipment, or network device, etc.) or processor to execute a portion of an environmental image generation method according to various embodiments of this application.

[0082] If the modules / units integrated in the electronic device 10 are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the methods of the above embodiments can also be implemented by a computer program instructing related hardware devices. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above.

[0083] Computer programs include computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. Computer-readable media can include: any entity or device capable of carrying computer program code, recording media, USB flash drives, portable hard drives, magnetic disks, optical disks, computer memory, read-only memory (ROM), random access memory, and other types of memory.

[0084] Furthermore, the computer-readable storage medium may primarily include a stored program area and a stored data area, wherein the stored program area may store the operating system, an application program required for at least one function, etc.; and the stored data area may store data created based on the use of blockchain nodes, etc.

[0085] The bus can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. This bus can be divided into address bus, data bus, control bus, etc. For ease of representation, in... Figure 4 The symbol is represented by only one arrow, but this does not mean that there is only one bus or one type of bus. The bus is configured to implement communication between memory 11 and at least one processor 12, etc.

[0086] This application also provides a computer-readable storage medium (e.g., ...). Figure 4 The memory 11 in the electronic device stores computer-readable instructions, which are executed by a processor in the electronic device to implement an environmental image generation method of any of the above embodiments.

[0087] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of modules is only a logical functional division, and other division methods may be used in actual implementation.

[0088] The modules described as separate components may or may not be physically separate. The components shown as modules may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0089] Furthermore, the functional modules in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or in the form of hardware plus software functional modules.

[0090] Furthermore, it is clear that the word "comprising" does not exclude other units or steps, and the singular does not exclude the plural. Multiple units or devices described in the specification may also be implemented by a single unit or device through software or hardware. Terms such as "first," "second," etc., are used to indicate names and do not indicate any specific order.

[0091] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application and are not intended to limit it. Although this application has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of this application without departing from the spirit and scope of the technical solutions of this application.

Claims

1. A method for generating environmental images, characterized in that, The method includes: In response to a first environmental image and an image acquisition command sent from a terminal device, the vehicle uses multiple cameras to acquire external environmental images. The image acquisition command includes the shooting posture information corresponding to when the terminal device captures the first environmental image. Based on the shooting posture information and the calibration parameters of the multiple cameras, a first mapping relationship based on pixels is established between the first environmental image and each external environmental image. Based on the first mapping relationship, the pixel features of multiple external environment images are fused to obtain a fused environment image; Based on the fused environmental image, the occluded areas in the first environmental image are repaired to obtain the second environmental image; The second environmental image is sent to the terminal device.

2. The environmental image generation method as described in claim 1, characterized in that, The step of establishing a pixel-based first mapping relationship between the first environmental image and each external environmental image based on the shooting posture information and the calibration parameters of the multiple cameras includes: Based on the shooting posture information, the first pose parameter of the terminal device in the world coordinate system is determined; Based on the calibration parameters of the multiple cameras, the second pose parameters of the multiple cameras in the world coordinate system are determined; Based on the first pose parameters and the second pose parameters of the plurality of cameras in the world coordinate system, a first mapping relationship based on pixels is established between the first environmental image and each external environmental image.

3. The environmental image generation method as described in claim 2, characterized in that, The calibration parameters of the plurality of cameras include first position parameters and camera pose parameters of the plurality of cameras in the vehicle coordinate system. The step of determining the second pose parameters of the plurality of cameras in the world coordinate system based on the calibration parameters of the plurality of cameras includes: Based on the first position parameters and camera attitude parameters of the multiple cameras in the vehicle coordinate system, the camera pose parameters of the multiple cameras in the vehicle coordinate system are determined. Based on the second position parameters and vehicle attitude parameters of the vehicle in the world coordinate system, the vehicle pose parameters in the world coordinate system are determined. The second pose parameter is determined based on the camera pose parameter and the vehicle pose parameter.

4. The environmental image generation method as described in claim 1, characterized in that, The step of repairing occluded areas in the first environmental image based on the fused environmental image to obtain a second environmental image includes: Establish a second pixel-based mapping relationship between the fused environment image and the first environment image; Based on the second mapping relationship, the pixel feature difference between the pixels in the fused environment image and the first environment image is calculated; The occlusion region is determined from the first environmental image based on the pixel feature difference. Based on the second mapping relationship, the pixel features of the pixels corresponding to the occluded area in the fused environment image are mapped to the first environment image to generate the second environment image.

5. The environmental image generation method as described in claim 1, characterized in that, Before sending the second environmental image to the terminal device, the method further includes: Acquire adjacent frame images, ambient lighting information, and vehicle driving information for each of the external environment images; Based on the adjacent frame images, the ambient lighting information, and the vehicle driving information, the repair traces in the second environmental image are eliminated to obtain an updated second environmental image.

6. The environmental image generation method as described in claim 5, characterized in that, Sending the second environmental image to the terminal device includes: Based on the adjacent frame images, the updated second environment image is subjected to quality verification, which includes at least one of scene continuity verification, lighting consistency verification, and occlusion removal integrity verification. If the updated second environment image passes the verification, the updated second environment image will be sent to the terminal device; If the updated second environment image fails verification, the updated second environment image is regenerated until the regenerated second environment image passes verification. The verified second environment image is then sent to the terminal device.

7. The environmental image generation method as described in claim 1, characterized in that, The image generation instruction includes the acquisition time of the first environmental image acquired by the terminal device, and the acquisition of external environmental images using multiple cameras of the vehicle includes: Acquire historical environmental images captured by the multiple cameras; Based on the acquisition time, the external environment image is extracted from the historical environment image.

8. An electronic device, characterized in that, The electronic device includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program, when executed by the processor, implements the environmental image generation method as described in any one of claims 1 to 7.

9. A vehicle, characterized in that, The vehicle includes the electronic equipment as described in claim 8.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the environmental image generation method as described in any one of claims 1 to 7.

Images