Moving image map processing method, device, system and carrier thereof

Abstract

The invention discloses an image map updating method, device and system suitable for a carrier and the carrier thereof, and the method comprises the following steps: receiving information: receiving information used for judging whether an image map updating condition is satisfied or not; a determination step of determining whether the condition for updating the image map is satisfied according to the information; when the condition is met, executing a receiving step of receiving carrier information and collecting images shot by more than one camera arranged on the carrier; an image processing step of respectively processing the images according to the lens characteristics of the camera and the received carrier information; and a writing step: writing the processed image into a corresponding position of the image map according to the received carrier information.

Description

[0001] priority

[0002] This Chinese patent application claims the priority of a Taiwan patent with an application date of October 23, 2023 and application number 112140345. This patent application combines the technical solution of the above-mentioned Taiwan patent. Technical Field

[0003] The present application relates to image processing, and in particular to image processing of dynamic vehicles. Background Art

[0004] The cabs of vehicles such as vehicles and boats are usually equipped with many glasses for the driver to see the surroundings, but there are still many blind spots, such as the ground at the bottom of the vehicle, the part blocked by the beam and column structure of the vehicle, etc. In order to make up for the blind spots and improve the safety of vehicle operation, multiple cameras or image sensors are generally installed to capture the surroundings of the vehicle, so as to transmit the images including the blind spots to the dashboard in the cab and / or the helmet display of the driver.

[0005] To serve as evidence of accidents or as sensors for autonomous driving, the resolution of modern cameras can reach 2K or 4K standards. The sampling rate of such full-color 24-bit cameras can reach more than tens of frames per second, which is higher than the 30 frames per second of human visual persistence. However, processing continuous images continuously transmitted by multiple cameras requires a lot of computing resources. Since the demand for ordinary vehicles is not to open a mobile studio, but to provide drivers with situational awareness at normal driving speeds, an information system that can provide vehicle surrounding awareness with less computing resources is needed to save costs. Summary of the invention

[0006] According to one embodiment of the present application, a method for updating an image map applicable to a vehicle is provided, comprising: an information receiving step, receiving information used to determine whether a condition for updating the image map is satisfied; a determination step, determining whether the condition for updating the image map is satisfied based on the information; when it is determined that the condition is satisfied, executing a receiving step, receiving vehicle information and images captured by one or more cameras installed on the vehicle; an image processing step, processing the images separately based on the lens characteristics of the cameras and the received vehicle information; and a writing step, writing the processed images to a corresponding position of the image map based on the received vehicle information.

[0007] Preferably, in order to know the corresponding position of the image map, the vehicle information includes the current position and current pointing angle of the vehicle.

[0008] Preferably, in order to obtain the vehicle information, the vehicle information comes from one or any combination of the following: a vehicle information system; and a navigation and positioning system.

[0009] Preferably, in order to make the brightness of the image in the image map smooth, the image processing step further includes adjusting the brightness of the written image according to one or any combination of the following: according to the existing images around the corresponding position of the image map; and according to the adjacent received image.

[0010] Preferably, in order to satisfy the driver's situational awareness, the above-mentioned conditions for updating the image map include one of the following or any combination thereof: the relative displacement of the vehicle exceeds a critical value; the relative pointing angle of the vehicle exceeds another critical value; the current speed of the vehicle exceeds a speed critical value; the steering angle of the vehicle's steering mechanism exceeds a steering angle critical value; and the time from the last writing step exceeds a time critical value.

[0011] Preferably, in order to provide a condition for determining whether to update the image map next time, the above-mentioned method for updating the image map further includes: setting the condition for updating the image map.

[0012] Preferably, in order to utilize time to organize the image map, the above-mentioned method for updating the image map further includes executing one of the following or any combination thereof when it is determined that the condition is not satisfied: organizing the image map; and pausing for a period of time.

[0013] Preferably, in order to display the portion of the image map corresponding to the current position of the vehicle, a method for displaying an image map is provided, comprising executing the method for updating the image map as described above, and the following steps: a step of receiving vehicle information; reading the corresponding portion of the image map based on the above-mentioned vehicle information; and transmitting the read corresponding portion of the image map to a display module for display.

[0014] Preferably, in order to initialize the image before updating the image map, a method for initializing the image map is provided, comprising executing the following steps before the aforementioned method for updating the image map: establishing a blank image map; executing the aforementioned receiving step; executing the aforementioned image processing step; executing the aforementioned writing step; and executing the method for updating the image map.

[0015] According to an embodiment of the present application, a mobile image map processing device is provided, comprising: at least one processor for executing a plurality of instructions to implement the aforementioned image map updating method; and a memory for storing the image map.

[0016] According to one embodiment of the present application, a mobile image map processing device is provided, comprising: at least one processor for executing multiple instructions to implement the aforementioned method of displaying an image map; a memory for storing the image map; and an input / output interface for transmitting the corresponding portion of the read image map to a display module.

[0017] According to an embodiment of the present application, a mobile image map processing system is provided, comprising: the aforementioned mobile image map processing device; and the display module.

[0018] According to an embodiment of the present application, a vehicle having a mobile image map processing capability is provided, comprising: the aforementioned mobile image map processing system; and the camera.

[0019] According to an embodiment of the present application, a mobile image map processing device is provided, comprising: at least one processor for executing a plurality of instructions to implement the aforementioned method of initializing an image map; and a memory for storing the image map.

[0020] According to various embodiments, the present application provides an information system that can provide vehicle surrounding perception capabilities using less computing resources to save costs, and can provide drivers with situational awareness capabilities at normal driving speeds. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] Figure 1 is a block diagram of a mobile image map processing device 100 according to an embodiment of the present application;

[0022] Figure 2A is a top view of the shooting range of a plurality of cameras 140 carried by a vehicle according to an embodiment of the present application;

[0023] Figure 2B It is a top view of the shooting range of multiple cameras 140 carried by a vehicle according to an embodiment of the present application;

[0024] Figure 3 is a schematic diagram of a vehicle motion model according to an embodiment of the present application;

[0025] FIG. 4A to FIG. 4C Schematic diagram of vehicle motion model and rotation angle;

[0026] Figure 5 is a schematic diagram of updating an image map at intervals according to an embodiment of the present application;

[0027] Figure 6 It is a schematic diagram of updating an image map according to a time sequence according to an embodiment of the present application;

[0028] Figure 7 is a block diagram of a mobile image map processing method 700 according to an embodiment of the present application;

[0029] Figure 8 is a block diagram of a method 710 for initializing an image map according to an embodiment of the present application;

[0030] Fig. 9 is a block diagram of a method 720 for updating an image map according to an embodiment of the present application;

[0031] Fig.10 FIG. 7 is a block diagram of a method 730 for displaying an image map according to an embodiment of the present application.

[0032] 100: mobile image map processing device; 110: processor; 120: memory; 130: input / output interface; 140: camera; 150: vehicle information system; 160: navigation and positioning system; 170: display module; 180: driving module; 210: forward camera shooting range; 220: backward camera shooting range; 230: left camera shooting range; 240: right camera shooting range; 250: forward camera shooting range; 260: left and rear camera shooting range ; 270: Camera shooting range to the right rear; 290: Rectangular or square area; 299: Center position; 500: Image map; 510: Position of the center of the vehicle; 515: Shooting range; 520: Position of the center of the vehicle; 525: Shooting range; 700: Mobile image map processing method; 710: Initialization image map method; 720: Update image map method; 730: Display image map method; 810~850: Steps; 910~980: Steps; 1010~1040: Steps DETAILED DESCRIPTION

[0033] To make the purpose, technical solution and advantages of the present application clearer, the technical solution of the present application will be described in detail below. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all of the embodiments. Based on the embodiments in the present application, all other implementation methods obtained by ordinary technicians in the field without creative work belong to the scope of protection of the present application.

[0034] The terms "first", "second", "third", etc. (if any) in the specification and scope of the patent application of this application and the drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the objects described in these descriptions can be interchanged where appropriate. In the description of this application, "plurality" means two or more, unless otherwise clearly and specifically defined. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions. Some of the block diagrams shown in the drawings are functional entities and do not necessarily correspond to physically or logically independent entities. These functional entities can be implemented in software form, or in one or more hardware circuits or integrated circuits, or in different networks and / or processor devices and / or microcontroller devices.

[0035] In the description of this application, it should be noted that, unless otherwise clearly specified and limited, the terms "installation", "connection", and "connection" should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection, an electrical connection, or mutual communication; it can be a direct connection, or an indirect connection through an intermediate medium, it can be the internal connection of two components or the interaction relationship between two components. For ordinary technicians in this field, the specific meanings of the above terms in this application can be understood according to specific circumstances.

[0036] In order to make the purpose, features and advantages of the present application more obvious and easy to understand, the present application is further described in detail below in conjunction with drawings and specific implementation methods.

[0037] Please refer to Figure 1 As shown, it is a block diagram of a mobile image map processing device 100 according to an embodiment of the present application. In some embodiments, the mobile image map processing device 100 can be an embedded device. For example, it can be implemented in a daughter board, having one or more external input and output interfaces 130 (such as PCI-Express, SCSI, SATA, USB, etc. interfaces) connected to a host bus. The daughter board can be inserted into the expansion slot of the host and communicate with the host and other devices installed on the host through the input and output interface 130.

[0038] In other embodiments, the mobile image map processing device 100 may be an independent device, and also communicates with other devices through one or more external input / output interfaces 130. For example, there is a universal information bus on the vehicle, and the mobile image map processing device 100 can be connected to the information bus through the input / output interface 130 to interact with other devices on the information bus. For example, military vehicles have buses such as MIL-STD 1553B and MIL STD-1776A.

[0039] The mobile image map processing device 100 may include one or more processors 110. The processor 110 may include a microprocessor, a micro-controller, a graphics processor (GPU, Graphics Processor Unit), a digital signal processor (digital signal processor) and / or an application-specific integrated circuit (ASIC, Application-Specified Integrated Circuits).

[0040] The processor 110 can execute an operating system stored in a non-volatile memory to control the mobile image map processing device 100. In addition to being a common operating system, the operating system can also be an operating system suitable for an embedded system, or a real-time operating system. The processor 110 can execute one or more application programs to actually operate the embodiments provided by the present application, wherein the application program includes a plurality of instructions executable by the processor 110.

[0041] The memory 120 may include a non-volatile memory (e.g., flash, EEPROM) for storing the above-mentioned application and operating system, a volatile memory (e.g., various DDR SDRAM) as a system memory, and / or a high-speed image memory for image processing. For example, the above-mentioned high-speed image memory may have two I / O ports, which can perform storage and reading operations simultaneously, so as to accelerate the image processing operation.

[0042] The mobile image map processing device 100 can be connected to one or more cameras 140 through the input / output interface 130. The above-mentioned cameras 140 and the mobile image map processing device 100 can be installed on the same vehicle. In one embodiment, there can be four cameras 140 installed in four corresponding directions of the front, rear, left and right of the vehicle. In another embodiment, there can be two wide-angle cameras (for example, the shooting angle exceeds 180 degrees), which are installed back to back in two corresponding directions of the vehicle. In a further embodiment, there can be three wide-angle cameras (for example, the shooting angle exceeds 120 degrees), whose shooting centers are separated by 120 degrees and installed in three corresponding directions of the vehicle.

[0043] In one embodiment, the lens characteristics of the above-mentioned cameras 140 may be the same. However, in another embodiment, since the vehicle is more likely to move forward or backward, in order to enhance the resolution of the front or rear, the camera shooting forward and / or backward may have a narrower shooting angle to reduce image distortion. In other words, the camera shooting leftward and / or rightward may have a wider shooting angle with less image distortion.

[0044] The processor 110 can make the camera 140 shoot and collect images through the input / output interface 130. In one embodiment, the output of the camera 140 is shared with other systems, such as an automatic visual driving system, an accident recording system, etc. Since the above-mentioned automatic visual driving system and / or accident recording system require high frame rate and high-quality images, and the mobile image map processing device 100 does not require continuous or continuous images. The mobile image map processing device 100 can intermittently or periodically obtain one frame from multiple frames of images that are continuously output. For example, when the frame rate of the camera 140 is 60 frames per second, one frame can be periodically obtained every two frames, so that the frame rate is 30 frames per second. Alternatively, the image frame is obtained only after receiving an instruction.

[0045] The processor 110 can obtain relevant information of the vehicle from the vehicle information system 150 of the same vehicle. The above information can be used to infer and estimate the position information of the vehicle after combining the movement and turning characteristics referred to by the physical model of the vehicle. In one embodiment, the above vehicle information system 150 can output a vehicle bus standard such as the Controller Area Network (CAN bus), and notify the control status of the vehicle on the bus through a broadcast mechanism. Combined with the vehicle-specific physical model information, the relative position of the vehicle after a certain time, the relative direction of the vehicle, and other information can be calculated.

[0046] In addition to obtaining relevant information of the vehicle from the vehicle information system 150, the relative position of the vehicle after a certain time, the relative orientation of the vehicle, and other information can also be estimated from the navigation and positioning system 160. The above-mentioned navigation and positioning system 160 can be a terminal device of one or more global satellite positioning systems (GNSS), such as the common global positioning system GPS, Beidou satellite positioning system, GLONASS system, Galileo satellite positioning system, etc. The above-mentioned navigation and positioning system can also be an inertial navigation system (INS) using a gyroscope, accelerometer, angular accelerometer or any combination thereof. It can also be a navigation and positioning system 160 combined with a global satellite positioning system (GNSS) and an inertial navigation system (INS). According to the trajectory drawn by the navigation and positioning system, the relative position of the vehicle after a certain time, the relative orientation of the vehicle, and other information can be estimated.

[0047] In one embodiment, the memory 120 includes an image map data structure. The processor 110 processes the image captured by the camera 140 and writes, replaces or displaces the processed image into the corresponding part of the map data structure according to the vehicle position and orientation corresponding to the time sequence.

[0048] The surrounding part of the image map near the vehicle can be output to a display module 170. The display module 170 can include a display in the cab or a helmet display of the driver. After viewing the surrounding map of the vehicle, the driver can enhance the situation perception and make driving the vehicle safer.

[0049] In one embodiment, all or part of the above-mentioned image map can also be output to a driving module 180. Using artificial intelligence, the driving module 180 can include automatic or assisted driving functions. If a dangerous area is found in the image map, it can automatically avoid it or warn the driver to avoid it. If an area to be approached is found in the image map, such as a charging connector, a parking space, a gas pump, etc., it can also be navigated automatically or with assisted driving.

[0050] Please refer to Figure 2A , which is a top view of the shooting range of multiple cameras 140 carried by a vehicle according to an embodiment of the present application. Figure 2A The vehicle is equipped with four cameras 140, which can be respectively converted into a forward camera shooting range 210, a backward camera shooting range 220, a left camera shooting range 230, and a right camera shooting range 240. The shooting angles of the shooting ranges 210 and 220 can be different. Similarly, the shooting angles of the shooting ranges 210 and 230 can also be different.

[0051] The imaging plane of the camera 140 carried by the vehicle is not parallel to the ground (or sea surface, hereinafter referred to as the ground), but has an angle with the ground. The mobile image map processing device 100 can convert the image formed on the imaging plane into an image of the ground according to the lens characteristics of the camera 140 by using a linear conversion or a nonlinear conversion corresponding to the lens characteristics. In one embodiment, the camera 140 can be focused to infinity, but the above-mentioned linear or nonlinear conversion only maps the image to the ground at a certain distance. Figure 2A As shown, the forward camera shooting range 210 may be a trapezoid.

[0052] Similarly, the images from the other three cameras can also be converted into shooting ranges 220, 230, and 240 respectively according to their lens characteristics. Figure 2A As shown, the camera shooting ranges 220-240 may also be trapezoids. Due to the configuration relationship of the camera 140 in the vehicle, the four trapezoids may be blank. The relationship between the above-mentioned shooting ranges 210-240 and the center position 299 of the vehicle is fixed. Therefore, in one embodiment, the processor 110 may write, replace or replace the above-mentioned shooting ranges 210-240 into the corresponding parts of the data structure of the above-mentioned map.

[0053] Since the above-mentioned shooting range 210-240 is a trapezoid, it is inconvenient to write it into the data structure and requires more computing resources. Therefore, a rectangular or square area 290 can be set corresponding to the center position 299 of the vehicle. The processor 110 can only write the above-mentioned shooting range 210-240 in the area 290 of the image map to save computing resources. However, it can be understood by ordinary technicians in this field that it is not necessary to set the above-mentioned area 290.

[0054] Although in Figure 2A In the top view shown, the vehicle is pointing toward the top or 0 degrees of the image map, but during the movement of the vehicle, the vehicle may point to other directions of the image map. A person skilled in the art can understand that the above-mentioned shooting ranges 210-240 and area 290 will turn on the image map as the vehicle moves.

[0055] Since there is a partial overlap between two adjacent shooting ranges, in one embodiment, the shooting range written later can cover the overlapped area of ​​the shooting range written earlier according to the writing sequence of the shooting ranges 210 to 240. In another embodiment, in order to compensate for the difference in the characteristics of the two lenses, such as different aberrations, the overlapping area or the edge of the image can be specially processed in the aforementioned linear conversion or nonlinear conversion to reduce the difference between the overlapping area and other adjacent areas.

[0056] In one embodiment, after a certain shooting range is written, it may have a brightness difference with the image map next to it, resulting in an obvious seam line in the image map. Therefore, the shooting range to be written can be adjusted so that the image map after writing the shooting range does not have an obvious sense of disobedience or high-frequency part. For example, the edge of the shooting range can be processed by gradient ratio fusion or other softening image processing methods.

[0057] Please refer to Figure 2B , which is a top view of the shooting range of multiple cameras 140 carried by a vehicle according to an embodiment of the present application. Figure 2B The vehicle is equipped with three cameras 140, which can be respectively converted into a forward camera shooting range 250, a left rear camera shooting range 260, and a right rear camera shooting range 270. The shooting angles of the shooting ranges 250, 260, and 270 can be the same or different.

[0058] It can be understood by those skilled in the art that, in addition to the number and installation direction of the cameras 140, Figure 2B The embodiments can be applied Figure 2A As mentioned above, the present application does not limit the number of cameras carried by a vehicle.

[0059] One of the advantages of the present application is that it saves computing resources. Since the vehicle carrying the mobile image map processing device 100 has quite a lot of restrictions, such as volume / capacity restrictions, weight restrictions, power supply restrictions, environmental control restrictions, and vibration restrictions, all of which will affect the design of the mobile image map processing device 100, it is necessary to save computing resources as much as possible, especially the frequency of image processing and writing image maps.

[0060] According to an embodiment of the present application, after receiving the vehicle information broadcasted by the vehicle resource system 150, the processor 110 calculates a new relative position and pointing angle according to the vehicle motion model. Figure 3As shown, it is a schematic diagram of a vehicle motion model according to an embodiment of the present application. The vehicle has a pair of rear wheels that cannot turn and a pair of front wheels that can turn at the same time. The center of the vehicle can be set to the midpoint of the rear wheel axle between the pair of rear wheels. Combining the wheelbase between the front wheel axle and the rear wheel axle and the front wheel steering angle, the corresponding relationship between the front wheel axle and the center of the vehicle (i.e., the midpoint of the rear wheel axle) can be obtained. The line connecting the two is the pointing angle of the vehicle. The x-axis and the y-axis can correspond to the two axes of the image map respectively.

[0061] As shown in Table 1 below, equations (1) to (4) are used to describe the vehicle motion model.

[0062]

[0063] Table 1 Vehicle motion model

[0064] The above vehicle motion model includes the wheelbase (L), wheel angle (φ), front / rear vehicle rotation radius (R f / R r ), the center coordinates of the front wheel axle and the center coordinates of the rear wheel axle ((x f ,y f ),(x r ,y r )), speed (v), vehicle steering angle (θ) and time (t) corresponding to the vehicle steering angle. The vehicle coordinates are calculated by introducing the vehicle speed and wheel steering angle, using the vehicle motion model to convert the driving distance and vehicle steering angle, and then continuously accumulating and calculating the estimated vehicle position information.

[0065] Assume that the global initial angle and coordinates of the vehicle are 90° and (0,0) respectively. The vehicle movement relationship at two moments can be decomposed into a rotation matrix R(θ) plus a translation matrix T(x). The rotation matrix describes the amplitude of the vehicle body angle transformation (θ), and the translation matrix expresses the movement relationship of the vehicle on the xy plane. The combination of the rotation matrix and the translation matrix is ​​the affine transformation matrix.

[0066]

[0067] Table 2 Affine transformation matrix

[0068] When the broadcast interval of vehicle information is short, it is assumed that the front wheel angle does not change, that is, the vehicle at two moments starts rotating with a fixed rotation center as the origin. FIG. 4A to FIG. 4C As shown, it can be concluded that the instantaneous rotation angle is the distance moved by the vehicle within two moments divided by the rotation radius of the rear wheel axle. The instantaneous rotation angle can be continuously accumulated to obtain the vehicle pointing angle (θ).

[0069]

[0070] Table 3 Line segment detection loss function

[0071] Furthermore, the displacement of the vehicle is obtained by the distance moved by the center of the vehicle at two moments and the vehicle pointing angle (θ). Figure 4C As shown in Table 3, where: T C is the translation distance of the vehicle, α is the angle between the rear axle and the rotation center and the vehicle center, and ω is the translation direction. T is calculated from the above parameters. C The component T on the x-axis X With component Ty on the y-axis.

[0072] In one embodiment, the estimated relative displacement of the vehicle and the vehicle pointing angle can be obtained based on the vehicle information through the above calculation. In another embodiment, the estimated relative displacement of the vehicle and the vehicle pointing angle can also be obtained according to the positioning information provided by the navigation and positioning system 160. Therefore, the present application can calculate the image map update point, and when the vehicle travels to the image map update point or within the error range nearby, the above-mentioned image acquisition steps are executed, the shooting range is processed (such as ground projection, map coordinate transformation, brightness conversion and other steps), and the shooting range is written, replaced and substituted to the corresponding part of the image map.

[0073] Please refer to Figure 5 As shown, it is a schematic diagram of an interval updated image map according to an embodiment of the present application. The image map 500 is a data structure present in the aforementioned memory 120. When the application is initialized, the image map 500 can be filled with initialized image values, such as all white or all black. The position 510 of the center of the vehicle (which can be the center point of the rear axle) can also be configured at the initialization point of the image map 500, such as the center point, during initialization. After initialization, the shooting range after the first processing can be written, replaced or substituted to the initialized image value of the corresponding part.

[0074] When the vehicle moves, the processor 110 can obtain the estimated relative displacement and pointing angle of the vehicle after a certain time point from the aforementioned vehicle information and / or historical positioning information. The above time point can be fixed or change with the speed of the vehicle. For example, when the shooting range is 30 meters away and the speed is 2 meters per second, the time point for updating the image map can be set after 0.5 seconds. When the speed becomes faster to 10 meters per second, the time point for updating the image map can be set after 0.1 seconds.

[0075] In order to provide continuous situational awareness, in some cases, the time to update the image map is set to the minimum value even when the vehicle is stationary, that is, when the estimated vehicle position and vehicle heading angle are the same as the current vehicle position and vehicle heading angle. For example, when the vehicle is stationary, the time to update the image map can be set to 0.8 seconds or 1 second.

[0076] In one embodiment, the processor 110 can calculate the relative displacement and the pointing angle of the vehicle at the time point of updating the image map, and execute the above-mentioned image acquisition steps, process the shooting range (such as ground projection, map coordinate transformation, brightness conversion, etc.), and write, replace and substitute the shooting range to the corresponding part of the image map. Then, calculate the next time point of updating the image.

[0077] by Figure 5 For the embodiment shown, when the center of the vehicle is at position 510, the shooting range 515 in the image map 500 can be updated. Then, the time point for the next image update is calculated. After the time point for the next image map update, the vehicle has moved to the position 520 of the second vehicle center, and its corresponding shooting range 525 also changes with the displacement of the vehicle center and the pointing angle of the vehicle. Similarly, the processor 110 can perform the above-mentioned image acquisition steps, process the shooting range (such as ground projection, map coordinate transformation, brightness conversion and other steps), and write, replace and substitute the shooting range to the corresponding part of the image map. Then, the time point for the next image update is calculated, and this cycle is repeated.

[0078] In another embodiment, the processor 110 may continue to receive the aforementioned vehicle information and / or positioning information, and calculate the current relative displacement and vehicle pointing angle of the vehicle. When certain conditions are met, such as when the rotation angle of the relative displacement and / or vehicle pointing angle exceeds a critical value, the above-mentioned image acquisition steps may be performed, the shooting range may be processed (such as ground projection, map coordinate transformation, brightness conversion, etc.), and the shooting range may be written, replaced, and substituted to the corresponding part of the image map. Next, the processor 110 may continue to receive the aforementioned vehicle information and / or positioning information, calculate the current relative displacement and vehicle pointing angle of the vehicle, and determine whether the above-mentioned conditions are met, and repeat this process.

[0079] Please refer to Figure 6 , which is a schematic diagram of updating an image map in a time sequence according to an embodiment of the present application. Figure 6The upper part includes the shooting ranges in the multiple image update steps. For the sake of easy understanding, these shooting ranges can belong to the above-mentioned area 290. The lower part contains the image map. The middle time sequence represents the six image update steps from left to right. Figure 6 In the embodiment shown, the vehicle continues to make a left turn.

[0080] As mentioned above, when a certain period of time has passed or certain conditions are met, the processor 110 can perform an image update. In the first image update step, the image area of ​​the first shooting range is updated to the corresponding part of the initialized image map to form an image map P. In the second image update step, the image area of ​​the second shooting range is updated to the corresponding part of the image map to form an image map P+1. In the image map P+1, since the vehicle turns forward to the left, the image area of ​​the second shooting range is updated to the left front of the first image update part. And because the pointing angle of the vehicle rotates counterclockwise, the image area of ​​the second shooting range also rotates counterclockwise.

[0081] This cycle repeats itself, and in the sixth image update step, the image area of ​​the sixth shooting range is updated to the corresponding part of the image map, forming an image map P+5. Figure 6 In the lower part of the image map, you can see how the image map is updated sequentially.

[0082] According to an embodiment of the present application, the processor 110 may periodically read a portion of the image map according to the current position and pointing angle of the vehicle in the image map according to the display frame rate of the display module 170. In one embodiment, the portion of the image may include an image of the vehicle near the image map.

[0083] As mentioned above, the execution frequency or rate of the above-mentioned reading step is determined according to the needs of the display module 170, and has nothing to do with whether the above-mentioned image update conditions are met. For example, when the vehicle is stationary for five minutes. Within these five minutes, the processor 110 will perform 300 image updates according to the minimum update time interval required to provide situation awareness (such as the aforementioned 1 second). However, when the display module 170 displays at 60 frames per second, 18,000 display steps are executed within these five minutes.

[0084] Since the vehicle may be stationary at a high rate, the memory 120 module may include a high-speed display image map cache. Each time a portion of the image map is read, the read portion of the image may be stored in the high-speed display image map cache. If the vehicle is stationary, the content of the display image map cache may be provided to the display module 170 without the processor 110 having to read the image map each time, thereby further saving computing resources.

[0085] Please refer to Figure 7 FIG. 7 is a block diagram of a mobile image map processing method 700 according to an embodiment of the present application. The mobile image map processing method 700 may be Figure 1 The mobile image map processing device 100 shown is executed, in particular, by an application consisting of a plurality of instructions executed by the processor 110. If there is no direct or indirect causal relationship between any two steps, the present application does not limit the execution order of the two steps. The mobile image map processing method 700 can start to execute the initialization image map method 710 after the vehicle is started. In one embodiment, after the image map is initialized, the update image map method 720 and the display image map method 730 can be executed in parallel. In one embodiment, the processor 110 may include a plurality of execution units, which are respectively used to execute the update image map method 720 and the display image map method 730 in parallel. In another embodiment, the operating system executed by the processor 110 can support multi-thread (multi-thread) or multi-process multi-tasking functions, and virtually execute the update image map method 720 and the display image map method 730 in parallel. Ordinary technicians in this field should have knowledge of information science and information engineering, and understand the various changes of the above embodiments.

[0086] Please refer to Figure 8 , which is a block diagram of a method 710 for initializing an image map according to an embodiment of the present application. The method 710 for initializing an image map may start from step 810 .

[0087] Step 810: Create a blank image map. In one embodiment, a data structure may be constructed in the memory 120 to store a blank image map. In another embodiment, the data structure may be initialized so that the image value of the image map is an initialization value, or a blank value. For example, the initialization value may be a full black value, a full white value, or a transparent value. In one embodiment, the image map includes two perpendicular direction axes, one of which may be aligned to the vehicle pointing angle.

[0088] Step 820: Capture images captured by more than one camera 140. In one embodiment, the captured images may be stored in parallel via direct memory access (DMA).

[0089] Step 830: Process the captured image according to the lens characteristics of the camera. As mentioned above, the above image can be converted into a ground image through linear or nonlinear conversion. As mentioned above, the brightness value of the ground image can also be adjusted according to the difference in brightness between adjacent images. It can be understood by ordinary people in this field that when there are more than two cameras 140, the scales of the processed multiple ground images should be consistent.

[0090] Step 840: Receive vehicle information, and write the processed image to the corresponding position of the image map according to the received vehicle information. The above-mentioned vehicle information may include the position of the vehicle and the vehicle pointing angle. In one embodiment, since the map is initialized, the position of the vehicle can be located in the center of the blank image map, and the vehicle pointing angle can point to the axis of one of the two axes of the blank image map. In another embodiment, the vehicle pointing position stored by the mobile image map processing device 100 before shutdown can be read, without the need for the previous vehicle pointing angle to point to the axis of one of the two axes of the blank image map.

[0091] Optional step 850: Setting conditions for updating the image map. Various conditions for updating the image map have been mentioned previously and will not be repeated here. After executing the initializing image map method 710, the updating image map method 720 and the displaying image map method 730 may be executed in parallel.

[0092] Please refer to Fig. 9 , which is a block diagram of a method 720 for updating an image map according to an embodiment of the present application. The method 720 for updating an image map may start from step 910 .

[0093] Step 910: Receive information required for the following decision step 920.

[0094] Step 920: Determine whether the conditions for updating the image map are met. If so, the process proceeds to step 930. If not, the process may proceed to optional step 930, optional step 940, or directly return to step 910.

[0095] Step 930: Receive vehicle information and capture images captured by more than one camera 140. In one embodiment, the captured images can be stored in parallel via direct memory access (DMA).

[0096] Step 940: Process the above-mentioned image according to the lens characteristics of the camera and the vehicle information, such as ground projection, map coordinate transformation, brightness conversion, etc.

[0097] Step 950: Write the processed image to the corresponding position of the image map according to the received vehicle information.

[0098] Step 960: Set the conditions for updating the image map. Then, the process can return to step 910.

[0099] Optional step 970: tidy up the image map 930. Since the initialized image map cannot be infinitely large, the image map that is too far from the vehicle position can be deleted or the blank image map can be expanded when the conditions for updating the image map are not met. In one embodiment, step 930 can be performed in another parallel thread or process without affecting the update of the map.

[0100] Optional step 980: Pause for a period of time. As mentioned above, when the vehicle is stationary or moving very slowly, it may pause for a period of time to determine whether the conditions for updating the image map are met. Then, the process returns to step 910.

[0101] Please refer to Fig.10 , which is a block diagram of a method 730 for displaying an image map according to an embodiment of the present application. The method 730 for displaying an image map may start from step 1010 .

[0102] Step 1010: Receive vehicle information.

[0103] Step 1020: Read the corresponding part of the image map according to the vehicle information. When reading, the corresponding part of the image map can be converted into the corresponding direction of the vehicle pointing angle.

[0104] Step 1030 : Transmit the corresponding portion of the read image map to the display module 170 .

[0105] Optional step 1040: Pause for a period of time. Since the update rate of the display module 170 (eg, 30 frames per second) may be lower than the running time of step 1010 to step 1030, it is possible to pause for a period of time and then return to step 1010.

[0106] According to one embodiment of the present application, a method for updating an image map applicable to a vehicle is provided, comprising: an information receiving step, receiving information used to determine whether a condition for updating the image map is satisfied; a determination step, determining whether the condition for updating the image map is satisfied based on the information; when it is determined that the condition is satisfied, executing a receiving step, receiving vehicle information and images captured by one or more cameras installed on the vehicle; an image processing step, processing the images separately based on the lens characteristics of the cameras and the received vehicle information; and a writing step, writing the processed images to a corresponding position of the image map based on the received vehicle information.

[0107] Preferably, in order to know the corresponding position of the image map, the vehicle information includes the current position and current pointing angle of the vehicle.

[0108] Preferably, in order to obtain the vehicle information, the vehicle information comes from one or any combination of the following: a vehicle information system; and a navigation and positioning system.

[0109] Preferably, in order to make the brightness of the image in the image map smooth, the image processing step further includes adjusting the brightness of the written image according to one or any combination of the following: according to the existing images around the corresponding position of the image map; and according to the adjacent received image.

[0110] Preferably, in order to satisfy the driver's situational awareness, the above-mentioned conditions for updating the image map include one of the following or any combination thereof: the relative displacement of the vehicle exceeds a critical value; the relative pointing angle of the vehicle exceeds another critical value; the current speed of the vehicle exceeds a speed critical value; the steering angle of the vehicle's steering mechanism exceeds a steering angle critical value; and the time from the last writing step exceeds a time critical value.

[0111] Preferably, in order to provide a condition for determining whether to update the image map next time, the above-mentioned method for updating the image map further includes: setting the condition for updating the image map.

[0112] Preferably, in order to utilize time to organize the image map, the above-mentioned method for updating the image map further includes executing one of the following or any combination thereof when it is determined that the condition is not satisfied: organizing the image map; and pausing for a period of time.

[0113] Preferably, in order to display the portion of the image map corresponding to the current position of the vehicle, a method for displaying an image map is provided, comprising executing the method for updating the image map as described above, and the following steps: a step of receiving vehicle information; reading the corresponding portion of the image map based on the above-mentioned vehicle information; and transmitting the read corresponding portion of the image map to a display module for display.

[0114] Preferably, in order to initialize the image before updating the image map, a method for initializing the image map is provided, comprising executing the following steps before the aforementioned method for updating the image map: establishing a blank image map; executing the aforementioned receiving step; executing the aforementioned image processing step; executing the aforementioned writing step; and executing the method for updating the image map.

[0115] According to an embodiment of the present application, a mobile image map processing device is provided, comprising: at least one processor for executing a plurality of instructions to implement the aforementioned image map updating method; and a memory for storing the image map.

[0116] According to one embodiment of the present application, a mobile image map processing device is provided, comprising: at least one processor for executing multiple instructions to implement the aforementioned method of displaying an image map; a memory for storing the image map; and an input / output interface for transmitting the corresponding portion of the read image map to a display module.

[0117] According to an embodiment of the present application, a mobile image map processing system is provided, comprising: the aforementioned mobile image map processing device; and the display module.

[0118] According to an embodiment of the present application, a vehicle having a mobile image map processing capability is provided, comprising: the aforementioned mobile image map processing system; and the camera.

[0119] According to an embodiment of the present application, a mobile image map processing device is provided, comprising: at least one processor for executing a plurality of instructions to implement the aforementioned method of initializing an image map; and a memory for storing the image map.

[0120] According to various embodiments, the present application provides an information system that can provide vehicle surrounding perception capabilities using less computing resources to save costs, and can provide drivers with situational awareness capabilities at normal driving speeds.

Claims

1. A method for updating an image map applicable to a vehicle, characterized in that: Include: The step of receiving information is to receive information for determining whether a condition for updating the image map is satisfied; the step of determining whether the condition for updating the image map is satisfied based on the information; and when the condition is satisfied, executing: A receiving step, receiving vehicle information and collecting images taken by one or more cameras installed on the vehicle; an image processing step, processing the above-mentioned images respectively according to the lens characteristics of the above-mentioned camera and the received vehicle information; and The writing step is to write the processed image into the corresponding position of the image map according to the received vehicle information.

2. The method for updating an image map according to claim 1, wherein: The vehicle information includes the current position and current pointing angle of the vehicle.

3. The method for updating an image map according to claim 1, wherein: The vehicle information comes from one or any combination of the following: a vehicle information system; and a navigation and positioning system.

4. The method for updating an image map according to claim 1, wherein: The image processing step further includes adjusting the brightness of the written image according to one of the following or any combination thereof: according to existing images around the corresponding position of the image map; and according to the adjacent received image.

5. The method for updating an image map according to claim 1, wherein: The above-mentioned conditions for updating the image map include one of the following or any combination thereof: the relative displacement of the vehicle exceeds a critical value; the relative pointing angle of the vehicle exceeds another critical value; the current speed of the vehicle exceeds a speed critical value; the steering angle of the steering mechanism of the vehicle exceeds a steering angle critical value; and the time from the last writing step exceeds a time critical value.

6. The method for updating an image map according to claim 1, wherein: Furthermore, the method includes setting the condition for updating the image map.

7. The method for updating an image map according to claim 1, wherein: It further includes executing one of the following or any combination thereof when it is determined that the condition is not satisfied: organizing the image map; and pausing for a period of time.

8. A method for displaying an image map, characterized in that: It includes executing a method for updating an image map as described in any one of claims 1 to 7, and the following steps: a step of receiving vehicle information; reading a corresponding portion of the image map based on the above-mentioned vehicle information; and transmitting the read corresponding portion of the image map to a display module for display.

9. A method for initializing an image map, characterized in that: The method comprises executing the following steps before the method for updating the image map as described in any one of claims 1 to 7: establishing a blank image map; executing the receiving step; executing the image processing step; executing the writing step; and executing the method for updating the image map.

10. A mobile image map processing device, characterized in that: The method comprises: at least one processor for executing a plurality of instructions to implement the method for updating an image map as claimed in any one of claims 1 to 7; and a memory for storing the image map.

11. A mobile image map processing device, characterized in that: It comprises: at least one processor for executing multiple instructions to implement the method for displaying an image map as described in claim 8; a memory for storing the image map; and an input / output interface for transmitting the corresponding part of the read image map to a display module.

12. A mobile image map processing system, characterized in that: It comprises the mobile image map processing device as claimed in claim 11; and the display module.

13. A vehicle with mobile image map processing capability, characterized in that: It comprises: the mobile image map processing system as claimed in claim 12; and the camera.

14. A mobile image map processing device, characterized in that: The method comprises: at least one processor for executing a plurality of instructions to implement the method for initializing an image map as claimed in claim 9; and a memory for storing the image map.

Images