Multi-module image system and image synchronization method

Abstract

A multi-module image system includes a camera assembly having a plurality of photosensitive modules, an image signal processor and a data processor, and the system generates a plurality of groups of dynamic images via different photosensitive modules in the camera assembly. The image signal processor processes the dynamic images to obtain continuous frame images from each group of dynamic images, and then the data processor obtains multiple frames of images generated at the same time from multiple groups of continuous frame images, and the multiple frames of images generated at the same time can be combined into a combined frame in a vertical mode through a vertical coding method. The plurality of groups of dynamic images are coded to form a plurality of combined frames which are continuously output, so that the multi-module image system can synchronously output a plurality of frames of images generated by different photosensitive modules.

Description

Technical Field

[0001] The specification discloses a technology for synchronous image output, specifically a multi-module image system and an image synchronization method for splicing multiple images output by an image signal processor to achieve synchronous images. Background Art

[0002] The conventional automatic license plate recognition (ALPR) system is equipped with multiple photosensitive modules, including an infrared sensor and a visible light sensor that can obtain color images, so as to simultaneously obtain the image in front of the vehicle and the license plate image in the image in front. The color image obtained by the visible light sensor can cover the vehicle and the environment, but it is easily affected by the change of ambient light and causes the problem of license plate recognition error. Therefore, an infrared sensor is used to obtain the license plate image. The infrared light image is not affected by the ambient light. Specifically, a clear license plate image can be obtained smoothly under low light and bad weather conditions, thereby improving the efficiency of license plate recognition.

[0003] For example, the camera equipment in police vehicles is equipped with an automatic license plate recognition system, which can automatically detect the license plate of the vehicle in front and perform license plate recognition. When the vehicle in front uses a license plate that reflects light, the light from the police vehicle behind will cause glare on the license plate in front, thus affecting the effect of license plate recognition. Therefore, infrared light sensors can be used to filter out the glare and obtain clear license plate characters.

[0004] However, the images obtained by using infrared light sensors and visible light sensors are compressed and output separately. After decompression, the images obtained separately need to be output synchronously to ensure that they are images taken at the same time. In the prior art, for example, in a vehicle-mounted imaging system using infrared light and visible light camera modules, the circuits and software programs that process the infrared light sensor and the visible light sensor separately in the system need to handle the image synchronization problem at the back end, such as using the time code (timecode) or timestamp (timestamp) when the image is output for image synchronization, but this needs to be run on a device with greater computing power and consumes computing and time resources. Summary of the invention

[0005] In view of the fact that conventional technology requires the use of a device with greater computing power to synchronize images when processing images generated by multiple photosensitive modules, but is not suitable for edge devices with less computing power, such as a driving recorder, the present invention proposes a method for executing image synchronization with a device with less computing power, and a multi-module imaging system for running the image synchronization method.

[0006] The proposed multi-module imaging system includes a camera assembly having multiple photosensitive modules, an image signal processor and a data processor, and the multi-module imaging system executes an image synchronization method.

[0007] In the image synchronization method, a plurality of sets of dynamic images are generated by a camera component having a plurality of photosensitive modules, and then a set of continuous frame images is obtained from one set of dynamic images generated by each photosensitive module in the camera component through an image signal processor. Therefore, the plurality of sets of dynamic images generated by different photosensitive modules will form a plurality of sets of continuous frame images.

[0008] Next, a data processor combines multiple frames of images obtained at the same time from multiple groups of continuous frame images into a combined frame. Therefore, multiple groups of dynamic image encoding form multiple combined frames that are output continuously, wherein each combined frame includes multiple frames of images generated by the camera component at the same time, thereby enabling the multi-module imaging system to synchronously output multiple frames of images generated by different photosensitive modules.

[0009] When multiple groups of dynamic images are obtained, the image signal processor executes an image processing program on each group of dynamic images, including performing one or any combination of noise reduction, color correction, white balance adjustment, contrast adjustment, saturation adjustment and sharpness enhancement optimization programs on each frame image in each group of dynamic images.

[0010] Furthermore, when a plurality of combined frames are continuously generated, the plurality of combined frames can be transmitted to an external system in a time sequence, and in the external system, a plurality of synchronized sets of continuous frame images can be obtained through a decoding process.

[0011] In one embodiment, the multi-module imaging system is provided in a driving recorder, and the driving recorder includes a communication module that can transmit multiple combined frames to the external system in a wireless or wired manner.

[0012] The camera assembly may include an infrared light module and at least one visible light module, and the formed combined frame includes an infrared light image generated by the infrared light module and at least one visible light image generated by the at least one visible light module.

[0013] Furthermore, the multi-module imaging system can be applied to an automatic license plate recognition system, so that the license plate image can be obtained through the infrared light module, and the environment image including the vehicle can be obtained through the visible light module. The automatic license plate recognition system is used to recognize multiple characters of English and digital combination in the license plate image, and the multiple characters obtained by recognition can also be combined in the combined frame.

[0014] Furthermore, the camera assembly may adopt a global shutter so that all the photosensitive pixels of each photosensitive component in the infrared light module and at least one visible light module are simultaneously photosensitive to generate a synchronously photosensitive frame image.

[0015] Furthermore, the combined frame formed by the multiple frames of images obtained at the same time is a spliced ​​frame formed by the data processor combining the multiple frames of images in a splicing manner, and, specifically, a vertical encoding method can be used to splice the multiple frames of images obtained at the same time. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 A schematic diagram showing a situation in which a multi-module imaging system captures an image;

[0017] Figure 2A A schematic diagram showing an image of a black background and a bright license plate captured by an infrared light module;

[0018] Figure 2B A schematic diagram showing images continuously generated by the first camera module;

[0019] Figure 2C A schematic diagram showing images continuously generated by the second camera module;

[0020] Figure 3 A schematic diagram showing an embodiment of a circuit assembly of a multi-module imaging system;

[0021] Figure 4 A schematic diagram showing an embodiment of obtaining a combined frame by a vertical coding method;

[0022] Figure 5 A schematic diagram showing an embodiment of an external system decoding a split frame image;

[0023] Figure 6 A flowchart showing an embodiment of an image synchronization method in a multi-module image system; and

[0024] Figure 7 An example diagram showing a combined frame and a corresponding image. DETAILED DESCRIPTION

[0025] The present invention provides a multi-module imaging system and an image synchronization method. The multi-module imaging system is a system including multiple camera components. Multiple camera components are provided in an electronic device, which also includes multiple lenses, multiple photosensitive components and related processing circuits. Figure 1 A schematic diagram of a situation in which images are captured by a multi-module imaging system.

[0026] This example shows the scene in front captured by the driving recorder 10 on the vehicle. Specifically, the driving recorder 10 is provided with a multi-module imaging system, which includes a first camera module 101 and a second camera module 102. The two camera modules are respectively provided with respective camera lenses to capture images of two overlapping perspectives in front of the vehicle. The images cover the vehicle 110 in front. One of the main purposes is to obtain an image of the license plate 111 of the vehicle 110 in front and perform license plate recognition.

[0027] In one embodiment, it is used in license plate recognition, and a clear license plate image needs to be obtained. The driving recorder 10 can use image photosensitive components of various bands, such as using two or more visible light photosensitive components, or using infrared light photosensitive components and visible light photosensitive components at the same time, wherein the infrared light photosensitive component is used to obtain the license plate image in the infrared light band, and the visible light photosensitive component is used to obtain the color image. Furthermore, the camera component can also use a photosensitive component with a global shutter (GS) so that all the photosensitive pixels of each photosensitive component in the infrared light module and at least one visible light module are simultaneously photosensitive to generate a frame of image with synchronous photosensitive. Among them, the use of a global shutter has other advantages, such as preventing image distortion problems; in addition, the photosensitive component in the global shutter and the infrared light source (such as an IR LED flash) can be used to synchronize the exposure time at a high speed to reduce the heating of the photosensitive component.

[0028] Taking a police vehicle as an example, the driving recorder 10 in the police vehicle captures the front image all the time while driving, and can obtain images in real time and recognize multiple license plates (including license plate 111) of multiple vehicles (including vehicle 110) in real time during law enforcement. In addition to performing the required license plate recognition, the driving recorder 10 in the police vehicle can also determine the traffic and road conditions based on the front image.

[0029] The images produced by the multi-module imaging system can be referred to FIG. 2A to FIG. 2C Schematic diagram shown.

[0030] Figure 2A The schematic diagram shows an image captured by an infrared light module in front of a vehicle. The infrared light image 201 shown in this example has a relatively narrow viewing angle and displays an image with a black background and a bright license plate.

[0031] Figure 2B and Figure 2C Shown is a schematic diagram of an embodiment of a visible light image generated by a multi-module imaging system, wherein a license plate image in the image is obtained through image processing or intelligent model recognition.

[0032] Figure 2BThe multi-module imaging system is shown as a license plate image 202 that is obtained by continuously generating images through the camera modules and then undergoing image processing or intelligent model recognition. Figure 2C The diagram schematically shows a visible light image 203 continuously generated by the visible light module, which covers the image of the entire scene in front of the vehicle.

[0033] Different from the conventional technology which needs to be processed post-production to synchronize the continuous images captured by the camera modules, the multi-module imaging system and image synchronization method proposed in the present invention directly combines the images output by the image signal processor (ISP) in the multi-module imaging system. The images output by the multi-module imaging system are not multiple continuous images generated by different camera modules, but a continuous output of combined frames.

[0034] The circuit design of multi-module imaging system can refer to Figure 3 A schematic diagram of an embodiment of a circuit component of a multi-module imaging system is shown. Figure 3 The multi-module imaging system shown in the figure can be provided in a driving recorder, wherein the main circuit includes a camera assembly 300 having multiple photosensitive modules electrically connected to each other, an image signal processor 303, a data processor 304 and a memory 305, etc. It can also include a communication module 306, which can transmit images to an external system in a wireless or wired communication manner.

[0035] The camera assembly 300 of the multi-module imaging system includes multiple photosensitive modules, such as Figure 3 The first photosensitive module 301 and the second photosensitive module 302 in the embodiment are not limited to photosensitive components of a specific wavelength band in actual implementation. According to one embodiment, the first photosensitive module 301 can be an infrared light module for obtaining infrared light images; the second photosensitive module 302 can be a visible light module for generating visible light images. The number of photosensitive modules is not limited to this. Figure 3 shown.

[0036] The image signal processor 303 in the multi-module imaging system is used to process the electrical signals generated by the first photosensitive module 301 and the second photosensitive module 302 and convert them into image data. Taking the infrared light module and the visible light module as an example, the image signal processor 303 processes the image signal to obtain the infrared light image and the visible light image.

[0037] Furthermore, the multi-module imaging system generates multiple groups of dynamic images through multiple photosensitive modules in the camera component 300, and the image signal processor 303 can execute an image processing program on each group of dynamic images, including performing one or any combination of noise reduction, color correction, white balance adjustment, contrast adjustment, saturation adjustment and sharpness enhancement optimization programs on each frame image in each group of dynamic images.

[0038] The dynamic image data generated by the image signal processor 303 includes continuous frame images, and multiple photosensitive modules respectively generate multiple sets of continuous frame images, which can be temporarily stored in the memory 305, and then obtained by the data processor 304 to execute the image synchronization program, and combine the multiple frame images generated at the same time from the image signal processor 303, such as splicing, to generate a combined frame at one time. The combined frame can then be transmitted to the external system via the communication circuit 306.

[0039] According to an embodiment, the camera component in the multi-module imaging system may include an infrared light module and at least one visible light module, which respectively generate infrared light continuous frames and visible light continuous frames, so that the generated combined frame is a combination of infrared light images and at least one visible light image.

[0040] Taking the dashcam installed in the police vehicle as an example, the multi-module imaging system can continuously generate combined frames obtained by splicing (synchronization), wherein the first image can be an infrared image to obtain a license plate image that is not affected by ambient light, and the second image is a visible light image including all the scenery in front of the vehicle. The continuously generated combined frames can be transmitted to the computer device in the police vehicle through the communication circuit in the dashcam for subsequent law enforcement use; on the other hand, they can also be transmitted to the control center for future reference.

[0041] Examples can be referenced Figure 4 The diagram shows an embodiment of obtaining a combined frame by a vertical coding method.

[0042] The camera assembly 400 in the multi-module imaging system includes an infrared light module 401 and a visible light module 402. The multi-module imaging system can be applied to an automatic license plate recognition (ALPR) system. The infrared light module 401 can obtain the license plate image of the vehicle that is not affected by the ambient light, so that the system can still obtain the image that can identify the license plate content in low light or bad weather and poor vision scenes, and the automatic license plate recognition system continues to recognize the multiple characters of English and digital combination in the license plate image. At the same time, the visible light module 402 can also be used to obtain the environmental image including the vehicle. The obtained visible light image can be a color image formed by red, blue and green light, which can be used to judge the color, brand or manufacturer, model and other details of the vehicle.

[0043] The multi-module imaging system generates multiple sets of dynamic image signals through multiple photosensitive modules such as infrared light module 401 and visible light module 402. The image signals generated at the same time are processed by image signal processor 403 to form first image 405 and second image 406 respectively. Figure 3The encoding program executed in the data processor 304 in the multi-module image system shown in the figure, as shown in the figure, the encoder 407 performs encoding, and the first image 405 and the second image 406 are spliced ​​to form a combined frame 408 of the icon. This schematic diagram shows that the combined frame 408 can be a spliced ​​frame formed by the system using a vertical encoding method to splice multiple frames of images.

[0044] The combined frame 408 includes multiple frames of images generated by the camera assembly 400 at the same time, so that the multi-module imaging system synchronously outputs multiple frames of images generated by different photosensitive modules without the need for other post-production programs to synchronize the images generated by different photosensitive modules according to time information.

[0045] Next, the combined frame 408 that has completed image synchronization can be transmitted to another system, as shown in FIG. Figure 5 The schematic diagram of an embodiment of an external system decoding separated frame images is shown.

[0046] When the combined frame 408 including the first image 405 and the second image 406 is transmitted to the external system, the external system 408 decodes the combined frame 408 according to the encoding format through the decoder 501 to separate the combined frame 408 to restore the first image 405 ′ and the second image 406 ′ and store them in the storage 505 .

[0047] Above Figure 4 The image synchronization method showing the combined frame obtained by the vertical coding method can be referred to Figure 6 Flow chart of the embodiment shown.

[0048] Multiple groups of dynamic images are generated by multiple photosensitive modules in the camera assembly (step S601), referring to Figure 4 In the illustrated embodiment, different photosensitive modules (infrared light module 401, visible light module 402) continuously generate infrared light images and visible light images. Then, through the image signal processor 403, a group of continuous frame images is obtained from one group of dynamic images, and multiple groups of dynamic images are used to obtain multiple groups of continuous frame images (step S603). Then, multiple frame images (first image 405 and second image 406) obtained at the same time from the multiple groups of continuous frame images are obtained (step S605). The frame images at the same time are encoded by the encoder 407 to form a combined frame 408 (step S607). Therefore, the multiple groups of dynamic images are encoded into multiple combined frames 408 that are continuously output, wherein each combined frame 408 includes multiple frame images generated by the camera component at the same time, thereby synchronously and continuously outputting multiple combined frames.

[0049] Afterwards, according to a purpose, multiple combined frames can be continuously output in time sequence, such as Figure 5In the display embodiment, the continuously output combined frames are transmitted to the external system, and then the external system is decoded to obtain multiple sets of synchronized continuous frame images. Furthermore, since the infrared light image in the combined frame can be a license plate image, the automatic license plate recognition system used can successfully recognize the license plate, and finally the multiple characters of the recognized license plate can be combined in the combined frame.

[0050] Figure 7 A diagram showing an embodiment of splicing multiple frames of images acquired at the same time using a vertical encoding method to form a vertically spliced ​​combined frame.

[0051] The first image 701 is displayed to be composed of pixels with pixel numbers 0 to 79, and then the second image 702 is displayed to be composed of pixels with pixel numbers 80 to 159 in the vertical direction. Figure 7 In the illustrated example, the vertically encoded first image 701 may correspond to Figure 2A The infrared image 201 captured by the infrared module is displayed, and the second image 702 can correspond to Figure 2C The visible light image 203 captured by the visible light module is shown. According to the embodiment of the image synchronization method, the combined frame is not limited to the vertical encoding method shown in the figure, and there are other horizontal encoding methods. However, the advantage of the combined frame obtained by vertical encoding is that in the external system, as long as the encoder knows the size or resolution of the image (the number of pixels in the length and width direction), it can immediately know the dividing point of adjacent images in the combined frame without the need for an edge-finding algorithm.

[0052] It is worth mentioning here that, taking the combined frame obtained by horizontal encoding as an example, although it also includes synchronized images, when decoding, horizontal scanning is required to detect the boundary of the image, and then the image can be separated based on this boundary; taking the combined frame obtained by vertical encoding as an example, when decoding, there is no need to find the edge, and the boundary that can separate the two images can be obtained based on the image size or resolution.

[0053] like Figure 7 In the embodiment shown, when the external system receives the combined frame, the image information therein can be obtained at the same time. For example, both images are 10 by 8 pixel images. Therefore, when it is known that the pixel numbers of the first image are 0 to 79, it can be inferred that the pixel numbers of the second image are 80 to 159, and the boundary of the two images can be obtained. It is mentioned here that the image synchronization method can still be used for the combined frame obtained by vertically encoding images of different sizes or resolutions. Accordingly, the decoding end can perform decoding according to the known image size.

[0054] In summary, according to the multi-module imaging system and image synchronization method described in the above embodiments, the system has multiple photosensitive modules and can take multiple images at the same time, which may include infrared images that are beneficial for identifying license plate images, and visible light images of the vehicle and its environment. In addition to successfully identifying license plates, the overall environment image can also be obtained. The image synchronization method uses splicing of images generated at the same time to obtain the effect of synchronized images, which is different from the conventional technology that requires a certain amount of computing power to process synchronized images. It further uses a vertical coding splicing method, which is a method that is beneficial for the decoding end to quickly separate the spliced ​​images.

[0055] The contents disclosed above are only preferred feasible embodiments of the present invention, and are not intended to limit the scope of the present invention. Therefore, all equivalent technical changes made using the contents of the present invention's specification and drawings are included in the scope of the present invention.

Claims

1. A multi-module imaging system, characterized in that: The multi-module imaging system comprises: A camera assembly having a plurality of photosensitive modules; an image signal processor; a data processor electrically connected to the camera assembly and the image signal processor; and An image synchronization method is executed in the multi-module image system, and the image synchronization method includes: Producing multiple sets of dynamic images by the camera assembly having the multiple photosensitive modules; By using the image signal processor, a group of continuous frame images is obtained from one group of dynamic images generated by each photosensitive module in the camera assembly, so that the multiple groups of dynamic images form multiple groups of continuous frame images; and Through the data processor, multiple frame images obtained at the same time from the multiple groups of continuous frame images are combined to form a combined frame, so that the multiple groups of dynamic images are encoded into multiple combined frames that are output continuously, wherein each combined frame includes multiple frame images generated by the camera component at the same time, thereby allowing the multi-module imaging system to synchronously output the multiple frame images generated by different photosensitive modules.

2. The multi-module imaging system according to claim 1, characterized in that: The image signal processor executes an image processing program on each group of dynamic images, and the image processing program includes an image optimization program that performs one or any combination of noise reduction, color correction, white balance adjustment, contrast adjustment, saturation adjustment and sharpness enhancement on each frame image in each group of dynamic images.

3. The multi-module imaging system according to claim 1, characterized in that: The plurality of combined frames are continuously generated by encoding through the data processor, and the plurality of combined frames are transmitted to an external system in a time sequence, and then the plurality of groups of synchronized continuous frame images are obtained through a decoding process.

4. The multi-module imaging system according to claim 3, characterized in that: The multi-module imaging system is disposed in a driving recorder, and the driving recorder includes a communication module for transmitting the plurality of combined frames to the external system in a wireless or wired manner.

5. The multi-module imaging system according to claim 1, characterized in that: The camera assembly includes an infrared light module and at least one visible light module. The combined frame includes an infrared light image generated by the infrared light module and at least one visible light image generated by the at least one visible light module.

6. The multi-module imaging system according to claim 5, characterized in that: The multi-module imaging system is applied to an automatic license plate recognition system, and a license plate image of a vehicle is obtained through the infrared light module, and an environment image including the vehicle is obtained through the visible light module.

7. The multi-module imaging system according to claim 6, characterized in that: The automatic license plate recognition system is used to recognize a plurality of characters of English and number combination in the license plate image.

8. The multi-module imaging system according to claim 7, characterized in that: The automatic license plate recognition system recognizes the license plate to combine the derived multiple characters into the combined frame.

9. The multi-module imaging system according to claim 5, characterized in that: The camera assembly adopts a global shutter to make all the photosensitive pixels of each photosensitive component in the infrared light module and the at least one visible light module sense light at the same time, thereby generating a frame of image sensed synchronously at high speed.

10. The multi-module imaging system according to any one of claims 1 to 9, characterized in that: The combined frame formed by multiple frames of images acquired at the same time is a spliced ​​frame formed by the data processor combining the multiple frames of images in a splicing manner.

11. The multi-module imaging system according to claim 10, characterized in that: The data processor splices the multiple frames of images acquired at the same time in a vertical encoding manner.

12. An image synchronization method, characterized in that: The image synchronization method comprises: Producing multiple sets of dynamic images through a camera assembly having multiple photosensitive modules; Acquire a set of continuous frame images from one set of dynamic images generated by each photosensitive module in the camera assembly, so that the multiple sets of dynamic images form multiple sets of continuous frame images; and The multiple frame images obtained at the same time in the multiple groups of continuous frame images are combined to form a combined frame, and the multiple groups of dynamic images are encoded into multiple combined frames that are output continuously, wherein each combined frame includes multiple frame images generated by the camera component at the same time, thereby synchronously outputting the multiple frame images generated by different photosensitive modules.

13. The image synchronization method according to claim 12, characterized in that: After obtaining the multiple groups of dynamic images, an image processing program is executed on the multiple groups of dynamic images. The image processing program includes an image optimization program that performs one or any combination of noise reduction, color correction, white balance adjustment, contrast adjustment, saturation adjustment and sharpness enhancement on each frame image in the multiple groups of dynamic images.

14. The image synchronization method according to claim 12, characterized in that: The image synchronization method is executed in a vehicle recorder, and the plurality of combined frames output continuously are transmitted to an external system in a time sequence, and the synchronized plurality of sets of continuous frame images are obtained in the external system through a decoding process.

15. The image synchronization method according to claim 12, characterized in that: The camera assembly includes an infrared light module and at least one visible light module. The combined frame includes an infrared light image generated by the infrared light module and at least one visible light image generated by the at least one visible light module.

16. The image synchronization method according to claim 15, characterized in that: The image synchronization method is executed in an automatic license plate recognition system, and a license plate image of a vehicle is obtained through the infrared light module, and an environment image including the vehicle is obtained through the visible light module.

17. The image synchronization method according to claim 16, characterized in that: The automatic license plate recognition system is used to recognize a plurality of characters of English and number combination in the license plate image.

18. The image synchronization method according to claim 17, characterized in that: The automatic license plate recognition system recognizes the license plate to combine the derived multiple characters into the combined frame.

19. The image synchronization method according to claim 15, characterized in that: The camera assembly adopts a global shutter to make all the photosensitive pixels of each photosensitive component in the infrared light module and the at least one visible light module sense light at the same time, thereby generating a frame of image sensed synchronously.

20. The image synchronization method according to any one of claims 12 to 19, characterized in that: Each of the combined frames is a spliced ​​frame formed by splicing the multiple frames of images obtained at the same time in a vertical coding manner.

Images