Real-time deblurring processing method for passenger image

Abstract

The invention belongs to the technical field of image processing, and particularly relates to a passenger image real-time deblurring processing method, which comprises the following steps: S1, an image acquisition module shoots a passenger through a camera module to obtain a picture of the passenger; the camera shooting module shoots a plurality of photos with different focus positions for the passenger; s2, the image processing module combines the shot pictures with different focus positions and performs image stacking to obtain a full-clear picture about the passenger; s3, the image calculation module carries out recognition analysis and deblurring processing on the full-clear image of the passenger, and a deblurred passenger image is obtained; the image analysis module analyzes the pictures before and after deblurring to obtain a picture acquisition optimization scheme; according to the method, multiple focuses are photographed for multiple times, the pictures are preprocessed and merged in sequence, the size of a blurred area on the passenger image is reduced, then the merged picture is deblurred, and therefore the effect and efficiency of image deblurring processing are improved.

Description

Technical Field

[0001] This invention belongs to the field of image processing technology, specifically a real-time deblurring method for occupant images. Background Technology

[0002] With the development of automotive intelligence, cameras are being used more and more in cars. According to the installation location, they can be divided into external cameras and internal cameras. External cameras are used for 360-degree panoramic imaging, electronic rearview mirrors, etc., while internal cameras are mainly used for algorithm recognition, such as face recognition, driver status monitoring, gesture recognition, etc. Many camera parameters affect the imaging effect, including exposure time, aperture size, focal length, dynamic range, gain, and signal-to-noise ratio. Since the realization of many functions in intelligent vehicles depends heavily on the quality of images captured by in-vehicle cameras, the reliability and stability of image quality have become important factors affecting system performance. However, when acquiring occupant images, the movement of the vehicle, the ambient light environment, and changes in the occupant's actions and postures can cause motion blur interference and blurring of the images. This blurring not only reduces the image's clarity but also significantly affects the accuracy of subsequent visual perception and processing tasks, thus impacting the performance of the vehicle system. Summary of the Invention

[0003] To overcome the shortcomings of existing technologies, this invention proposes a real-time deblurring method for occupant images. This method involves taking multiple photos at multiple focal points, preprocessing and merging the images sequentially to reduce the size of blurred areas in the occupant images, and then deblurring the merged images, thereby improving the effect and efficiency of image deblurring.

[0004] The technical solution adopted by this invention to solve its technical problem is: a real-time deblurring method for occupant images, comprising an image acquisition module, a camera module, an image processing module, an image calculation module, and an image analysis module, wherein the method includes: S1: The image acquisition module captures images of the occupants inside the vehicle using a camera module installed inside the vehicle; the in-vehicle camera module takes multiple photos of the occupants with different focus positions. S2: The image processing module merges photos taken at different focus points and stacks the images to obtain a clear image of the occupants inside the vehicle. S3: The image calculation module performs recognition analysis and deblurring on the full-clear image of the occupants inside the vehicle to obtain a deblurred image of the occupants inside the vehicle; the image analysis module analyzes the images before and after deblurring to obtain an image acquisition optimization scheme. In step S1, the camera module continuously takes multiple photos of the occupants in the vehicle at different focal positions, and the image processing module selects the photo with the best clarity from the multiple photos taken at each focal position and stacks the images. The camera module includes at least an infrared vision sensor, an event vision sensor, and a conventional vision sensor.

[0005] Preferably, the camera module includes a fill light, the light scattering degree of the fill light changes when the camera module takes different photos of the same focal position, the exposure time used when the camera module takes multiple photos of the same focal position is different, and the fill light operates according to the image acquisition optimization scheme; The camera module has an FPGA chip built in, which preprocesses the captured images. The preprocessing includes interpolation, flipping, gamma transformation, filtering, stitching, and image stacking operations.

[0006] Preferably, when the camera module takes photos of the occupants inside the vehicle, it first takes photos using an event vision sensor, and the image acquisition module analyzes the photos taken by the event vision sensor to determine the focal points of various parts of the occupants inside the vehicle at the time of shooting.

[0007] Preferably, the image analysis module analyzes the images collected by the event vision sensor to obtain dynamic information on the actions of the occupants inside the vehicle. When the image analysis module determines that an occupant is abnormal, it re-analyzes the deblurred image of that occupant to obtain the specific details of the abnormality and issues an alarm. The alarm information issued by the image analysis module is displayed on the vehicle's in-vehicle screen or through the horn to remind other occupants inside the vehicle.

[0008] Preferably, the image acquisition module acquires images of the seats inside the vehicle, the image processing module and the image calculation module perform deblurring processing on the images of the seats inside the vehicle, and the image analysis module analyzes the deblurred images of the seats and occupants inside the vehicle to identify information about items that have fallen onto or under the seats.

[0009] Preferably, the method further includes an image locking module, which locks the deblurred occupant image; The image acquisition module acquires images of the occupants both inside and outside the vehicle. The image calculation module deblurs the images of the occupants both inside and outside the vehicle. The image analysis module analyzes the deblurred images of the occupants both inside and outside the vehicle to identify items lost by the occupants inside the vehicle. When the image analysis module detects lost items inside the vehicle, it issues an alarm. The alarm issued by the image analysis module is displayed on the vehicle's in-vehicle screen or through the horn to alert other occupants inside the vehicle.

[0010] Preferably, the camera module includes a mounting base, on which a mounting cylinder is mounted, and a vision sensor is mounted on the mounting base. The vision sensor is located inside the mounting cylinder, and the vision sensor includes an infrared vision sensor, an event vision sensor, and a conventional vision sensor. The mounting cylinder is transparent to the end face of the vision sensor, and a supplementary light is installed inside the mounting cylinder on the end face of the vision sensor. An opaque partition is installed between the vision sensors.

[0011] Preferably, the supplementary light includes a lamp holder, and a mounting groove is provided on the side of the lamp holder facing outward from the mounting cylinder. An LED bead is mounted on the bottom surface of the mounting groove, and a light diffuser plate and a lens are mounted in the mounting groove. The light diffuser plate is located between the LED bead and the lens. An electric push rod is installed on the side of the lamp holder. The extended end of the electric push rod is inserted into the mounting groove. The extended end of the electric push rod is located between the lens and the light-diffusing plate. The extended end of the electric push rod is conical and contacts the transparent lower surface. The lens moves up and down in the mounting groove.

[0012] The beneficial effects of this invention are as follows: 1. The present invention discloses a real-time deblurring method for occupant images. This method involves setting up an image acquisition module, a camera module, an image processing module, an image calculation module, and an image analysis module to acquire multiple photos of the occupant with different focal points. These photos are then merged and stacked to obtain a fully clear image of the occupant. The fully clear image is then used for identification, analysis, and deblurring, resulting in fewer blurred areas in the occupant image and improving the effectiveness and efficiency of real-time deblurring. Furthermore, the deblurred occupant image is analyzed and identified, and alerts are issued when anomalies are detected, ensuring occupant safety and preventing occupants from leaving items on the vehicle.

[0013] 2. The real-time deblurring method for occupant images described in this invention, by setting up a camera module, vision sensors, LEDs, a light diffuser, a lens, an electric push rod, and an extension end, enables the camera module to acquire images of the occupants through multiple vision sensors. The various vision sensors perform mutual supervision and verification, improving the effect and efficiency of deblurring processing and avoiding errors caused by damage to the vision sensors. Simultaneously, during occupant image acquisition, the intensity of the supplementary lighting is adjusted as needed to ensure good image clarity, reducing the difficulty of subsequent deblurring processing. Furthermore, an FPGA chip is built into the camera module to preprocess the acquired images, improving the effect and efficiency of subsequent occupant image deblurring and reducing the computational requirements for deblurring processing. Attached Figure Description

[0014] The invention will now be further described with reference to the accompanying drawings.

[0015] Figure 1 This is a flowchart of the processing method of the present invention; Figure 2 This is a perspective view of the camera module in this invention; Figure 3 This is a schematic diagram of the structure inside the mounting cylinder of the camera module in this invention; Figure 4 This is a schematic diagram of the camera module in this invention; Figure 5 yes Figure 4 Enlarged view of a portion of point A in the middle; In the diagram: Mounting base 1, mounting cylinder 2, partition 3, lamp holder 4, electric push rod 41, extension end 411, mounting groove 42, lamp bead 43, light diffuser 44, lens 45, vision sensor 5. Detailed Implementation

[0016] To make the technical means, creative features, objectives and effects of this invention easier to understand, the invention will be further described below in conjunction with specific embodiments.

[0017] like Figures 1 to 5 As shown, the real-time deblurring method for occupant images according to the present invention includes an image acquisition module, a camera module, an image processing module, an image calculation module, and an image analysis module. The method includes: S1: The image acquisition module captures images of the occupants inside the vehicle using a camera module installed inside the vehicle; the in-vehicle camera module takes multiple photos of the occupants with different focus positions. S2: The image processing module merges photos taken at different focus points and stacks the images to obtain a clear image of the occupants inside the vehicle. S3: The image calculation module performs recognition analysis and deblurring on the full-clear image of the occupants inside the vehicle to obtain a deblurred image of the occupants inside the vehicle; the image analysis module analyzes the images before and after deblurring to obtain an image acquisition optimization scheme. In step S1, the camera module continuously takes multiple photos of the occupants in the vehicle at different focal positions, and the image processing module selects the photo with the best clarity from the multiple photos taken at each focal position and stacks the images. The camera module includes at least an infrared vision sensor, an event vision sensor, and a conventional vision sensor. When collecting, recognizing, and analyzing occupant images, the image acquisition module sends instructions to the camera modules installed throughout the vehicle, enabling them to capture images of the occupants. However, due to the influence of vehicle movement, occupant posture and movements, and ambient lighting, the occupant images captured by the image acquisition module through the vehicle's camera modules may be blurry. This can affect subsequent recognition and analysis of the occupant images, such as facial recognition, status monitoring, and gesture recognition, impacting the normal operation and accuracy of various related functions within the vehicle. Meanwhile, after receiving the instruction, the camera module will take multiple photos of the occupant, and the focus position of each photo is different. That is, the depth of field of each photo is different, and the position of the blurred area caused by the depth of field is also different. Therefore, the image processing module stitches the occupant photos with different focus positions taken by the camera module and performs depth of field merging to obtain a clear photo of the occupant. This reduces the size of the blurred area that may exist in the collected occupant photos, improves the clarity of the occupant image, and ensures that the functions of occupant image recognition and analysis on the vehicle operate normally and accurately. Meanwhile, even after obtaining a fully clear photo of the occupants, blurry areas may still exist due to the vehicle's motion, occupant posture and movements, and lighting conditions. Therefore, the image processing module performs real-time recognition analysis and deblurring on the fully clear occupant image. When deblurring the image, existing filtering algorithms such as mean, median, Gaussian, bilateral, and guided filters are used, and the appropriate algorithm is selected based on the actual situation. Furthermore, further deblurring based on the fully clear photo reduces computation time, improves the deblurring effect and efficiency, and enables real-time deblurring of occupant images, ensuring the normal operation of various functions on the vehicle that rely on occupant images. Meanwhile, after deblurring the occupant image to obtain a clear image, the image analysis module analyzes the images before and after deblurring to determine the cause of the blurring and provide optimization solutions. This allows the image acquisition module to control the camera module to make corresponding adjustments when acquiring occupant images again, such as adjusting the focus position and supplementing lighting. This ensures that the external environment during occupant image capture is relatively good, reducing the size of the blurred area in the acquired occupant image. This facilitates subsequent deblurring calculations and improves the effectiveness and efficiency of deblurring. Meanwhile, since the camera module takes multiple photos of the occupant image at different focal points, the image processing module compares the multiple photos using a gradient-based method in the existing technology. The module then selects the photo with the best clarity from the multiple photos as the image representing the focal point. The images at each focal point are then merged and stacked to obtain a fully clear image of the occupant. This avoids the situation where the camera module only takes one photo of the occupant to obtain an image, which is prone to random errors or interference, affecting the difficulty of deblurring the occupant image and the effect and efficiency of the deblurring process. Meanwhile, since the camera module includes multiple vision sensors, such as infrared vision sensors, event vision sensors, and conventional vision sensors, after the image acquisition module issues a command, each vision sensor in the camera module will sequentially capture images of the occupants. Then, the captured occupant images are merged, stacked, and deblurred to obtain deblurred occupant images. This allows for mutual supervision and verification among the various vision sensors, preventing damage or errors in a single sensor from affecting the effectiveness and accuracy of subsequent occupant image deblurring.

[0018] In one embodiment of the present invention, the camera module includes a fill light, the light scattering degree of the fill light changes when the camera module takes different photos of the same focal position, the exposure time used when the camera module takes multiple photos of the same focal position is different, and the fill light operates according to the image acquisition optimization scheme. The camera module has an FPGA chip built in, which preprocesses the captured images. The preprocessing includes interpolation, flipping, gamma change, filtering, stitching, and image stacking operations. Since the camera module includes a supplementary light, when the camera module is capturing occupant images, the supplementary light will be turned on to supplement the ambient light, improving the quality of the captured occupant images and reducing the size of blurred areas in the images. At the same time, the image analysis module will analyze the captured and deblurred occupant images to identify the factors affecting the occupant images. Then, it will provide an image acquisition optimization plan, so that the supplementary light in the image acquisition module and the camera module can adjust and work according to the image acquisition optimization plan, improving the clarity of the captured occupant images and reducing the blurred areas in the images, thereby improving the effect and efficiency of subsequent real-time deblurring of occupant images. Meanwhile, when the image acquisition module captures images of the occupants through the camera module, and multiple photos are taken at the same focal point, the exposure time for each photo is different. For example, if three images are taken consecutively, namely the first image, the second image, and the third image, and the exposure time for each image is the first time, the second time, and the third time respectively, the FPGA chip built into the camera module will preprocess the first image, the second image, and the third image. Based on the gradient-based method, the best and clearest image, such as the second image, is selected. Then, the camera module uses the second image as one of the images at one of the focal points to synthesize it with the images taken at other focal points to obtain a fully clear image.

[0019] As one embodiment of the present invention, when the camera module takes photos of the occupants inside the vehicle, it first takes photos through an event vision sensor, and the image acquisition module analyzes the photos taken by the event vision sensor to determine the focal positions of various parts of the occupants inside the vehicle at the time of shooting. The camera module first captures images of the occupants using an event vision sensor, and then determines the focal points of various parts of the occupants based on these images. Because of the characteristics of the event vision sensor, it does not record static background information of the entire scene, but only dynamic information. This allows it to mark the focal points of the occupants where changes in position or posture occur. Then, when the image acquisition module captures images of the marked focal points, it controls the camera module to use a shorter exposure time to capture these images. This improves the clarity and quality of the captured images, reduces the size of blurred areas in the captured occupant images, and thus improves the effectiveness and efficiency of subsequent real-time deblurring processing. Meanwhile, for other occupants inside the vehicle whose position or posture has not changed, the image acquisition module will control the camera module to use a relatively long exposure time to take pictures, thereby improving the clarity and effect of the acquired occupant images. Meanwhile, by selecting focal points on various parts of the occupant's body, it is easier to obtain a fully clear image after subsequent image merging, reducing the size of the blurred areas in the fully clear image, reducing the difficulty and time required for subsequent deblurring processing, and improving the effect and efficiency of deblurring processing.

[0020] In one embodiment of the present invention, the image analysis module analyzes the images collected by the event visual sensor to obtain dynamic information on the actions of the occupants in the vehicle. When the image analysis module determines that an occupant is abnormal, it analyzes the deblurred image of the occupant again to obtain the specific circumstances of the abnormality and issues an alarm. The alarm information issued by the image analysis module is displayed on the vehicle's screen or through the horn to remind other occupants in the vehicle. By analyzing the dynamic information about occupants collected by the event vision sensor, when the image analysis module determines that there is an abnormality in the occupant, the infrared vision sensor or conventional vision sensor continues to collect images of the occupant. After deblurring the collected occupant images, the specific circumstances of the abnormality are analyzed, and an alarm is triggered based on the specific circumstances of the abnormality. This alerts other occupants in the vehicle through the display screen or speaker, allowing them to check and handle the abnormal occupant. For example, if a child in the vehicle makes a dangerous move without the adult's awareness, the adult can receive a timely alert through the display screen or speaker, thus stopping the child's dangerous behavior. Or, if a passenger in a ride-hailing vehicle suddenly feels unwell or prepares to attack the driver, the driver can receive a timely alert, thus handling the abnormality of the passenger and ensuring the safety of the passenger or driver. Simultaneously, when a passenger boards the vehicle, if the image analysis module determines that the passenger is experiencing an abnormality and requires medical attention (e.g., the passenger is unwell and taking a ride-hailing service to the hospital), the image analysis module will automatically send an alarm message to the vehicle's onboard emergency rescue module. This will connect the onboard emergency rescue module to the emergency center or hospital, sending data such as the vehicle's location and the passenger's condition. This allows emergency center or hospital staff to prepare for the passenger's rescue or provide temporary guidance to other accompanying passengers, ensuring that the passenger in distress can be maintained in good health before reaching the emergency center or hospital. The condition remains stable and does not deteriorate. At the same time, by deblurring the collected occupant images, the images of occupants with abnormal conditions are clear enough to help emergency center or hospital staff more accurately assess the occupant's condition. Meanwhile, when the vehicle's emergency rescue module connects to the emergency center, it will simultaneously send a record information to the traffic management center so that the traffic management center can implement corresponding traffic control measures based on the severity of the occupant's condition, such as dispatching traffic police motorcycles to clear the way or controlling traffic lights at intersections to facilitate passage, ensuring smooth vehicle passage and rapid arrival at the emergency center or hospital for treatment of occupants with abnormal conditions. Meanwhile, during the journey, when the image analysis module analyzes the deblurred occupant image and determines that the occupant's condition is abnormal, the image analysis module will alert other occupants through the vehicle's display screen or speaker. This allows other occupants to confirm the abnormal occupant's condition and avoids misjudgment by the image analysis module. If it is confirmed that the occupant has indeed suddenly become unwell, other occupants will confirm the image analysis module's alert. At this time, the image analysis module will automatically send an alarm message to the vehicle's emergency rescue module, sending relevant information to the emergency center and traffic management center. This allows emergency center staff to provide remote guidance to other occupants, prevent the occupant's condition from worsening, and facilitates traffic management center traffic guidance. After the alarm message from the image analysis module is confirmed, the module further identifies and analyzes the deblurred occupant image, comparing and analyzing the images to determine changes in the occupant's condition. The analysis results are then sent to emergency center or hospital staff, enabling them to promptly guide other occupants in appropriate actions and ensure the occupant's condition remains stable. Meanwhile, the image analysis module identifies and analyzes the clear occupant images after deblurring. The image analysis module compares the clear occupant images with publicly available information on wanted fugitives online. When the image analysis module detects that an occupant is a wanted fugitive, it issues an alarm to alert the driver so that the driver can call the police or take the occupant directly to the relevant authorities. At the same time, the alarm information issued by the image analysis module is displayed on the vehicle's display screen, HUD, or the driver's mobile phone, and the alarm information is a specific voice or symbol to prevent the occupant from being discovered and causing danger to the driver or other occupants. Meanwhile, since occupant images are prioritized for acquisition and analysis via event vision sensors, when the image analysis module determines that an anomaly exists among the occupants—such as when an occupant is making gestures for gesture recognition or when the occupant's condition is abnormal—the image acquisition module will acquire occupant images again via infrared vision sensors or conventional vision sensors. These images will then be deblurred and accurately analyzed to determine the specific nature of the anomaly. During this process, if no anomalies are detected, the image acquisition module will stop acquiring occupant images via infrared vision sensors or conventional vision sensors, or reduce the frequency of acquisition. This reduces the difficulty of real-time deblurring of occupant images, lowers the computational requirements for deblurring, and improves the effectiveness and efficiency of real-time deblurring.

[0021] In one embodiment of the present invention, the image acquisition module acquires images of the seats inside the vehicle, the image processing module and the image calculation module perform deblurring processing on the images of the seats inside the vehicle, and the image analysis module analyzes the deblurred images of the seats inside the vehicle and the occupants to identify information about items that have fallen on or under the seats. The image acquisition module captures images of the seats inside the vehicle and deblurs them to obtain clear images. Then, the image analysis module analyzes the deblurred, clear images of the seats to identify items inside the vehicle and determine their location. When passengers are searching for items inside the vehicle, the image analysis module can provide information on the distribution of items to remind them and prevent items from falling off the seats or getting stuck in the gaps between the seats and becoming difficult to find. Meanwhile, by analyzing items on the seats and items carried by passengers through the image analysis module, the system can alert passengers when items accidentally fall onto or under the seats. For example, if a passenger sits down and their phone falls from their pocket onto the seat without their knowledge, making it difficult to find when needed, the system can alert the passenger to the location of the fallen phone.

[0022] In one embodiment of the present invention, the method further includes an image locking module, which locks the deblurred occupant image; The image acquisition module acquires images of the occupants both inside and outside the vehicle. The image calculation module deblurs the images of the occupants both inside and outside the vehicle. The image analysis module analyzes the deblurred images of the occupants both inside and outside the vehicle to identify items lost by the occupants inside the vehicle. When the image analysis module detects that items have been lost by the occupants inside the vehicle, it issues an alarm. The alarm issued by the image analysis module is displayed on the vehicle's in-vehicle screen or through the horn to alert other occupants inside the vehicle. Because the vehicle is equipped with cameras both inside and outside, the image acquisition module captures images of occupants when they approach and board or alight from the vehicle, and the image acquisition module captures images of occupants inside the vehicle after boarding. Then, the image processing module deblurs the captured images of occupants inside and outside the vehicle, and the image analysis module identifies and analyzes the occupant images. This allows for comparison of the types of items occupants are carrying inside the vehicle with the types of items they are carrying outside when boarding, and the types of items they are carrying outside when alighting, thus preventing the spread of unwanted items inside and outside the vehicle. The types of items carried may change. For example, passengers may place their items on the ground while waiting for the bus, but forget to bring them when boarding, resulting in the loss of their items. Or, passengers may place their items inside the bus after boarding, but forget to bring them when getting off, resulting in the loss of their items. Therefore, the image analysis module will issue an alarm when passengers are carrying different items inside and outside the bus, so as to promptly remind the driver and passengers, avoid passengers forgetting to bring their items and losing them, or prevent some drivers from deliberately ignoring items left behind by passengers and failing to remind passengers when they get off, thus causing the loss of their items. Meanwhile, when the image analysis module issues an alarm for items lost by a passenger in the vehicle, the image locking module locks the corresponding passenger image upon receiving the alarm. If the passenger receives the alarm and takes the lost items with them, the image analysis module, after analyzing the passenger image and determining that the items have been taken, deactivates the alarm. At this time, the image locking module simultaneously unlocks the passenger image, allowing it to be deleted or overwritten. If a passenger leaves the vehicle without noticing the alarm from the image analysis module, the module will re-analyze the images inside the vehicle and after the passenger has left. If the module determines that the passenger's belongings are missing and the passenger has left, it will issue another alarm to alert other passengers. The driver should collect and store any lost items to prevent damage or theft by subsequent passengers, and report the incident to the transportation management agency or relevant departments. Upon receiving the alarm again, the image locking module will not unlock the passenger's image to prevent the driver from manually deleting it, thus eliminating evidence of lost items or unintentionally deleting or overwriting the image. Furthermore, the image locking module will automatically upload and back up the locked passenger image to the transportation management agency or relevant departments to further prevent its deletion or overwriting.

[0023] In one embodiment of the present invention, the camera module includes a mounting base 1, a mounting cylinder 2 is mounted on the mounting base 1, a vision sensor 5 is mounted on the mounting base 1, the vision sensor 5 is located inside the mounting cylinder 2, and the vision sensor 5 includes an infrared vision sensor, an event vision sensor, and a conventional vision sensor. The mounting cylinder 2 is transparent to the end face of the vision sensor 5. A supplementary light is installed inside the mounting cylinder 2 on the end face of the vision sensor 5. An opaque partition 3 is installed between the vision sensors 5. The mounting cylinder 2 installed on the mounting base 1 makes the camera module relatively enclosed, avoiding adverse effects of the external environment on the internal vision sensor 5. At the same time, the end face of the mounting cylinder 2 is made of transparent material or is not painted, so that the vision sensor 5 inside the camera module can take pictures of the outside world normally. Meanwhile, when the image acquisition module acquires occupant images through the camera module, each vision sensor 5 in the camera module takes pictures and acquires occupant images. During this process, the supplementary light in the camera module works to supplement the light during the shooting process of the vision sensor 5, so as to avoid the light in the external environment being too weak, which would affect the clarity of the acquired occupant images and make the subsequent occupant image deblurring process less effective and efficient. Meanwhile, by installing a partition 3 inside the mounting cylinder 2, the multiple visual sensors 5 installed in the camera module are isolated from each other by using the opaque partition 3, so as to avoid the light supplemented by the supplementary light during shooting from interfering with the shooting process of other visual sensors 5, causing the occupant images captured by other visual sensors 5 to be significantly blurred, which would affect the effect and efficiency of subsequent deblurring processing.

[0024] In one embodiment of the present invention, the supplementary light includes a lamp holder 4, and a mounting groove 42 is provided on the side of the lamp holder 4 facing the outside of the mounting cylinder 2. A lamp bead 43 is installed on the bottom surface of the mounting groove 42. A light-diffusing plate 44 and a lens 45 are installed in the mounting groove 42, and the light-diffusing plate 44 is located between the lamp bead 43 and the lens 45. An electric push rod 41 is installed on the side of the lamp holder 4. The extended end 411 of the electric push rod 41 is inserted into the mounting groove 42. The extended end 411 of the electric push rod 41 is located between the lens 45 and the light-diffusing plate 44. The extended end 411 of the electric push rod 41 is conical and contacts the transparent lower surface. The lens 45 moves up and down in the mounting groove 42. By evenly installing multiple LED beads 43 in the mounting slot 42 and installing a light-diffusing plate 44 above the LED beads 43, the light emitted by the LED beads 43 is evenly diffused through the light-diffusing plate 44, avoiding uneven light distribution from the supplementary light lamp, which would affect the effect of the occupant image captured by the camera module and the subsequent image deblurring effect and efficiency. At the same time, since the extended end 411 of the electric push rod 41 on the side of the lamp holder 4 is inserted into the mounting slot 42, and the extended end 411 is tapered, when the electric push rod 41 drives the extended end 411 into the mounting slot 42... This causes the side of the extended end 411 to press against the lens 45 in the mounting groove 42, causing the lens 45 to move out of the mounting groove 42. That is, the distance between the lens 45 and the light-diffusing plate 44 and the lamp bead 43 in the mounting groove 42 increases, making the light emitted by the supplementary light relatively dispersed and increasing the illumination area, thus reducing the intensity of the supplementary light. This allows the camera module to adjust the intensity of the supplementary light during shooting by controlling the action of the electric push rod 41, avoiding the light being too strong or too weak, ensuring that the blurred area on the captured occupant image is small, which facilitates the effect and efficiency of subsequent deblurring processing.

[0025] The specific workflow is as follows: The image acquisition module sends commands to the camera modules installed in various parts of the vehicle, which then capture images of the occupants. At the same time, after receiving the instruction, the camera module will take multiple photos of the occupant, and the focus position of each photo is different, that is, the depth of field corresponding to each photo is different, and the position of the blurred area caused by the depth of field is also different. Therefore, the image processing module stitches the occupant photos with different focus positions taken by the camera module and performs depth of field merging to obtain a clear photo of the occupant. Meanwhile, the image computing module identifies and analyzes the clear images of the occupants and performs deblurring. When deblurring the images, existing filtering algorithms such as mean, median, Gaussian, bilateral, and guided are used, and the appropriate algorithm is selected based on the actual situation. Meanwhile, after deblurring the occupant image to obtain a clear image, the image analysis module will analyze the images before and after deblurring to determine the cause of the blurring and provide optimization solutions. Meanwhile, the camera module continuously takes multiple photos of the occupant at different focal points. The image processing module then compares these multiple photos using gradient-based methods in existing technologies, and selects the photo with the best clarity from among them as the image representing the focal point. Meanwhile, after the image acquisition module issues a command, the various visual sensors in the camera module will sequentially capture images of the occupants. Then, the captured occupant images will be merged, stacked, and deblurred to obtain deblurred occupant images, so that the various visual sensors can be used for mutual supervision and verification. When the camera module captures images of occupants, the supplementary light inside the camera module will be turned on to supplement the ambient light, improve the quality of the captured images of occupants, and reduce the size of blurry areas in the images; Meanwhile, when the image acquisition module captures images of the occupants through the camera module, and multiple photos are taken at the same focal point, the exposure time for each photo is different. For example, if three photos are taken consecutively, namely the first photo, the second photo, and the third photo, and the exposure time for each photo is the first time, the second time, and the third time respectively, the FPGA chip built into the camera module will preprocess the first photo, the second photo, and the third photo. Based on the gradient-based method, the best and clearest photo, such as the second photo, is found. Then, the camera module uses the second photo as one of the focal points and combines it with the photos taken at other focal points to obtain a fully clear image. The camera module first captures images of the occupants using the event vision sensor, then determines the focal points of various parts of the occupants based on the images captured by the event vision sensor. Based on the characteristics of the event vision sensor, the focal points of various parts of the occupants that have changed position or posture are marked. Then, when the image acquisition module captures images of the marked focal points, the image acquisition module controls the camera module to use a shorter exposure time to capture images of the marked focal points. Meanwhile, for other occupants inside the vehicle whose position or posture has not changed, the image acquisition module will control the camera module to use a relatively long exposure time for shooting; By analyzing the dynamic information about the occupants collected by the event vision sensor, when the image analysis module determines that there is an abnormality in the occupants, the infrared vision sensor or conventional vision sensor continues to collect occupant images. After deblurring the collected occupant images, the specific circumstances of the abnormality in the occupants are analyzed, and an alarm is triggered based on the specific circumstances of the abnormality. This alerts other occupants in the vehicle to receive the alert through the vehicle's display screen or speaker, allowing them to check and handle the abnormal occupant. Meanwhile, since the event vision sensor prioritizes the acquisition and analysis of occupant images, when the image analysis module determines that there is an anomaly among the occupants, the image acquisition module will acquire occupant images again using the infrared vision sensor or conventional vision sensor. After deblurring the occupant images, the module will perform accurate analysis to determine the specific circumstances of the anomaly. If no anomaly is found, the image acquisition module will stop acquiring occupant images using the infrared vision sensor or conventional vision sensor or reduce the frequency of acquiring occupant images using the infrared vision sensor or conventional vision sensor, thereby reducing the difficulty of real-time deblurring processing and reducing computing power requirements. The image acquisition module captures images of the seats inside the vehicle and deblurs them to obtain clear images. Then, the image analysis module analyzes the deblurred and clear images of the seats to identify items in the vehicle and determine their location. When passengers are searching for items in the vehicle, the image analysis module can provide information on the distribution of items to remind them. Meanwhile, by analyzing the items on the seats inside the vehicle and the items carried by the passengers through the image analysis module, the system can alert passengers when items accidentally fall onto or under the seats. When a passenger approaches and boards or disembarks from the vehicle, the image acquisition module captures images of the passenger outside the vehicle and inside the vehicle after boarding. Then, the image calculation module deblurs the captured images of the passenger outside and inside the vehicle, and the image analysis module identifies and analyzes the passenger images. This allows for a comparison of the types of items the passenger is carrying inside the vehicle with the types of items they are carrying outside the vehicle when boarding and when disembarking. If the image analysis module detects discrepancies between the items the passenger is carrying inside and outside the vehicle, it will issue an alarm to promptly alert the driver and passengers. The camera module is relatively enclosed by the mounting cylinder 2 installed on the mounting base 1. At the same time, the end face of the mounting cylinder 2 is made of transparent material or is not painted. Meanwhile, when capturing images of the occupants, each visual sensor 5 in the camera module takes pictures and captures images of the occupants. During this process, the supplementary light in the camera module is activated to supplement the light during the shooting process of the visual sensors 5. Meanwhile, by installing a partition 3 inside the mounting cylinder 2, the multiple visual sensors 5 installed in the camera module are isolated from each other by using the opaque partition 3, so as to avoid the light supplemented by the supplementary light during shooting from interfering with the shooting process of other visual sensors 5. By evenly installing multiple LED beads 43 in the mounting slot 42 and installing a light diffuser plate 44 above the LED beads 43, the light emitted by the LED beads 43 is evenly diffused through the light diffuser plate 44. At the same time, when the electric push rod 41 drives the extension end 411 to be inserted into the mounting slot 42, the side of the extension end 411 will squeeze the lens 45 in the mounting slot 42, causing the lens 45 to move out of the mounting slot 42. That is, the distance between the lens 45 and the light diffuser plate 44 and LED beads 43 in the mounting slot 42 increases, making the light emitted by the supplementary light relatively dispersed, increasing the illumination area, and reducing the intensity of the supplementary light. This allows the camera module to adjust the intensity of the supplementary light during shooting by controlling the movement of the electric push rod 41.

[0026] The foregoing has shown and described the basic principles, main features, and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited to the above embodiments. The embodiments and descriptions in the specification are merely illustrative of the principles of the invention. Various changes and modifications can be made to the invention without departing from its spirit and scope, and all such changes and modifications fall within the scope of the present invention as claimed. The scope of protection of the present invention is defined by the appended claims and their equivalents.

Claims

1. A method for real-time deblurring of an occupant image, characterized in that: The method comprises an image acquisition module, a camera module, an image processing module, an image calculation module and an image analysis module, and the method comprises the following steps: S1: The image acquisition module captures the pictures of the passengers in the vehicle through the camera module installed in the vehicle; the camera module in the vehicle captures multiple photos of the passengers in the vehicle at different focal point positions; S2: The image processing module combines the photos captured at different focal point positions and performs image stacking to obtain a clear picture of the passengers in the vehicle; S3: The image calculation module identifies and analyzes the clear picture of the passengers in the vehicle and performs deblurring processing to obtain a deblurred passenger image; the image analysis module analyzes the pictures before and after deblurring to obtain a picture acquisition optimization scheme; In the S1 step, the camera module continuously captures multiple photos of the passengers in the vehicle at different focal point positions, and the image processing module selects the photo with the best clarity from the multiple photos captured at different focal point positions for image stacking; The camera module comprises at least an infrared visual sensor, an event visual sensor and a conventional visual sensor.

2. The method of claim 1, wherein: The camera module comprises a fill light, the light scattering degree of the fill light changes when the camera module captures different photos at the same focal point position, the exposure time used by the camera module when capturing multiple photos at the same focal point position is different, and the fill light acts according to the picture acquisition optimization scheme; The FGPA chip is installed in the camera module, the FGPA chip pre-processes the captured pictures, and the pre-processing includes interpolation, flipping, gamma change, filtering, splicing and image stacking.

3. The method of claim 2, wherein: When the camera module captures the photos of the passengers in the vehicle, the event visual sensor is used to capture the photos first, and the image acquisition module analyzes the photos captured by the event visual sensor to determine the focal point positions of each part of the passengers in the vehicle during the capturing.

4. The method of claim 2, wherein: The image analysis module analyzes the pictures collected by the event visual sensor to obtain dynamic information of the actions of the passengers in the vehicle, the image analysis module analyzes the deblurred image of the passenger again when the passenger is abnormal to obtain specific information of the abnormality and issue an alarm, and the alarm information issued by the image analysis module is used by the display screen or the loudspeaker in the vehicle to remind other passengers in the vehicle.

5. The method of claim 2, wherein: The image acquisition module collects the pictures of the seats in the vehicle, the image processing module and the image calculation module perform deblurring processing on the images of the seats in the vehicle, and the image analysis module analyzes the deblurred images of the passengers and the seats in the vehicle to identify the information of the objects on or under the seats.

6. The method of claim 4, wherein: The method further comprises an image locking module, which locks the passenger image after deblurring processing. The image acquisition module acquires images of the passenger when inside and outside the vehicle respectively, the image calculation module deblurs the images of the passenger when inside and outside the vehicle, the image analysis module analyzes the deblurred images of the passenger when inside and outside the vehicle to obtain the lost items of the passenger inside the vehicle, the image analysis module issues an alarm when the lost items of the passenger are found inside the vehicle, and the alarm issued by the image analysis module reminds other passengers inside the vehicle through the display screen or the loudspeaker in the vehicle.

7. The method of claim 1, wherein: The camera module comprises a mounting seat (1), a mounting cylinder (2) is mounted on the mounting seat (1), a visual sensor (5) is mounted on the mounting seat (1), the visual sensor (5) is located in the mounting cylinder (2), and the visual sensor (5) comprises an infrared visual sensor, an event visual sensor and a conventional visual sensor. The end face of the mounting cylinder (2) opposite to the visual sensor (5) is transparent, a light supplement lamp is mounted on the end face of the visual sensor (5) in the mounting cylinder (2), and an opaque partition plate (3) is mounted between the visual sensors (5).

8. The method of claim 7, wherein: The light supplement lamp comprises a lamp holder (4), a mounting groove (42) is formed in the side face of the lamp holder (4) facing the outside of the mounting cylinder (2), a lamp bead (43) is mounted on the bottom face in the mounting groove (42), a light uniformizing plate (44) and a lens (45) are mounted in the mounting groove (42), and the light uniformizing plate (44) is located between the lamp bead (43) and the lens (45). An electric push rod (41) is mounted on the side face of the lamp holder (4), the elongated end (411) of the electric push rod (41) is inserted into the mounting groove (42), the elongated end (411) of the electric push rod (41) is located between the lens (45) and the light uniformizing plate (44), the elongated end (411) of the electric push rod (41) is tapered and contacts the transparent lower surface, and the lens (45) moves up and down in the mounting groove (42).

Images