Image processing method, device, apparatus, and computer-readable storage medium

Abstract

The present invention discloses an image processing method, apparatus, device, and computer-readable storage medium, relating to the field of image processing technology. The image processing method comprises: obtaining an original scene image and pixel movement information of the original scene image; determining a target pixel in the original scene image and a movement trajectory of the target pixel based on the pixel movement information; restoring the target pixel based on the movement trajectory to obtain a target scene image corresponding to the original scene image. The present invention solves the technical problem that images are prone to ghosting when the camera has a long exposure time, is shaken, or when external scenes move, thereby improving the clarity of the output image and ensuring the user's MR experience.

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and in particular to an image processing method, apparatus, device and computer-readable storage medium. Background Art

[0002] With the explosive growth of the VR (Virtual Reality) and AR (Augmented Reality) industries in the past two years, their value in industry, medical care, entertainment, office, and social interaction has gradually been revealed. Currently, major manufacturers are also exploring and deploying in the MR (Mixed Reality) field.

[0003] The current primary technology for implementing MR is based on VST (video see-through) technology. VST uses a camera 10 to capture a real-time view of the real world, then combines this real-time view with a virtual view of the digital world, presenting the combined image on a display screen to the user. VST technology allows for complete control over visual integration, allowing for complete occlusion between virtual and real objects, and even higher-level modifications to real objects.

[0004] However, cameras that use VST technology to capture real-world scenes typically use a rolling shutter. Therefore, images are prone to ghosting in situations such as long exposure time, jitter, or motion of external scenery, resulting in poor output image quality and affecting the user's MR experience. Summary of the Invention

[0005] The main purpose of the present invention is to provide an image processing method, device, equipment and computer-readable storage medium, aiming to solve the technical problem that images are prone to ghosting when the camera has a long exposure time, is shaken or the external scene moves.

[0006] To achieve the above objectives, in a first aspect, the present invention provides an image processing method, comprising:

[0007] Acquiring an original scene image and pixel movement information of the original scene image;

[0008] Determining a target pixel point in the original scene image and a movement trajectory of the target pixel point according to the pixel point movement information;

[0009] The target pixel points are restored according to the movement trajectory to obtain a target scene image corresponding to the original scene image.

[0010] Based on the above technical solution, an original scene image and pixel movement information of the original scene image are obtained; based on the pixel movement information, a target pixel in the original scene image and the movement trajectory of the target pixel are determined; and based on the movement trajectory, the target pixel is restored to obtain a target scene image corresponding to the original scene image. Thus, this embodiment restores the target pixel using its movement trajectory, thereby eliminating the ghosting in the original scene image caused by long exposure time, jitter, or external scene movement, and obtaining the target scene image corresponding to the original scene image. This solves the technical problem that images are prone to ghosting due to long exposure time, jitter, or external scene movement, improves the clarity of the output image, and ensures the user's MR experience.

[0011] According to the first aspect, the step of restoring the target pixel points according to the movement trajectory to obtain the target scene image corresponding to the original scene image includes:

[0012] Determining a starting position of the target pixel point within an exposure period of the original scene image according to the movement trajectory;

[0013] Obtaining a fill pixel at the current position of the target pixel;

[0014] The target pixel point is moved from the current position to the starting position, and the filling pixel point is filled to the current position to obtain a target scene image corresponding to the original scene image.

[0015] Based on the above technical solution, the starting position of the target pixel point within the exposure period of the original scene image is determined according to the movement trajectory, and the filling pixel point at the current position of the target pixel point is obtained. Therefore, on the one hand, the target pixel point is moved from the current position to the starting position so that all target pixels are restored to the position at the beginning of exposure (i.e., the initial position), and then the filling pixel point is filled into the current position to fill the vacancy left at the current position after the target pixel point is moved from the current position to the starting position, thereby obtaining a target scene image corresponding to the original scene image.

[0016] According to the first aspect, or any implementation of the first aspect above, the step of obtaining a filling pixel point at the current position of the target pixel point includes:

[0017] Acquire a previous frame of target scene image, and inter-frame pixel movement information between the previous frame of target scene image and the original scene image;

[0018] According to the previous frame target scene image and the inter-frame pixel movement information, the initial pixel point of the current position within the exposure period of the original scene image is determined, and the initial pixel point is used as the filling pixel point of the current position.

[0019] Based on the above technical solution, the accuracy of the pixels used to fill the gap at the current position is improved by using the previous target scene image and the inter-frame pixel movement information between the previous target scene image and the original scene image, thereby ensuring the accuracy of the output target scene image. In addition, the previous target scene image and inter-frame pixel movement information can also effectively reduce the amount of image processing data, avoiding the need to update the initial EVS image in real time, thereby improving image processing efficiency.

[0020] According to the first aspect, or any implementation of the first aspect above, the step of determining the target pixel point in the original scene image and the movement trajectory of the target pixel point based on the pixel point movement information includes:

[0021] Determining, according to the pixel point movement information, moving pixels in the original scene image that have moved during an exposure period, and using the moving pixels as target pixels;

[0022] Determining a starting position of the target pixel within an exposure period according to the pixel movement information;

[0023] A moving trajectory of the target pixel is generated according to the starting position and the current position of the target pixel.

[0024] Based on the above technical solution, the moving pixels in the original scene image that have moved during the exposure period are determined based on the pixel movement information, and the moving pixels are used as target pixels. Furthermore, the starting position of the target pixel within the exposure period is determined based on the pixel movement information; thus, a movement trajectory of the target pixel is generated based on the starting position and the current position of the target pixel. This embodiment determines the target pixel that has moved in the original scene image during the exposure period of the standard camera capture and the movement trajectory of the target pixel based on the pixel movement information, so that the position of the target pixel can be subsequently restored based on the movement trajectory.

[0025] According to the first aspect, or any implementation of the first aspect above, before the step of obtaining the original scene image, the method includes:

[0026] The target scene is photographed by a standard camera to obtain an original scene image corresponding to the target scene.

[0027] Based on the above technical solution, the target scene is photographed by a standard camera to obtain an original scene image that may have smear.

[0028] According to the first aspect, or any implementation of the first aspect above, before the step of obtaining pixel movement information of the original scene image, the method includes:

[0029] During the exposure period of the original scene image, the pixel movement information of the original scene image is collected by an event-driven camera.

[0030] Based on the above technical solution, during the exposure period of the original scene image, an event-driven camera is used to collect pixel movement information of the original scene image to determine the target pixel points in the original scene image that have moved during the exposure period and the movement trajectory of the target pixel points, which are used to restore the position of the target pixel points.

[0031] According to the first aspect, or any implementation of the first aspect above, before the step of determining the target pixel point and the movement trajectory of the target pixel point in the original scene image based on the pixel point movement information, the image processing method further includes:

[0032] When the pixel point movement information indicates that there is no smear, the original scene image is used as the target scene image, and the target scene image is output.

[0033] Based on the above technical solution, when the pixel point movement information indicates that there is no smear, the original scene image is directly output as the target scene image, thereby effectively improving the processing efficiency of the original scene image.

[0034] In a second aspect, the present invention provides an image processing device, comprising:

[0035] An acquisition module, configured to acquire an original scene image and pixel movement information of the original scene image;

[0036] a determination module, configured to determine a target pixel point in the original scene image and a movement trajectory of the target pixel point based on the pixel point movement information;

[0037] The restoration module is used to restore the target pixel points according to the movement trajectory to obtain a target scene image corresponding to the original scene image.

[0038] According to the second aspect, the restoration module is further configured to:

[0039] Determining a starting position of the target pixel point within an exposure period of the original scene image according to the movement trajectory;

[0040] Obtaining a fill pixel at the current position of the target pixel;

[0041] The target pixel point is moved from the current position to the starting position, and the filling pixel point is filled to the current position to obtain a target scene image corresponding to the original scene image.

[0042] According to the second aspect, or any implementation of the second aspect above, the restoration module is further configured to:

[0043] Acquire a previous frame of target scene image, and inter-frame pixel movement information between the previous frame of target scene image and the original scene image;

[0044] According to the previous frame target scene image and the inter-frame pixel movement information, the initial pixel point of the current position within the exposure period of the original scene image is determined, and the initial pixel point is used as the filling pixel point of the current position.

[0045] According to the second aspect, or any implementation of the second aspect above, the determining module is further configured to:

[0046] Determining, according to the pixel point movement information, moving pixels in the original scene image that have moved during an exposure period, and using the moving pixels as target pixels;

[0047] Determining a starting position of the target pixel within an exposure period according to the pixel movement information;

[0048] A moving trajectory of the target pixel is generated according to the starting position and the current position of the target pixel.

[0049] According to the second aspect, or any implementation of the second aspect above, the image processing device further includes a first camera module; the first camera module is configured to:

[0050] The target scene is photographed by a standard camera to obtain an original scene image corresponding to the target scene.

[0051] According to the second aspect, or any implementation of the second aspect above, the image processing device further includes a second camera module; the second camera module is configured to:

[0052] During the exposure period of the original scene image, the pixel movement information of the original scene image is collected by an event-driven camera.

[0053] According to the second aspect, or any implementation of the second aspect above, the image processing device further includes a direct input module; the direct input module is configured to:

[0054] When the pixel point movement information indicates that there is no smear, the original scene image is used as the target scene image, and the target scene image is output.

[0055] The second aspect and any implementation of the second aspect correspond to the first aspect and any implementation of the first aspect, respectively. The technical effects corresponding to the second aspect and any implementation of the second aspect can be referred to the technical effects corresponding to the first aspect and any implementation of the first aspect, and will not be repeated here.

[0056] In a third aspect, the present invention provides an image processing device, comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program is configured to implement the steps of the image processing method described above.

[0057] The third aspect and any implementation of the third aspect correspond to the first aspect and any implementation of the first aspect, respectively. The technical effects corresponding to the third aspect and any implementation of the third aspect can be referred to the technical effects corresponding to the first aspect and any implementation of the first aspect, and will not be repeated here.

[0058] In a fourth aspect, the present invention provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the processor executes an image processing method as described in any one of the above-mentioned first aspect or possible implementations of the first aspect.

[0059] The fourth aspect and any implementation of the fourth aspect correspond to the first aspect and any implementation of the first aspect, respectively. The technical effects corresponding to the fourth aspect and any implementation of the fourth aspect can be referred to the technical effects corresponding to the first aspect and any implementation of the first aspect, and will not be repeated here.

[0060] In a fifth aspect, an embodiment of the present invention provides a computer program, which includes instructions for executing the image processing method in the first aspect and any possible implementation of the first aspect.

[0061] The fifth aspect and any implementation of the fifth aspect correspond to the first aspect and any implementation of the first aspect, respectively. The technical effects corresponding to the fifth aspect and any implementation of the fifth aspect can be referred to the technical effects corresponding to the first aspect and any implementation of the first aspect, and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0062] Figure 1 is a schematic diagram of a video see-through technology involved in an embodiment of the present invention;

[0063] Figure 2 A schematic diagram of the structure of an image processing device in a hardware operating environment according to an embodiment of the present invention;

[0064] Figure 3 1 is a flow chart of a first embodiment of an image processing method according to the present invention;

[0065] Figure 4 2 is a flow chart of a second embodiment of an image processing method according to the present invention;

[0066] Figure 5 Schematic diagram of the structure of an image processing device involved in an embodiment of the present invention.

[0067] The purpose, features and advantages of the present invention will be further described with reference to the accompanying drawings and in conjunction with the embodiments. DETAILED DESCRIPTION

[0068] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of them. All other embodiments obtained by ordinary technicians in this field based on the embodiments of the present invention without making any creative efforts shall fall within the scope of protection of the present invention.

[0069] The term "and / or" in this article is merely a description of the association relationship between associated objects, indicating that three relationships may exist. For example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone.

[0070] The terms "first" and "second" in the description and claims of the embodiments of the present invention are used to distinguish different objects rather than to describe a specific order of objects. For example, "first target object" and "second target object" are used to distinguish different objects rather than to describe a specific order of objects.

[0071] In the embodiments of the present invention, words such as "exemplary" or "for example" are used to indicate examples, illustrations, or descriptions. Any embodiment or design described as "exemplary" or "for example" in the embodiments of the present invention should not be construed as being preferred or advantageous over other embodiments or designs. Rather, the use of words such as "exemplary" or "for example" is intended to present the relevant concepts in a concrete manner.

[0072] In the description of the embodiments of the present invention, unless otherwise specified, "a plurality" means two or more. For example, a plurality of processing units means two or more processing units; a plurality of systems means two or more systems.

[0073] To make the description of the following embodiments clear and concise, a brief introduction to an implementation of an image processing method is first given:

[0074] With the explosive growth of the VR (Virtual Reality) and AR (Augmented Reality) industries in the past two years, their value in industry, medical care, entertainment, office, and social interaction has gradually been revealed. Currently, major manufacturers are also exploring and deploying in the MR (Mixed Reality) field.

[0075] The main technical route for realizing MR at present is based on VST (Video See-Through) technology. Figure 1 , Figure 1 Figure 2 is a schematic diagram of the video see-through technology involved in an embodiment of the present invention. In VST technology, a real-time view of the real world is captured by a camera 10. This real-time view is then combined with a virtual view of the digital world, presented on a display screen 20, and the combined image is presented to the user. VST technology allows for complete control of visual integration, allowing for complete occlusion between virtual and real objects, and even for higher-level modification of real objects.

[0076] However, cameras that use VST technology to capture real-world scenes typically use a rolling shutter. Therefore, images are prone to ghosting in situations such as long exposure time, jitter, or motion of external scenery, resulting in poor output image quality and affecting the user's MR experience.

[0077] The present invention designs an image processing method that restores the original scene image through pixel movement information, thereby solving the technical problem that the image is prone to ghosting when the camera has a long exposure time, is shaken, or when external scenery moves. It improves the clarity of the output image and ensures the user's MR experience.

[0078] In some embodiments, an original scene image and pixel movement information of the original scene image are obtained; based on the pixel movement information, a target pixel in the original scene image and a movement trajectory of the target pixel are determined; and based on the movement trajectory, the target pixel is restored to obtain a target scene image corresponding to the original scene image. Thus, this embodiment restores the target pixel using its movement trajectory, thereby eliminating ghosting in the original scene image caused by long exposure time, jitter, or external scene movement, and obtaining a target scene image corresponding to the original scene image. This solves the technical problem that images are prone to ghosting due to long exposure time, jitter, or external scene movement, thereby improving the clarity of the output image and ensuring the user's MR experience.

[0079] Reference Figure 2 , Figure 2 This is a schematic diagram of the structure of an image processing device in the hardware operating environment involved in an embodiment of the present invention.

[0080] Specifically, the image processing device may be an MR device, a PC (Personal Computer), a tablet computer, a portable computer, a server or other device.

[0081] like Figure 2As shown, the image processing device may include: a processor 1001, such as a central processing unit (CPU), a communication bus 1002, a user interface 1003, a network interface 1004, and a memory 1005. The communication bus 1002 is used to implement connection and communication between these components. The user interface 1003 may include a display screen and an input unit such as a keyboard. Optionally, the user interface 1003 may also include a standard wired interface or a wireless interface. The network interface 1004 may optionally include a standard wired interface or a wireless interface (such as a wireless fidelity (WI-FI) interface). The memory 1005 may be a high-speed random access memory (RAM) memory or a stable non-volatile memory (NVM), such as a disk storage. The memory 1005 may also be a storage device independent of the aforementioned processor 1001.

[0082] Those skilled in the art will understand that Figure 2 The structure shown in the figure does not constitute a limitation on the image processing device, and may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently.

[0083] like Figure 2 As shown, the memory 1005 as a storage medium may include an operating system, a data storage module, a network communication module, a user interface module and a computer program.

[0084] exist Figure 2 In the image processing device shown, the network interface 1004 is mainly used for data communication with other devices; the user interface 1003 is mainly used for data interaction with the user; the processor 1001 and the memory 1005 in the image processing device of the present invention can be set in the image processing device, and the image processing device calls the computer program stored in the memory 1005 through the processor 1001 and executes the image processing method provided by the embodiment of the present invention.

[0085] It should be understood that the above description is merely an example listed for a better understanding of the technical solution of this embodiment, and is not intended to be the sole limitation on this embodiment.

[0086] The following combination Figure 3 The flowchart of the first embodiment of the image processing method is shown in FIG.

[0087] See also Figure 3An image processing method according to an embodiment of the present invention provides an image processing method, the image processing method comprising:

[0088] Step S100, obtaining an original scene image and pixel movement information of the original scene image;

[0089] In this embodiment, it should be noted that the original scene image is an image obtained by photographing the target scene with a standard camera, wherein the standard camera is a camera equipped with a conventional image sensor such as a CIS (CMOS Image Sensor). The pixel point movement information at least includes the coordinate position of the pixel point of the original scene image captured by the EVS (Event-based Vision Sensor) in the imaging image of the standard camera during the exposure period and the movement information such as the time point corresponding to the coordinate position. It is understandable that an event-driven camera equipped with an EVS can be used to collect the pixel point movement information.

[0090] Before obtaining the original scene image in step S100, the following steps are included:

[0091] Step S110 : photographing a target scene with a standard camera to obtain an original scene image corresponding to the target scene.

[0092] Specifically, the target scene is a scene that the user desires to shoot. In this embodiment, the target scene is shot by a standard camera, and an original scene image corresponding to the target scene can be obtained. Before obtaining the pixel movement information of the original scene image in step S100, the following steps are included:

[0093] Step S120 : During the exposure period of the original scene image, the pixel movement information of the original scene image is collected by an event-driven camera.

[0094] In this embodiment, it should be noted that the exposure period is the time period from the moment the shutter of the standard camera opens to the moment pixels in the original image are exposed, during the process of the standard camera capturing the target scene to obtain the original scene image. For example, using a rolling shutter as an example, this is achieved through a CI S line-by-line exposure method. At the beginning of exposure, the CI S scans and exposes the standard camera's image line by line until all pixels in the image are exposed, thereby obtaining the original scene image.

[0095] It is understandable that CIS is based on charge, and the charge needs to be integrated before there can be an output. In other words, in the process of charge integration, CIS pixels do not distinguish the changes in this process, which limits the sampling frequency of the standard camera. EVS is based on PD (Photo-Diode) current, monitoring whether the current signal of PD has changed. The transformation exceeds a given threshold. Taking the 2-bit signal output as an example, if the current signal becomes stronger, 01 can be output, and if the current signal becomes weaker, 10 can be output. If the current signal change does not exceed the threshold, the output is 00. At the same time, its calculation process is that all pixels perform digital-to-analog conversion at the same time, which is a parallel process. And because the conversion is simple, this 2-bit digital-to-analog conversion is very fast, thereby achieving very high-speed conversion and output of the entire pixel array, so the sampling frequency of the event-driven camera is much higher than that of the standard camera.

[0096] For example, a standard camera using a CIS and an event-driven camera using an EVS can be provided in an MR device. Thus, while the standard camera is capturing a target scene to obtain an original scene image, the event-driven camera can collect pixel motion information of the original scene image during the exposure period of the original scene image. The pixel motion information includes at least the coordinate positions of the pixels of the original scene image captured by the EVS within the imaging frame of the standard camera during the exposure period, as well as the corresponding time points of the coordinate positions.

[0097] Step S200, determining a target pixel point in the original scene image and a movement trajectory of the target pixel point according to the pixel point movement information;

[0098] After obtaining the pixel movement information, the pixel points that moved during the exposure period of the original scene image (i.e., the target pixel points in the original scene image) and the movement trajectory of the target pixel points during the exposure period can be determined based on the pixel movement information.

[0099] Wherein, step S200 determines the target pixel point in the original scene image and the movement trajectory of the target pixel point according to the pixel point movement information, including:

[0100] Step S210, determining a moving pixel point in the original scene image that moves during an exposure period according to the pixel point movement information, and using the moving pixel point as a target pixel point;

[0101] Step S220, determining the starting position of the target pixel point within the exposure period according to the pixel point movement information;

[0102] Step S230 : generating a moving trajectory of the target pixel point according to the starting position and the current position of the target pixel point.

[0103] It is understandable that EVS detects changes in each pixel in each imaging screen in an asynchronous manner. Therefore, based on the pixel movement information, it can determine the moving pixel points in the original scene image that move during the exposure period, and use the moving pixel points as target pixels. Then, based on the pixel movement information, the starting position of the target pixel point within the exposure period (i.e., the position of the target pixel point at the beginning of exposure) is determined. Then, based on the starting position and the current position of the target pixel point, the movement trajectory of the target pixel point can be generated.

[0104] This embodiment determines, based on the pixel movement information, the moving pixels in the original scene image that have moved during the exposure period and uses these moving pixels as target pixels. Furthermore, based on the pixel movement information, the starting position of the target pixel within the exposure period is determined; thus, a movement trajectory of the target pixel is generated based on the starting position and the current position of the target pixel. Based on the pixel movement information, this embodiment determines the target pixels in the original scene image that have moved during the exposure period captured by the standard camera and the movement trajectory of the target pixels, so that the position of the target pixels can be subsequently restored based on this movement trajectory.

[0105] Step S300 : Restoring the target pixel points according to the movement trajectory to obtain a target scene image corresponding to the original scene image.

[0106] After obtaining the movement trajectory of the target pixel, the position of the target pixel on the original scene image at the time of exposure on the imaging screen can be determined (i.e., the starting position of the target pixel within the exposure period of the original scene image). The target pixel is then restored to the starting position, thereby eliminating smear caused by reasons such as long exposure time, jitter, or external scene movement, and obtaining a target scene image corresponding to the original scene image after smear elimination.

[0107] Furthermore, before determining the target pixel point and the movement trajectory of the target pixel point in the original scene image according to the pixel point movement information in step S200, the image processing method further includes:

[0108] Step S400 : When the pixel point movement information indicates that there is no smear, the original scene image is used as a target scene image, and the target scene image is output.

[0109] It is understandable that, based on the pixel movement information, it can be determined whether the standard camera shooting produces ghosting. Exemplarily, based on the pixel movement information, the number of target pixels is determined. When the number of target pixels is lower than a preset ratio threshold (0.3%, 0.5%, 1.0%, etc.), it indicates that the standard camera did not produce ghosting when shooting the original scene image, and it can be determined that the pixel movement information indicates no ghosting. When the number of target pixels is not lower than the preset ratio threshold, it indicates that the standard camera produced ghosting when shooting the original scene image, and it can be determined that the pixel movement information indicates the presence of ghosting, and step S200 is executed.

[0110] This embodiment directly outputs the original scene image as the target scene image when the pixel point movement information indicates that there is no smear, thereby effectively improving the processing efficiency of the original scene image.

[0111] In a first embodiment of the present invention, an original scene image and pixel movement information of the original scene image are obtained; based on the pixel movement information, a target pixel in the original scene image and a movement trajectory of the target pixel are determined; and based on the movement trajectory, the target pixel is restored to obtain a target scene image corresponding to the original scene image. Thus, this embodiment restores the target pixel using its movement trajectory, thereby eliminating ghosting in the original scene image caused by long exposure time, jitter, or external scene movement, and obtaining a target scene image corresponding to the original scene image. This solves the technical problem that images are prone to ghosting due to long exposure time, jitter, or external scene movement, thereby improving the clarity of the output image and ensuring the user's MR experience.

[0112] The following combination Figure 4 FIG1 is a flow chart of a second embodiment of an image processing method shown in FIG1 , which describes the image processing method in detail.

[0113] See also Figure 4 In another embodiment of the image processing method of the present invention, step S300 restores the target pixel points according to the movement trajectory to obtain a target scene image corresponding to the original scene image, including:

[0114] Step S310, determining the starting position of the target pixel point within the exposure period of the original scene image according to the movement trajectory;

[0115] Step S320, obtaining a filling pixel point at the current position of the target pixel point;

[0116] Step S330 , moving the target pixel point from the current position to the starting position, and filling the filling pixel point to the current position, to obtain a target scene image corresponding to the original scene image.

[0117] In this embodiment, according to the movement trajectory, the starting position of the target pixel point within the exposure period of the original scene image (i.e., the position of the target pixel point on the original scene image when the exposure starts on the imaging screen) is determined. Since the target pixel point will be missing pixels after the current position of the target pixel point is restored to the starting position, it is necessary to obtain the filling pixel point of the current position of the target pixel point to fill the missing pixels after the target pixel point moves to the starting position. For example, since all pixels in the imaging screen are moving pixels after the event-driven camera is turned on, all pixels on the imaging screen can be obtained at this time to obtain an initial EVS image, and then the initial EVS image is updated by collecting real-time pixel movement information to obtain a real-time EVS image. Then, the current EVS image corresponding to the starting moment of the exposure period of the original scene image can be determined based on the real-time EVS image, and the pixel points of the current EVS image at the current position are used as corresponding filling pixels. Then, the target pixel point can be moved from the current position to the starting position, and the filling pixel points are filled to the current position to obtain the target scene image corresponding to the original scene image.

[0118] This embodiment determines the starting position of the target pixel point within the exposure period of the original scene image based on the movement trajectory, and obtains the filling pixel point at the current position of the target pixel point. Thus, on the one hand, the target pixel point is moved from the current position to the starting position so that all target pixels are restored to the position at the beginning of exposure (i.e., the initial position), and then the filling pixel point is filled into the current position to fill the vacancy left at the current position after the target pixel point is moved from the current position to the starting position, thereby obtaining a target scene image corresponding to the original scene image.

[0119] Wherein, step S320 obtains the filling pixel point at the current position of the target pixel point, including:

[0120] Step S321, obtaining a previous frame of target scene image, and inter-frame pixel movement information between the previous frame of target scene image and the original scene image;

[0121] Step S322: Determine the initial pixel of the current position within the exposure period of the original scene image based on the previous frame target scene image and the inter-frame pixel movement information, and use the initial pixel as the filling pixel of the current position.

[0122] In this embodiment, by obtaining the previous frame of the target scene image and the inter-frame pixel point movement information between the previous frame of the target scene image and the original scene image, the initial pixel point of the current position within the exposure period of the original scene image (i.e., the pixel point of the current position on the original scene image when exposure begins on the imaging screen) can be determined based on the previous frame of the target scene image and the inter-frame pixel point movement information, and the initial pixel point can be used as the filling pixel point of the current position. This can be used to fill the gap left by the target pixel point at the current position after it is moved from the current position to the starting position, thereby ensuring the integrity of the output target scene image.

[0123] This embodiment uses the previous target scene image and the inter-frame pixel movement information between the previous target scene image and the original scene image to improve the accuracy of the pixels used to fill the gap at the current location, thereby ensuring the accuracy of the output target scene image. Furthermore, using the previous target scene image and inter-frame pixel movement information can effectively reduce the amount of image processing data, avoiding the need to update the initial EVS image in real time, thereby improving image processing efficiency.

[0124] refer to Figure 5 , Figure 5 Schematic diagram of the structure of an image processing device involved in an embodiment of the present invention.

[0125] See also Figure 5 The present invention provides an image processing device, comprising:

[0126] An acquisition module 10 is configured to acquire an original scene image and pixel movement information of the original scene image;

[0127] A determination module 20 is configured to determine a target pixel point in the original scene image and a movement trajectory of the target pixel point based on the pixel point movement information;

[0128] The restoration module 30 is configured to restore the target pixel points according to the movement trajectory to obtain a target scene image corresponding to the original scene image.

[0129] Optionally, the restoration module 30 is further configured to:

[0130] Determining a starting position of the target pixel point within an exposure period of the original scene image according to the movement trajectory;

[0131] Obtaining a fill pixel at the current position of the target pixel;

[0132] The target pixel point is moved from the current position to the starting position, and the filling pixel point is filled to the current position to obtain a target scene image corresponding to the original scene image.

[0133] Optionally, the restoration module 30 is further configured to:

[0134] Acquire a previous frame of target scene image, and inter-frame pixel movement information between the previous frame of target scene image and the original scene image;

[0135] According to the previous frame target scene image and the inter-frame pixel movement information, the initial pixel point of the current position within the exposure period of the original scene image is determined, and the initial pixel point is used as the filling pixel point of the current position.

[0136] Optionally, the determination module 20 is further configured to:

[0137] Determining, according to the pixel point movement information, moving pixels in the original scene image that have moved during an exposure period, and using the moving pixels as target pixels;

[0138] Determining a starting position of the target pixel within an exposure period according to the pixel movement information;

[0139] A moving trajectory of the target pixel is generated according to the starting position and the current position of the target pixel.

[0140] Optionally, the image processing device further includes a first camera module; the first camera module is configured to:

[0141] The target scene is photographed by a standard camera to obtain an original scene image corresponding to the target scene.

[0142] Optionally, the image processing device further includes a second camera module; the second camera module is configured to:

[0143] During the exposure period of the original scene image, the pixel movement information of the original scene image is collected by an event-driven camera.

[0144] Optionally, the image processing device further includes a direct input module; the direct input module is used to:

[0145] When the pixel point movement information indicates that there is no smear, the original scene image is used as the target scene image, and the target scene image is output.

[0146] It can be understood that the image processing device implements the operations in the image processing method provided in the above embodiment. The specific implementation steps can refer to the description of the above embodiment and will not be elaborated here.

[0147] In addition, an embodiment of the present invention further provides a computer storage medium, on which a computer program is stored. When the computer program is executed by a processor, the operations in the image processing method provided in the above embodiment are implemented. The specific steps are not described in detail here.

[0148] It should be noted that, in this document, the terms "comprises," "includes," or any other variations thereof are intended to encompass non-exclusive inclusion, such that a process, method, article, or system comprising a series of elements includes not only those elements but also other elements not explicitly listed, or elements inherent to such process, method, article, or system. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of other identical elements in the process, method, article, or system comprising the element.

[0149] The serial numbers of the above embodiments of the present invention are for description only and do not represent the advantages or disadvantages of the embodiments.

[0150] Through the description of the above embodiments, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus the necessary general hardware platform, and of course can also be implemented by hardware, but in many cases the former is a better embodiment. Based on this understanding, the technical solution of the present invention is essentially or the part that contributes to the prior art can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) as described above, and includes a number of instructions for enabling a terminal device (which can be a mobile phone, computer, server, or network device, etc.) to execute the methods described in each embodiment of the present invention.

[0151] The above are only preferred embodiments of the present invention and are not intended to limit the patent scope of the present invention. Any equivalent structure or equivalent process transformation made using the contents of the present invention description and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present invention.

Claims

1. An image processing method, characterized in that: The image processing method comprises: Acquiring an original scene image and pixel movement information of the original scene image; Determining a target pixel point in the original scene image and a movement trajectory of the target pixel point according to the pixel point movement information; Restoring the target pixel points according to the movement trajectory to obtain a target scene image corresponding to the original scene image; The step of restoring the target pixel points according to the movement trajectory to obtain a target scene image corresponding to the original scene image includes: Determining a starting position of the target pixel point within an exposure period of the original scene image according to the movement trajectory; Obtaining a fill pixel at the current position of the target pixel; The target pixel point is moved from the current position to the starting position, and the filling pixel point is filled to the current position to obtain a target scene image corresponding to the original scene image.

2. The image processing method according to claim 1, wherein: The step of obtaining the filling pixel point at the current position of the target pixel point includes: Acquire a previous frame of target scene image, and inter-frame pixel movement information between the previous frame of target scene image and the original scene image; According to the previous frame target scene image and the inter-frame pixel movement information, the initial pixel point of the current position within the exposure period of the original scene image is determined, and the initial pixel point is used as the filling pixel point of the current position.

3. The image processing method according to claim 1, wherein: The step of determining the target pixel point in the original scene image and the movement trajectory of the target pixel point based on the pixel point movement information includes: Determining, according to the pixel point movement information, moving pixels in the original scene image that have moved during an exposure period, and using the moving pixels as target pixels; Determining a starting position of the target pixel within an exposure period according to the pixel movement information; A moving trajectory of the target pixel is generated according to the starting position and the current position of the target pixel.

4. The image processing method according to claim 1, wherein: Before the step of obtaining the original scene image, the method includes: The target scene is photographed by a standard camera to obtain an original scene image corresponding to the target scene.

5. The image processing method according to claim 4, wherein: Before the step of obtaining pixel movement information of the original scene image, the method includes: During the exposure period of the original scene image, the pixel movement information of the original scene image is collected by an event-driven camera.

6. The image processing method according to any one of claims 1 to 5, wherein: Before the step of determining the target pixel point and the movement trajectory of the target pixel point in the original scene image based on the pixel point movement information, the image processing method further includes: When the pixel point movement information indicates that there is no smear, the original scene image is used as the target scene image, and the target scene image is output.

7. An image processing device, characterized in that The image processing device comprises: An acquisition module, configured to acquire an original scene image and pixel movement information of the original scene image; a determination module, configured to determine a target pixel point in the original scene image and a movement trajectory of the target pixel point based on the pixel point movement information; A restoration module, configured to restore the target pixel points according to the movement trajectory to obtain a target scene image corresponding to the original scene image; The restore module is also used to: Determining a starting position of the target pixel point within an exposure period of the original scene image according to the movement trajectory; Obtaining a fill pixel at the current position of the target pixel; The target pixel point is moved from the current position to the starting position, and the filling pixel point is filled to the current position to obtain a target scene image corresponding to the original scene image.

8. An image processing device, characterized in that The image processing device includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program is configured to implement the steps of the image processing method according to any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the image processing method according to any one of claims 1 to 6 are implemented.

Images