A dynamic single-pixel imaging method, device, storage medium and terminal device

Abstract

The present application discloses a dynamic single-pixel imaging method, device, storage medium and terminal device, the method comprising: obtaining several candidate image frames of a target scene by a single-pixel imaging method; determining a background mask corresponding to the target scene based on an object mask of each candidate image frame; determining a projection matrix of each projection pattern in a projection pattern sequence corresponding to each candidate image frame based on the background mask and the object mask corresponding to the candidate image frame, and obtaining a projection matrix sequence; reconstructing a target image based on each projection matrix in the projection matrix sequence and its corresponding bucket detector value. The present application determines the background mask of the target scene based on the frequency of occurrence of background pixels in all candidate image frames, and then determines each background projection image based on the background mask, so that background areas with low occurrence frequency can be filtered out, the accuracy of the object area in the projection matrix is ​​improved, and the image quality of the target image reconstructed based on the projection matrix and the bucket detector value is improved.

Description

Technical Field

[0001] The present application relates to the field of optical imaging technology, and in particular to a dynamic single-pixel imaging method, device, storage medium and terminal equipment. Background Art

[0002] Single-pixel imaging is to illuminate the object with a light source, and then use a detector with spatial light field resolution to obtain the light field reflected or projected from the object for detection and imaging. Compared with traditional optical imaging, single-pixel imaging has broad application prospects in scenes such as non-visible light, low illumination, and scattering. In practical applications, most imaging scenes will have object occlusions, which will cause the target object area to be determined as the background area in the restored image due to occlusion, making it impossible for existing single-pixel imaging to guarantee the imaging quality of moving objects in occluded scenes.

[0003] Therefore the prior art still needs to be improved and enhanced. Summary of the invention

[0004] The technical problem to be solved by the present application is to provide a dynamic single-pixel imaging method, device, storage medium and terminal device in view of the deficiencies in the prior art.

[0005] In order to solve the above technical problems, the first aspect of the embodiments of the present application provides a dynamic single-pixel imaging method, the method comprising:

[0006] A single-pixel imaging method is used to acquire several candidate image frames of the target scene;

[0007] Determining a background mask corresponding to the target scene based on an object mask corresponding to each of the candidate image frames;

[0008] For each candidate image frame among the plurality of candidate image frames, based on the background mask and the object mask corresponding to the candidate image frame, determine the projection matrix corresponding to each projection pattern in the projection pattern sequence corresponding to the candidate image frame, so as to obtain a projection matrix sequence;

[0009] A target image is reconstructed based on each projection matrix in the projection matrix sequence and its corresponding bucket detector value.

[0010] In the dynamic single-pixel imaging method, the target scene includes a plurality of moving objects, and each of the plurality of candidate image frames includes each of the plurality of moving objects.

[0011] In the dynamic single-pixel imaging method, the step of determining the background mask corresponding to the target scene based on the object mask corresponding to each of the candidate image frames in the plurality of candidate image frames specifically comprises:

[0012] For each candidate image frame among the plurality of candidate image frames, determining a candidate background mask corresponding to the candidate image frame;

[0013] Calculating the pixel mean of each pixel position in all the determined candidate background masks, and determining the reference background mask based on all the determined pixel means;

[0014] Based on a preset background threshold and the reference background mask, a background mask corresponding to the target scene is determined.

[0015] The dynamic single-pixel imaging method, wherein for each candidate image frame among the plurality of candidate image frames, determining, based on the background mask and the object mask corresponding to the candidate image frame, the projection matrix corresponding to each projection pattern in the projection pattern sequence corresponding to the candidate image frame to obtain the projection matrix sequence specifically includes:

[0016] For each candidate image frame among the plurality of candidate image frames, determining, based on an object mask corresponding to the candidate image frame, an object projection image corresponding to each projection pattern in a projection pattern sequence corresponding to the candidate image frame;

[0017] Determine the background projection images corresponding to the projection patterns based on the object projection images corresponding to the projection patterns and the background mask;

[0018] Based on the object projection image and the background projection image corresponding to each projection pattern, the projection matrix corresponding to each projection pattern is determined to obtain a projection matrix sequence.

[0019] The dynamic single pixel imaging method, wherein the reconstructing the target image based on each projection matrix in the projection matrix sequence and its corresponding bucket detector value specifically comprises:

[0020] Construct the initial target matrix;

[0021] The initial target matrix is ​​updated based on each projection matrix in the projection matrix sequence and its corresponding bucket detector value to obtain a candidate target matrix;

[0022] Detecting whether the candidate target matrix meets a preset condition;

[0023] When the candidate target matrix satisfies a preset condition, determining a target image based on the candidate target matrix;

[0024] When the candidate target matrix does not meet the preset conditions, the reconstructed image is used as the initial target matrix, and the step of determining the candidate target matrix for the initial target matrix based on each projection matrix in the projection matrix sequence and its corresponding bucket detector value is continued until the candidate target matrix meets the preset conditions.

[0025] The dynamic single-pixel imaging method, wherein the updating of the initial target matrix based on each projection matrix in the projection matrix sequence and its corresponding bucket detector value to obtain a candidate target matrix specifically includes:

[0026] For a first projection matrix in the projection matrix sequence, determining a first intermediate variable based on the projection matrix and the initial target matrix;

[0027] Determining a second intermediate variable based on the bucket detector value corresponding to the projection matrix and the first intermediate variable;

[0028] Update the initial target matrix based on the first projection matrix, the first intermediate variable and the second intermediate variable;

[0029] The updated initial target matrix is ​​used as the initial target matrix, and the step of determining the first intermediate variable by the projection matrix and the initial target matrix is ​​performed for the second projection matrix in the projection matrix sequence until the last projection matrix is ​​obtained to obtain a candidate target matrix.

[0030] The dynamic single-pixel imaging method, wherein the expression of the first intermediate variable is:

[0031] I pm =(Object) T *Phi(k)

[0032] Among them, I pm represents the first intermediate variable, Object represents the initial target matrix, (·) T represents matrix transpose, and Phi(k) represents the kth projection matrix in the projection matrix sequence.

[0033] The dynamic single-pixel imaging method, wherein the expression of the second intermediate variable is:

[0034] I pm ′=I pm -mean(I pm )+B(k) / numel(I pm )

[0035] Among them, I pm ' represents the second intermediate variable, I pm represents the first intermediate variable, B(k) represents the bucket detector value corresponding to the k-th projection matrix, mean represents the average value, and numel represents the number of pixels contained in the projection matrix.

[0036] In the dynamic single-pixel imaging method, the update formula corresponding to the initial target matrix is:

[0037]

[0038] Among them, I pm ' represents the second intermediate variable, I pm represents the first intermediate variable, Phi(k) represents the kth projection matrix in the projection matrix sequence

[0039] The dynamic single-pixel imaging method, wherein the preset condition is that the image difference between the candidate target matrix and the previous candidate target matrix is ​​less than a preset difference threshold.

[0040] A second aspect of an embodiment of the present application provides a dynamic single-pixel imaging device, the device comprising:

[0041] An acquisition module, which acquires several candidate image frames of the target scene using a single-pixel imaging method;

[0042] A first determination module is used to determine a background mask corresponding to the target scene based on an object mask corresponding to each of the candidate image frames in the plurality of candidate image frames;

[0043] A second determination module is used to determine, for each candidate image frame among the plurality of candidate image frames, a projection matrix corresponding to each projection pattern in a projection pattern sequence corresponding to the candidate image frame based on the background mask and the object mask corresponding to the candidate image frame, so as to obtain a projection matrix sequence;

[0044] A reconstruction module is used to reconstruct a target image based on each projection matrix in the projection matrix sequence and its corresponding bucket detector value.

[0045] A third aspect of an embodiment of the present application provides a computer-readable storage medium, which stores one or more programs. The one or more programs can be executed by one or more processors to implement the steps in any of the dynamic single-pixel imaging methods described above.

[0046] A fourth aspect of the embodiments of the present application provides a terminal device, comprising: a processor, a memory, and a communication bus; the memory stores a computer-readable program that can be executed by the processor;

[0047] The communication bus realizes the connection and communication between the processor and the memory;

[0048] When the processor executes the computer-readable program, the steps in any of the dynamic single-pixel imaging methods described above are implemented.

[0049] Beneficial effects: Compared with the prior art, the present application provides a dynamic single-pixel imaging method, device, storage medium and terminal device, the method comprising: acquiring several candidate image frames of a target scene by a single-pixel imaging method; determining the background mask corresponding to the target scene based on the object mask corresponding to each candidate image frame in the several candidate image frames; for each candidate image frame in the several candidate image frames, determining the projection matrix corresponding to each projection pattern in the projection pattern sequence corresponding to the candidate image frame based on the background mask and the object mask corresponding to the candidate image frame, so as to obtain a projection matrix sequence; reconstructing the target image based on each projection matrix in the projection matrix sequence and its corresponding bucket detector value. The present application determines the background mask of the target scene based on the occurrence frequency of each background pixel in the background area in all candidate image frames, and then determines the background projection image of each projection image based on the background mask, so as to filter out the background area with low occurrence frequency in the projection pattern, improve the accuracy of the object area in the projection matrix, and thus improve the image quality of the target image reconstructed based on the projection matrix and the bucket detector value. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without inventive work.

[0051] Figure 1 A flow chart of the dynamic single-pixel imaging method provided in this application.

[0052] Figure 2 A schematic diagram of a device for acquiring candidate image frames in the dynamic single-pixel imaging method provided in the present application.

[0053] Figure 3 A schematic diagram of the process of acquiring a projection matrix sequence in the dynamic single-pixel imaging method provided in the present application.

[0054] Figure 4 A diagram showing the principle process of creating a projection matrix in the dynamic single-pixel imaging method provided in this application.

[0055] Figure 5 A schematic flow chart of the iterative process of the initial target matrix in the dynamic single-pixel imaging method provided in the present application.

[0056] Figure 6 A schematic diagram of the reconstruction process of the object area and the background area in the dynamic single-pixel imaging method provided in this application.

[0057] Figure 7This is a schematic diagram of the structure of the dynamic single-pixel imaging device provided in this application.

[0058] Figure 8 This is a schematic diagram of the structure of the terminal device provided in this application. DETAILED DESCRIPTION

[0059] The present application provides a dynamic single-pixel imaging method, device, storage medium and terminal device. In order to make the purpose, technical solution and effect of the present application clearer and more specific, the present application is further described in detail with reference to the accompanying drawings and examples. It should be understood that the specific embodiments described here are only used to explain the present application and are not used to limit the present application.

[0060] It will be understood by those skilled in the art that, unless expressly stated, the singular forms "one", "said", and "the" used herein may also include plural forms. It should be further understood that the term "comprising" used in the specification of the present application refers to the presence of the features, integers, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or groups thereof. It should be understood that when we refer to an element as being "connected" or "coupled" to another element, it may be directly connected or coupled to the other element, or there may be an intermediate element. In addition, the "connection" or "coupling" used herein may include wireless connection or wireless coupling. The term "and / or" used herein includes all or any unit and all combinations of one or more associated listed items.

[0061] It will be understood by those skilled in the art that, unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as those generally understood by those skilled in the art to which this application belongs. It should also be understood that terms such as those defined in general dictionaries should be understood to have meanings consistent with those in the context of the prior art, and will not be interpreted with idealized or overly formal meanings unless specifically defined as here.

[0062] It should be understood that the sequence numbers and sizes of the steps in this embodiment do not mean the order of execution. The execution order of each process is determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiment of the present application.

[0063] After research, the inventor found that single-pixel imaging is to illuminate the object with a light source, and obtain the light field reflected or projected from the object for detection and imaging through a detector with spatial light field resolution capability. Compared with traditional optical imaging, single-pixel imaging has broad application prospects in scenes such as non-visible light, low illumination, and scattering. In practical applications, the target objects in the imaging scene are generally dynamic, so the research on single-pixel imaging is undergoing a transition from static to dynamic. Among them, the dynamic single-pixel imaging scheme mainly includes a scheme for improving the imaging accuracy of shaking objects through Fourier transform ghost diffraction; a scheme for using motion estimation to correct the motion of the projection pattern and reduce the motion blur of the object; and a scheme for adaptively finding the imaging object range and reducing sampling through a series of methods. However, the existing dynamic single-pixel imaging scheme generally considers the situation where there is object occlusion in the imaging scene. However, there will be object occlusion in most imaging scenes. Object occlusion will determine the target object area as the background area in the restored image due to occlusion, so that the existing single-pixel imaging cannot guarantee the imaging quality of moving objects in the occluded scene.

[0064] In order to solve the above problems, in an embodiment of the present application, a single-pixel imaging method is used to obtain several candidate image frames of the target scene; based on the object mask corresponding to each candidate image frame in the several candidate image frames, the background mask corresponding to the target scene is determined; for each candidate image frame in the several candidate image frames, based on the background mask and the object mask corresponding to the candidate image frame, the projection matrix corresponding to each projection pattern in the projection pattern sequence corresponding to the candidate image frame is determined to obtain a projection matrix sequence; based on each projection matrix in the projection matrix sequence and its corresponding bucket detector value, the target image is reconstructed. The present application determines the background mask of the target scene based on the frequency of occurrence of each background pixel in the background area in all candidate image frames, and then determines the background projection image of each projection image based on the background mask, so that the background area with low occurrence frequency in the projection pattern can be filtered out, and the accuracy of the object area in the projection matrix can be improved, thereby improving the image quality of the target image reconstructed based on the projection matrix and the bucket detector value.

[0065] The application content is further explained below through the description of embodiments in conjunction with the accompanying drawings.

[0066] This embodiment provides a dynamic single pixel imaging method, such as Figure 1 As shown, the method includes:

[0067] S10. Acquire several candidate image frames of the target scene using a single-pixel imaging method.

[0068] Specifically, the target scene includes a plurality of moving objects, and each of the plurality of candidate image frames includes each of the plurality of moving objects, wherein, in a typical implementation, there is object occlusion in the target scene. For example, the target scene includes a plurality of moving objects and an occluding object, and each of the plurality of moving objects moves at a uniform speed in the target scene. When single-pixel imaging is used to capture the plurality of moving objects for imaging, the occluding object will occlude a portion of a portion of the plurality of moving objects.

[0069] Each of the several candidate image frames corresponds to a projection pattern sequence, and the projection pattern sequence includes several projection images. The candidate image frames are determined based on sequentially projecting each projection pattern in the projection pattern sequence into the target scene through a projection device, and measuring the target scene through a bucket detector when projecting each projection pattern. The bucket detector value is used to reflect the total light intensity of the target scene when projecting the projection pattern. Among them, the number of projection patterns included in the projection pattern sequence corresponding to each candidate image frame can be the same, for example, both include 100. Of course, in actual applications, the number of projection patterns included in the projection pattern sequence corresponding to each candidate image frame may also be different. For example, the projection pattern sequence corresponding to candidate image frame A includes 100 projection patterns, and the projection pattern sequence corresponding to candidate image frame B includes 80 projection patterns. In addition, several candidate image frames are image frame sequences captured during the movement of a moving object in a target scene, wherein each candidate image frame is formed by several captures. For example, as Figure 2 As shown, a projector and a barrel detector are provided in the target scene. The projector is used to project a projection pattern to the target scene, and the barrel detector is used to measure the total light intensity of the projection pattern projected by the projector to the target scene. The moving object moves 20 pixels at a uniform speed in the target scene, and 40 candidate image frames are obtained by performing 4096 acquisitions during the movement process. When determining each candidate image frame, the moving object is set to be stationary relative to the target scene.

[0070] S20: Determine a background mask corresponding to the target scene based on an object mask corresponding to each of the candidate image frames.

[0071] Specifically, the number of object masks corresponding to each candidate image frame is the same as the number of moving objects included in each candidate image frame, and the object masks corresponding to the candidate image frames correspond one-to-one to the moving objects included in the candidate image frames, wherein the object masks corresponding to the candidate image frames can be obtained by performing object recognition on the candidate image frames after the candidate image frames are acquired, for example, using a trained object recognition model, or using an image segmentation method, etc. It will not be described one by one here, and any method that can identify the target object in the candidate image frame is acceptable. In addition, the object mask can be a binary matrix, wherein the pixel value of the area where the object appears is 1, and the pixel value of the area where the object does not appear is 0.

[0072] In one implementation of this embodiment, Figure 3 As shown, the determining of the background mask corresponding to the target scene based on the object mask corresponding to each of the candidate image frames in the plurality of candidate image frames specifically includes:

[0073] For each candidate image frame among the plurality of candidate image frames, determining a candidate background mask corresponding to the candidate image frame;

[0074] Calculating the pixel mean of each pixel position in all the determined candidate background masks, and determining the reference background mask based on all the determined pixel means;

[0075] Based on a preset background threshold and the reference background mask, a background mask corresponding to the target scene is determined.

[0076] Specifically, the candidate background mask is used to reflect the background area in the candidate image frame, and the background area of ​​the candidate image frame can be the image area excluding all object areas included in the candidate image frame. Therefore, the candidate background mask is determined based on the object mask corresponding to the candidate image frame, and accordingly, the calculation formula of the candidate background mask can be:

[0077]

[0078] Among them, mask m represents the candidate background mask of the mth candidate image frame, ones represents a matrix of all 1s and the matrix size is equal to the image size of the candidate image frame; represents the object mask of the i-th moving object in the m-th candidate image frame, and N represents the number of moving objects in the candidate image frame.

[0079] The reference background mask is used to reflect the probability of occurrence of each background pixel in each candidate background mask, that is, the pixel value of each pixel in the reference background mask is used to reflect the probability of the pixel appearing as a background pixel in the candidate background mask corresponding to each candidate image frame. The pixel value of each pixel in the reference background mask is equal to the pixel mean of each pixel position in all candidate background masks. Accordingly, the expression formula of the reference background mask can be:

[0080]

[0081] Where counter_back represents the reference background mask, M represents the number of candidate image frames, and ones represents a matrix of all 1s and the matrix size is equal to the image size of the candidate image frame; represents the object mask of the i-th moving object in the m-th candidate image frame, and N represents the number of moving objects in the candidate image frame.

[0082] The background threshold is pre-set and is used as a basis for determining whether each pixel in the reference background mask of the beam is a background pixel, so as to ensure that when the probability of a background area pixel appearing in a number of candidate image frames is less than the background threshold, it is not used as a background pixel in the target image reconstruction process. This can avoid using object area pixels as background pixels due to occlusion, thereby improving the background reconstruction quality and thus improving the image quality of the target image. In one implementation, the background threshold is a probability threshold of the probability of appearing in a number of candidate image frames. When the pixel value of a pixel in the reference background mask is greater than the background threshold, the pixel is a background pixel. When the pixel value of a pixel in the reference background mask is less than or equal to the background threshold, the pixel is not a background pixel. Based on this, the expression of the background mask can be:

[0083]

[0084] Among them, mask_back represents the background mask, T represents the background threshold, and counter_back(x,y) represents the pixel value of the pixel point at the pixel position (x,y) in the reference background mask.

[0085] S30. For each candidate image frame among the plurality of candidate image frames, based on the background mask and the object mask corresponding to the candidate image frame, determine the projection matrix corresponding to each projection pattern in the projection pattern sequence corresponding to the candidate image frame to obtain a projection matrix sequence.

[0086] Specifically, each projection pattern in the projection pattern sequence corresponding to each candidate image frame corresponds to a projection matrix, and the number of projection matrices included in the projection matrix sequence is equal to the sum of the number of projection patterns included in the projection pattern sequence corresponding to each candidate image frame. For example, the number of candidate image frames is M, and the number of projection images included in the projection pattern sequence corresponding to each candidate image frame is h, then the number of projection matrices included in the projection matrix sequence is equal to M*h; or, the candidate image frames include candidate image frame A and candidate image frame B, the number of projection images included in the projection pattern sequence corresponding to candidate image frame A is C, and the number of projection images included in the projection pattern sequence corresponding to candidate image frame A is D, then the number of projection matrices included in the projection matrix sequence is equal to C+D.

[0087] The projection matrix is ​​obtained by splicing the background projection image and the object projection image in the projection pattern corresponding to the projection matrix. The matrix length of the projection matrix is ​​equal to the sum of the matrix length of the object projection matrix formed by the object projection image in the projection pattern corresponding to the projection matrix and the object mask in the candidate image frame corresponding to the projection matrix. Based on this, the expression of the matrix length of the projection matrix can be:

[0088]

[0089] Among them, P′ j m The projection matrix corresponding to the j-th projection pattern in the m-th candidate image frame is represented by the projection matrix, represents the jth projection pattern corresponding to the mth candidate image frame, Represents the object mask of the i-th moving object in the m-th candidate image frame.

[0090] In one implementation of this embodiment, Figure 3 As shown, for each candidate image frame among the plurality of candidate image frames, based on the background mask and the object mask corresponding to the candidate image frame, determining the projection matrix corresponding to each projection pattern in the projection pattern sequence corresponding to the candidate image frame to obtain the projection matrix sequence specifically includes:

[0091] For each candidate image frame among the plurality of candidate image frames, determining, based on an object mask corresponding to the candidate image frame, an object projection image corresponding to each projection pattern in a projection pattern sequence corresponding to the candidate image frame;

[0092] Determine the background projection images corresponding to the projection patterns based on the object projection images corresponding to the projection patterns and the background mask;

[0093] Based on the object projection image and the background projection image corresponding to each projection pattern, the projection matrix corresponding to each projection pattern is determined to obtain a projection matrix sequence.

[0094] Specifically, the object mask is a mask image of a moving object in a candidate image frame, and the object mask corresponding to the candidate image frame corresponds one-to-one to the moving object in the candidate image frame, that is, for each moving object in the candidate image frame, there is an object mask corresponding to the moving object, and the object mask corresponding to the moving object is used to reflect the image area where the moving object is located in the candidate image frame. The object projection image is used to reflect the projection area corresponding to the motion in the projection pattern, and the calculation formula of the object projection image can be:

[0095]

[0096] in, represents the object projection image of the i-th moving object corresponding to the j-th projection pattern in the m-th candidate image frame, represents the jth projection pattern in the projection pattern sequence corresponding to the mth candidate image frame, Represents the object mask of the i-th moving object in the m-th candidate image frame.

[0097] The background projection image is used to reflect the effective background area in the projection pattern, wherein the background projection image is determined based on the background mask and the projection pattern, and the background projection image may not be equal to the image area after the projection pattern removes the object projection image. This is because the background mask is determined based on the probability of occurrence of background pixels in each candidate image frame, and the background pixels whose occurrence probability is less than the background threshold are filtered out. Therefore, when the background projection image is determined based on the background mask and the projection pattern, some pixels in the image area after the object projection image is removed can be filtered out, which can improve the accuracy of the acquired background projection image. In one implementation, the calculation formula of the background projection image can be:

[0098]

[0099] in, represents the background projection image corresponding to the jth projection pattern in the projection pattern sequence corresponding to the mth candidate image frame, It represents the jth projection pattern in the projection pattern sequence corresponding to the mth candidate image frame, and mask_back represents the background mask.

[0100] After obtaining the object projection image and the background projection image of the projection pattern, the projection matrix is ​​determined according to the object projection image and the background projection image. Figure 4As shown, the projection matrix can be formed by splicing the background projection image and the projection images of each object, wherein the projection matrix can be expressed as Where [·] represents matrix concatenation; P′ j m represents the projection matrix corresponding to the jth projection pattern in the projection pattern sequence corresponding to the mth candidate image frame, represents the background projection image corresponding to the jth projection pattern in the projection pattern sequence corresponding to the mth candidate image frame, The object projection image of the i-th moving object corresponding to the j-th projection pattern in the m-th candidate image frame is represented, and N represents the number of moving objects. In this embodiment, the projection pattern is converted into a projection matrix, so that the projection matrix can carry the position information of the moving object in the corresponding candidate image frame, thereby improving the image quality of the subsequently reconstructed target image.

[0101] After obtaining the projection matrices corresponding to each projection pattern, a projection matrix sequence can be obtained, wherein the number of projection matrices included in the projection matrix sequence is equal to the sum of the number of projection patterns included in the projection pattern sequence corresponding to each candidate image frame. In addition, the number of projection patterns included in the projection pattern sequence corresponding to each candidate image frame may be the same or different, and the same is taken as an example for explanation. When the number of projection patterns included in the projection pattern sequence corresponding to each candidate image frame is the same, the projection matrix in the projection matrix sequence can be expressed as:

[0102] Phi(k)=P′ j m

[0103] k=(m-1)*h+j

[0104] Wherein, Phi(k) represents the kth projection matrix; m represents the sequence number of the candidate image frame among several candidate image frames, j represents the sequence number of the projection pattern corresponding to the projection matrix in the projection pattern sequence corresponding to the mth candidate image frame; h represents the number of projection patterns included in the projection pattern sequence corresponding to the candidate image frame.

[0105] S40, reconstructing a target image based on each projection matrix in the projection matrix sequence and its corresponding bucket detector value.

[0106] Specifically, each projection matrix in the projection matrix sequence corresponds to a bucket detector value, and the bucket detector value is a value measured by the bucket detector, which is used to reflect the total light intensity when the projection pattern corresponding to the projection matrix is ​​projected onto the target scene. In other words, the bucket detector value is the measurement value measured by the bucket detector when the projection pattern corresponding to the projection matrix is ​​projected onto the target scene. In addition, when reconstructing the target image based on each projection matrix in the projection matrix sequence and its corresponding bucket detector value, the target image can be reconstructed by differential ghost imaging, a reconstruction method based on compressed sensing, a reconstruction method based on inverse Fourier transform, and an iterative reconstruction method.

[0107] In one implementation of this embodiment, Figure 5 As shown, the reconstructing the target image based on each projection matrix in the projection matrix sequence and its corresponding bucket detector value specifically includes:

[0108] Construct the initial target matrix;

[0109] The initial target matrix is ​​updated based on each projection matrix in the projection matrix sequence and its corresponding bucket detector value to obtain a candidate target matrix;

[0110] Detecting whether the candidate target matrix meets a preset condition;

[0111] When the candidate target matrix satisfies a preset condition, determining a target image based on the candidate target matrix;

[0112] When the candidate target matrix does not meet the preset conditions, the reconstructed image is used as the initial target matrix, and the step of determining the candidate target matrix for the initial target matrix based on each projection matrix in the projection matrix sequence and its corresponding bucket detector value is continued until the candidate target matrix meets the preset conditions.

[0113] Specifically, the initial target matrix can be randomly generated, and the initial target matrix can be used as a target matrix, and the initial target matrix can be updated iteratively to update the target matrix, and finally the target matrix can meet the preset conditions to obtain the target image. The image size of the initial target matrix is ​​equal to the image size of the reconstructed target image. In addition, after the initial target matrix is ​​randomly generated, the initial target matrix can be initialized, wherein the process of matrix initialization can be:

[0114] Object = ones(1:len(P′ j m )+1,1)

[0115] Among them, Object represents the initial target matrix, ones represents a matrix of all 1s and the matrix size is equal to the image size of the initial target matrix, and P′ jm represents the projection matrix corresponding to the j-th projection pattern in the projection pattern sequence corresponding to the m-th candidate image frame, and len(·) represents a function for obtaining the matrix length.

[0116] The preset condition is a pre-set basis for measuring whether the candidate target matrix can be used to generate the target image, wherein when the candidate target matrix meets the preset condition, it means that the candidate target matrix can be used to generate the target image, otherwise, when the candidate target matrix does not meet the preset condition, it means that the candidate target matrix cannot be used to generate the target image and needs to continue to iterate. Thus, when the candidate target matrix meets the preset condition, the target image is determined based on the candidate target matrix; when the candidate target matrix does not meet the preset condition, the reconstructed image is used as the initial target matrix and the iteration process is continued. In a typical implementation, the preset condition can be that the image difference between the candidate target matrix and the previous candidate target matrix is ​​less than a preset difference threshold, wherein the previous candidate target matrix is ​​the candidate target matrix obtained by the previous iteration of the candidate target matrix, and the preset difference value is pre-set, and when the image difference between the candidate target matrix and the previous candidate target matrix is ​​less than the preset difference threshold, it means that the candidate target matrix meets the iteration stop condition, indicating that the iteration process can be stopped, and thus the candidate reconstructed image can be used as the target image. Of course, in practical applications, the preset conditions may adopt other conditions, for example, the similarity between the candidate target matrix and the previous candidate target matrix is ​​greater than a preset similarity threshold, etc., which will not be explained one by one here.

[0117] Further, the candidate target matrix that meets the preset conditions includes the reconstructed area of ​​the moving object and the reconstructed area of ​​the background area, wherein the reconstructed area of ​​the moving object and the reconstructed area of ​​the background area can be respectively expressed as:

[0118] Object back =Object(1:Num_p)

[0119]

[0120] Where Num_p is the image size of the projected pattern, Indicates the position of the i-th moving object pixel in Object. back Represents the reconstruction area of ​​the background area, Object i represents the reconstructed region of the i-th moving object.

[0121] After determining the reconstruction area of ​​the moving object and the reconstruction area of ​​the background area, the candidate target matrix determines the target image by filling the pixels in the reconstruction area of ​​the moving object into the reconstruction area of ​​the background area in order based on the object mask and the background mask corresponding to the motion, so as to obtain the target image. Figure 6 As shown in the figure, when the moving object is moving at a uniform speed and includes occlusion, after 40 frames, the total number of measurements is 4096, and the background and foreground reconstruction process diagram can be seen. It can be seen that the target image reconstructed in this embodiment can clearly display the moving object and the background area, thereby ensuring the image quality of the target image.

[0122] In an implementation of this embodiment, the updating of the initial target matrix based on each projection matrix in the projection matrix sequence and its corresponding bucket detector value to obtain a candidate target matrix specifically includes:

[0123] For a first projection matrix in the projection matrix sequence, determining a first intermediate variable based on the projection matrix and the initial target matrix;

[0124] Determining a second intermediate variable based on the bucket detector value corresponding to the projection matrix and the first intermediate variable;

[0125] Update the initial target matrix based on the first projection matrix, the first intermediate variable and the second intermediate variable;

[0126] The updated initial target matrix is ​​used as the initial target matrix, and the step of determining the first intermediate variable by the projection matrix and the initial target matrix is ​​performed for the second projection matrix in the projection matrix sequence until the last projection matrix is ​​obtained to obtain a candidate target matrix.

[0127] Specifically, the first intermediate variable is determined based on the initial target matrix and the projection matrix, wherein the expression of the first intermediate variable is:

[0128] I pm =(Object) T *Phi(k)

[0129] Among them, I pm represents the first intermediate variable, Object represents the initial target matrix, (·) T represents matrix transpose, and Phi(k) represents the kth projection matrix in the projection matrix sequence.

[0130] After obtaining the first intermediate variable, the first intermediate variable may be updated based on the bucket detector value corresponding to the projection matrix to obtain a second intermediate variable, wherein the expression of the second intermediate variable is:

[0131] I pm ′=I pm -mean(I pm )+B(k) / numel(I pm )

[0132] Among them, I pm ' represents the second intermediate variable, I pm represents the first intermediate variable, B(k) represents the bucket detector value corresponding to the k-th projection matrix, mean represents the average value, and numel represents the number of pixels contained in the projection matrix.

[0133] After obtaining the first intermediate variable and the second intermediate variable, the initial target matrix can be updated based on the first intermediate variable and the second intermediate variable, wherein the update formula corresponding to the initial target matrix is:

[0134]

[0135] Among them, I pm ' represents the second intermediate variable, I pm Represents the first intermediate variable, Phi(k) represents the kth projection matrix in the projection matrix sequence, and Object represents the initial target matrix.

[0136] In summary, the present embodiment provides a dynamic single-pixel imaging method, the method comprising acquiring a number of candidate image frames of a target scene by a single-pixel imaging method; determining a background mask corresponding to the target scene based on an object mask corresponding to each candidate image frame in the number of candidate image frames; for each candidate image frame in the number of candidate image frames, determining a projection matrix corresponding to each projection pattern in a projection pattern sequence corresponding to the candidate image frame based on the background mask and the object mask corresponding to the candidate image frame, so as to obtain a projection matrix sequence; and reconstructing a target image based on each projection matrix in the projection matrix sequence and its corresponding bucket detector value. The present application determines the background mask of the target scene based on the frequency of occurrence of each background pixel in the background area in all candidate image frames, and then determines the background projection image of each projection image based on the background mask, so that the background area with low occurrence frequency in the projection pattern can be filtered out, and the accuracy of the object area in the projection matrix can be improved, thereby improving the image quality of the target image reconstructed based on the projection matrix and the bucket detector value.

[0137] Based on the above-mentioned dynamic single pixel imaging method, this embodiment provides a dynamic single pixel imaging device, such as Figure 7 As shown, the device comprises:

[0138] An acquisition module 100 acquires a plurality of candidate image frames of a target scene by using a single-pixel imaging method;

[0139] A first determination module 200 is used to determine a background mask corresponding to the target scene based on an object mask corresponding to each of the candidate image frames in the plurality of candidate image frames;

[0140] A second determination module 300 is used to determine, for each candidate image frame among the plurality of candidate image frames, a projection matrix corresponding to each projection pattern in a projection pattern sequence corresponding to the candidate image frame based on the background mask and the object mask corresponding to the candidate image frame, so as to obtain a projection matrix sequence;

[0141] The reconstruction module 400 is used to reconstruct the target image based on each projection matrix in the projection matrix sequence and its corresponding bucket detector value.

[0142] Based on the above-mentioned dynamic single-pixel imaging method, this embodiment provides a computer-readable storage medium, which stores one or more programs. The one or more programs can be executed by one or more processors to implement the steps in the dynamic single-pixel imaging method as described in the above-mentioned embodiment.

[0143] Based on the above dynamic single pixel imaging method, the present application also provides a terminal device, such as Figure 8 As shown, it includes at least one processor (processor) 20; display screen 21; and memory (memory) 22, and may also include a communications interface (Communications Interface) 23 and a bus 24. Among them, the processor 20, the display screen 21, the memory 22 and the communication interface 23 can communicate with each other through the bus 24. The display screen 21 is configured to display a preset user guide interface in the initial setting mode. The communication interface 23 can transmit information. The processor 20 can call the logic instructions in the memory 22 to execute the method in the above embodiment.

[0144] In addition, the logic instructions in the memory 22 can be implemented in the form of software functional units and can be stored in a computer-readable storage medium when sold or used as an independent product.

[0145] The memory 22 is a computer-readable storage medium that can be configured to store software programs, computer executable programs, such as program instructions or modules corresponding to the methods in the embodiments of the present disclosure. The processor 20 executes functional applications and data processing by running the software programs, instructions or modules stored in the memory 22, that is, implementing the methods in the above embodiments.

[0146] The memory 22 may include a program storage area and a data storage area, wherein the program storage area may store an operating system and at least one application required for a function; the data storage area may store data created according to the use of the terminal device, etc. In addition, the memory 22 may include a high-speed random access memory and may also include a non-volatile memory. For example, a variety of media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a disk or an optical disk, may also be a transient storage medium.

[0147] In addition, the specific process of loading and executing the multiple instruction processors in the above-mentioned storage medium and the terminal device has been described in detail in the above-mentioned method, and will not be described one by one here.

[0148] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit it. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A dynamic single pixel imaging method, characterized in that: The method includes: A single-pixel imaging method is used to acquire several candidate image frames of the target scene; Determining a background mask corresponding to the target scene based on an object mask corresponding to each of the candidate image frames; For each candidate image frame among the plurality of candidate image frames, based on the background mask and the object mask corresponding to the candidate image frame, determine the projection matrix corresponding to each projection pattern in the projection pattern sequence corresponding to the candidate image frame, so as to obtain a projection matrix sequence; Reconstructing a target image based on each projection matrix in the projection matrix sequence and its corresponding bucket detector value; The determining of the background mask corresponding to the target scene based on the object mask corresponding to each of the candidate image frames in the plurality of candidate image frames specifically includes: For each candidate image frame among the plurality of candidate image frames, determining a candidate background mask corresponding to the candidate image frame; Calculating the pixel mean of each pixel position in all the determined candidate background masks, and determining the reference background mask based on all the determined pixel means; Determining a background mask corresponding to the target scene based on a preset background threshold and the reference background mask; The step of determining, for each candidate image frame among the plurality of candidate image frames, based on the background mask and the object mask corresponding to the candidate image frame, a projection matrix corresponding to each projection pattern in a projection pattern sequence corresponding to the candidate image frame to obtain a projection matrix sequence specifically includes: For each candidate image frame among the plurality of candidate image frames, determining, based on an object mask corresponding to the candidate image frame, an object projection image corresponding to each projection pattern in a projection pattern sequence corresponding to the candidate image frame; Determine the background projection images corresponding to the projection patterns based on the object projection images corresponding to the projection patterns and the background mask; Based on the object projection image and the background projection image corresponding to each projection pattern, the projection matrix corresponding to each projection pattern is determined to obtain a projection matrix sequence.

2. The dynamic single pixel imaging method according to claim 1, characterized in that: The target scene includes a plurality of moving objects, and each of the plurality of candidate image frames includes each of the plurality of moving objects.

3. The dynamic single pixel imaging method according to claim 1, characterized in that: The reconstructing the target image based on each projection matrix in the projection matrix sequence and its corresponding bucket detector value specifically includes: Construct the initial target matrix; The initial target matrix is ​​updated based on each projection matrix in the projection matrix sequence and its corresponding bucket detector value to obtain a candidate target matrix; Detecting whether the candidate target matrix meets a preset condition; When the candidate target matrix satisfies a preset condition, determining a target image based on the candidate target matrix; When the candidate target matrix does not meet the preset conditions, the reconstructed image is used as the initial target matrix, and the step of determining the candidate target matrix for the initial target matrix based on each projection matrix in the projection matrix sequence and its corresponding bucket detector value is continued until the candidate target matrix meets the preset conditions.

4. The dynamic single pixel imaging method according to claim 3, characterized in that: The updating of the initial target matrix based on each projection matrix in the projection matrix sequence and its corresponding bucket detector value to obtain a candidate target matrix specifically includes: For a first projection matrix in the projection matrix sequence, determining a first intermediate variable based on the projection matrix and the initial target matrix; Determining a second intermediate variable based on the bucket detector value corresponding to the projection matrix and the first intermediate variable; Update the initial target matrix based on the first projection matrix, the first intermediate variable and the second intermediate variable; The updated initial target matrix is ​​used as the initial target matrix, and the step of determining the first intermediate variable by the projection matrix and the initial target matrix is ​​performed for the second projection matrix in the projection matrix sequence until the last projection matrix is ​​obtained to obtain a candidate target matrix.

5. The dynamic single pixel imaging method according to claim 4, characterized in that: The expression of the first intermediate variable is: I pm =(Object) T *Phi(k) Among them, I pm represents the first intermediate variable, Object represents the initial target matrix, (g) T represents matrix transpose, and Phi(k) represents the kth projection matrix in the projection matrix sequence.

6. The dynamic single pixel imaging method according to claim 5, characterized in that: The expression of the second intermediate variable is: I pm ′=I pm -mean(I pm )+B(k) / numel(I pm ) Among them, I pm ' represents the second intermediate variable, I pm represents the first intermediate variable, B(k) represents the bucket detector value corresponding to the k-th projection matrix, mean represents the average value, and numel represents the number of pixels contained in the projection matrix.

7. The dynamic single pixel imaging method according to claim 4, characterized in that: The update formula corresponding to the initial target matrix is: Among them, I pm ' represents the second intermediate variable, I pm Represents the first intermediate variable, Phi(k) represents the kth projection matrix in the projection matrix sequence, and Object represents the initial target matrix.

8. The dynamic single pixel imaging method according to claim 3, characterized in that: The preset condition is that the image difference between the candidate target matrix and the previous candidate target matrix is ​​less than a preset difference threshold.

9. A dynamic single pixel imaging device, characterized in that: The device comprises: An acquisition module, which acquires several candidate image frames of the target scene using a single-pixel imaging method; A first determination module is used to determine a background mask corresponding to the target scene based on an object mask corresponding to each of the candidate image frames in the plurality of candidate image frames; A second determination module is used to determine, for each candidate image frame among the plurality of candidate image frames, a projection matrix corresponding to each projection pattern in a projection pattern sequence corresponding to the candidate image frame based on the background mask and the object mask corresponding to the candidate image frame, so as to obtain a projection matrix sequence; A reconstruction module, used for reconstructing a target image based on each projection matrix in the projection matrix sequence and its corresponding bucket detector value; The determining of the background mask corresponding to the target scene based on the object mask corresponding to each of the candidate image frames in the plurality of candidate image frames specifically includes: For each candidate image frame among the plurality of candidate image frames, determining a candidate background mask corresponding to the candidate image frame; Calculating the pixel mean of each pixel position in all the determined candidate background masks, and determining the reference background mask based on all the determined pixel means; Determining a background mask corresponding to the target scene based on a preset background threshold and the reference background mask; The step of determining, for each candidate image frame among the plurality of candidate image frames, based on the background mask and the object mask corresponding to the candidate image frame, a projection matrix corresponding to each projection pattern in a projection pattern sequence corresponding to the candidate image frame to obtain a projection matrix sequence specifically includes: For each candidate image frame among the plurality of candidate image frames, determining, based on an object mask corresponding to the candidate image frame, an object projection image corresponding to each projection pattern in a projection pattern sequence corresponding to the candidate image frame; Determine the background projection images corresponding to the projection patterns based on the object projection images corresponding to the projection patterns and the background mask; Based on the object projection image and the background projection image corresponding to each projection pattern, the projection matrix corresponding to each projection pattern is determined to obtain a projection matrix sequence.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores one or more programs, and the one or more programs can be executed by one or more processors to implement the steps in the dynamic single-pixel imaging method as described in any one of claims 1-8.

11. A terminal device, characterized in that: include: Processor, memory and communication bus; The memory stores a computer-readable program executable by the processor; The communication bus realizes the connection and communication between the processor and the memory; When the processor executes the computer-readable program, the steps in the dynamic single-pixel imaging method as described in any one of claims 1-8 are implemented.

Images