Photometric stereo image synthesis method, system, device, equipment and storage medium

Abstract

The present invention discloses a photometric stereo image synthesis method, system, device, equipment and storage medium. The method comprises: obtaining a moving pulse signal obtained by a trigger input circuit to collect a measured object; performing signal processing on the moving pulse signal to obtain a collection drive signal and a light source drive signal; sending the light source drive signal to the light source drive circuit, so that the light source drive circuit generates at least two light source angles of the light source brightness irradiating the measured object according to the light source drive signal; sending the collection drive signal to the image sensor, so that the image sensor performs image collection of the measured object under at least two light source angles according to the collection drive signal, and obtains and feeds back at least two images of the measured object under different light source angles; generating at least one photometric stereo synthesis image for representing the image stereo information based on the image grayscale information of the measured object images under different light source angles and a preset image decomposition method.

Description

Technical Field

[0001] The present invention relates to the field of image data processing technology, and in particular to a photometric stereo image synthesis method, system, device, equipment and storage medium. Background Art

[0002] Stereoscopic image processing technology is an important computer vision technology that can help people better understand and process stereoscopic images and provide important technical support for the realization of various applications.

[0003] Existing systems for stereo image detection are mainly composed of one or more cameras, zoom lenses and light sources. The system integration is poor. In wide scenes, multiple line scan cameras are needed to stitch the captured images. The on-site installation accuracy is high, and the image consistency and anti-vibration ability are poor. In addition, the traditional photometric stereo synthesis method requires the provision of light source direction and angle information as prior knowledge, and requires the input of three or more light source directions to achieve synthesis. Since the synthesis effect is very poor when the input light source information does not accurately correspond to the actual imaging system state, the traditional photometric stereo algorithm needs to perform a series of preprocessing operations such as light source calibration before execution.

[0004] Therefore, the existing stereo image detection system has a complex system composition, low detection accuracy and efficiency, and the existing photometric stereo synthesis algorithm has a cumbersome operation process and low synthesis quality and stability. Summary of the invention

[0005] The present invention provides a photometric stereo image synthesis method, system, device, equipment and storage medium to improve the synthesis quality and stability of photometric stereo images, simplify the operation flow in the image synthesis process, and improve the image synthesis efficiency.

[0006] According to one aspect of the present invention, a photometric stereo image synthesis method is provided, which is applied to a photometric stereo image synthesis system, the system comprising: a processor, a trigger input circuit, an image sensor and a light source driving circuit; wherein the processor is respectively connected to the trigger input circuit, the image sensor and the light source driving circuit for communication; the method is executed by the processor, and comprises:

[0007] Acquire the movement pulse signal obtained by the trigger input circuit collecting the measured object;

[0008] Performing signal processing on the moving pulse signal to obtain a collection drive signal and a light source drive signal;

[0009] Sending the light source driving signal to the light source driving circuit, so that the light source driving circuit generates at least two light source angles of the light source brightness irradiating the measured object according to the light source driving signal;

[0010] The acquisition driving signal is sent to the image sensor, so that the image sensor can acquire images of the object under at least two light source angles according to the acquisition driving signal, and obtain and feed back at least two images of the object under different light source angles;

[0011] At least one photometric stereo composite image for representing the stereo information of the image is generated according to the image grayscale information of the image of the measured object under the different light source angles.

[0012] According to another aspect of the present invention, a photometric stereo image synthesis system includes: a processor, a trigger input circuit, an image sensor, and a light source driving circuit; wherein the processor is communicatively connected with the trigger input circuit, the image sensor, and the light source driving circuit respectively;

[0013] The trigger input circuit is used to collect the movement pulse signal of the object under test and send the movement pulse signal to the processor;

[0014] The processor is used to perform signal processing on the acquired moving pulse signal to obtain an acquisition driving signal and a light source driving signal, and send the light source driving signal to the light source driving circuit, and send the acquisition driving signal to the image sensor;

[0015] The light source driving circuit is used to generate at least two light source angles of the light source brightness irradiating the object to be measured according to the light source driving signal;

[0016] The image sensor is used to collect images of the object under test under at least two light source angles according to the acquisition drive signal, and obtain and feed back at least two images of the object under test under different light source angles;

[0017] The processor is further used to generate at least one photometric stereo composite image for representing the stereo information of the image according to the image grayscale information of the image of the measured object under each of the different light source angles.

[0018] According to another aspect of the present invention, a photometric stereo image synthesis device is provided, which is applied to a photometric stereo image synthesis system, the system comprising: a processor, a trigger input circuit, an image sensor and a light source driving circuit; wherein the processor is respectively connected to the trigger input circuit, the image sensor and the light source driving circuit for communication; the device is configured on the processor, comprising:

[0019] A pulse signal acquisition module, used to acquire the moving pulse signal obtained by the trigger input circuit collecting the detected object;

[0020] A signal processing module, used for performing signal processing on the moving pulse signal to obtain a collection driving signal and a light source driving signal;

[0021] A light source driving signal sending module, used for sending the light source driving signal to the light source driving circuit, so that the light source driving circuit generates at least two light source angles for the light source brightness irradiating the measured object according to the light source driving signal;

[0022] An acquisition drive signal sending module is used to send the acquisition drive signal to the image sensor, so that the image sensor can acquire images of the object under at least two light source angles according to the acquisition drive signal, and obtain and feed back at least two images of the object under different light source angles;

[0023] The stereo image synthesis module is used to generate at least one photometric stereo synthesis image for representing the stereo information of the image according to the image grayscale information of the image of the measured object under the different light source angles.

[0024] According to another aspect of the present invention, an electronic device is provided, the electronic device comprising:

[0025] at least one processor; and

[0026] a memory communicatively connected to the at least one processor; wherein,

[0027] The memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute the photometric stereo image synthesis method described in any embodiment of the present invention.

[0028] According to another aspect of the present invention, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the photometric stereo image synthesis method described in any embodiment of the present invention when executed.

[0029] The technical solution of the embodiment of the present invention solves the problem that multiple line scan cameras need to be used to stitch the captured images in wide scenes, which has high requirements for on-site installation accuracy and poor image consistency and anti-vibration ability, and simplifies the system composition structure. At least one photometric stereo synthesis image for representing the stereo information of the image is generated according to the image grayscale information of the image of the measured object under different light source angles. During the generation process, there is no need to calibrate the light source angle, nor is there a need to limit whether there is a linear relationship between the light source angles. The image pixel data in different dimensional directions can be obtained by processing the data of different light source angles in a preset image decomposition method, thereby obtaining at least one photometric stereo synthesis image for representing the stereo information of the image, thereby improving the synthesis quality and stability of the photometric stereo image, simplifying the operation process during the image synthesis process, and improving the image synthesis efficiency.

[0030] It should be understood that the contents described in this section are not intended to identify the key or important features of the embodiments of the present invention, nor are they intended to limit the scope of the present invention. Other features of the present invention will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0032] Figure 1A is a structural schematic diagram of a photometric stereo image synthesis system provided according to Embodiment 1 of the present invention;

[0033] Figure 1B is a structural schematic diagram of a photometric stereo image synthesis system provided according to Embodiment 1 of the present invention;

[0034] Figure 1C is a flow chart of a photometric stereo image synthesis method provided according to the first embodiment of the present invention;

[0035] Figure 2 is a flow chart of a photometric stereo image synthesis method provided according to the second embodiment of the present invention;

[0036] Figure 3 is a structural schematic diagram of a photometric stereo image synthesis device provided according to Embodiment 3 of the present invention;

[0037] Figure 4 It is a structural schematic diagram of an electronic device for implementing the photometric stereo image synthesis method according to an embodiment of the present invention. DETAILED DESCRIPTION

[0038] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0039] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0040] Embodiment 1

[0041] Figure 1A A structural schematic diagram of a photometric stereo image synthesis system provided in Embodiment 1 of the present invention, the system comprising: a processor 10, a trigger input circuit 11, an image sensor 12 and a light source driving circuit 13; wherein the processor 10 is respectively communicatively connected with the trigger input circuit 11, the image sensor 12 and the light source driving circuit 13.

[0042] The trigger input circuit 11 is used to collect the moving pulse signal of the object under test and send the moving pulse signal to the processor 10. The processor 10 is used to perform signal processing on the obtained moving pulse signal to obtain a collection drive signal and a light source drive signal, and send the light source drive signal to the light source drive circuit 13, and send the collection drive signal to the image sensor 12. The light source drive circuit 13 is used to generate at least two light source angles of the light source brightness illuminating the object under test according to the light source drive signal. The image sensor 12 is used to collect images of the object under test at at least two light source angles according to the collection drive signal, and obtain and feedback at least two images of the object under test at different light source angles. The processor 10 is also used to generate at least one photometric stereo composite image for characterizing the stereo information of the image according to the image grayscale information of the image of the object under test at different light source angles.

[0043] Figure 1BA schematic diagram of the structure of a photometric stereo image synthesis system provided in Embodiment 1 of the present invention. As shown in the figure, the photometric stereo image synthesis system also includes at least one light source device 14, which can realize different light source combinations and generate light sources at different angles under the control of the light source driving circuit. Specifically, each light source device 14 is physically connected to the light source driving circuit 13; the light source driving circuit 13 is used to determine the number of light source devices connected to itself for communication, and according to the number of devices, determine the light source stroboscopic combination at different light source angles, so as to realize the illumination of the object under test at different light source angles.

[0044] The photometric stereo image synthesis system also includes a self-focusing lens array lens 15. The self-focusing lens array lens 15 can be physically connected to the image sensor 12. The image sensor 12 can collect image information of the object to be measured after the object to be measured is imaged by the self-focusing lens array lens 15. The image sensor 12 may include a plurality of CMOS (Complementary Metal-Oxide-Semiconductor) sensors 121; the CMOS sensor 121 is used to collect images of the object to be measured, obtain reference image information, and send the reference image information as the image information of the object to be measured to the processor 10. Among them, the number of CMOS sensors 121 in the image sensor 12 can be pre-set by relevant technical personnel based on actual experience values ​​or experimental values. For example, the number of CMOS sensors 121 can be 20.

[0045] The photometric stereo image synthesis system further includes a 10 Gigabit Ethernet 16. The 10 Gigabit Ethernet 16 is connected to the processor 10 for communication; the processor 10 performs image processing on the received image information of the measured object to obtain an image processing result, and can transmit the image processing result through the 10 Gigabit Ethernet 16 for data transmission, so as to realize the rapid transmission of image data from the processor to the background or back end.

[0046] Figure 1C This is a flowchart of a photometric stereo image synthesis method provided in the first embodiment of the present invention. This embodiment is applicable to the case of performing photometric stereo image synthesis on images of a measured object under different light source angles. The method can be executed by a photometric stereo image synthesis device. The photometric stereo image synthesis device can be implemented in the form of hardware and / or software. The photometric stereo image synthesis device can be configured in an electronic device, for example, in a processor in a photometric stereo image synthesis system. Figure 1C As shown, the method includes:

[0047] S110, acquiring a movement pulse signal obtained by the trigger input circuit collecting the measured object.

[0048] The object to be measured may be any object whose image is to be captured; and the moving pulse signal is used to capture and receive the moving pulse signal of the object to be measured.

[0049] S120, performing signal processing on the moving pulse signal to obtain a collection driving signal and a light source driving signal.

[0050] Exemplarily, the processor can calculate the real-time speed of the object under test according to the moving pulse signal, and thus generate a collection drive signal and a light source drive signal according to the real-time speed.

[0051] S130, sending a light source driving signal to a light source driving circuit, so that the light source driving circuit generates at least two light source angles for illuminating the measured object with light source brightness according to the light source driving signal.

[0052] Exemplarily, the processor sends a light source driving signal to the light source driving circuit, and the light source driving circuit can control at least one light source to generate light source brightness at at least two light source angles according to the light source driving signal, thereby illuminating the object to be measured with the light source brightness at at least two light source angles.

[0053] S140, sending an acquisition driving signal to the image sensor, so that the image sensor can acquire images of the object under at least two light source angles according to the acquisition driving signal, and obtain and feed back at least two images of the object under different light source angles.

[0054] Exemplarily, the processor sends an acquisition drive signal to the image sensor. After receiving the acquisition drive signal, the image sensor acquires images of the object under test at at least two different light source angles according to the acquisition drive signal, and sends the acquired images of the object under test at at least two light source angles to the processor for processing.

[0055] S150 , generating at least one photometric stereo composite image for representing stereo information of the image according to the image grayscale information of the image of the object under different light source angles and based on a preset image decomposition method.

[0056] It should be noted that, when the processor is synthesizing the stereo image of the measured object, the measured object image does not need to be calibrated with the light source direction and angle information.

[0057] Exemplarily, at least one photometric stereo synthesis image for characterizing the stereo information of the image can be generated based on the pre-trained photometric stereo synthesis model according to the image grayscale information of the image of the object under test at different light source angles. The photometric stereo synthesis model can be pre-trained. Specifically, the image grayscale information of the historical object images taken at different light source angles in the historical time period can be obtained, and used as a sample training data set to train the preset network model until the model converges to obtain the photometric stereo synthesis model. The network model can be pre-constructed by relevant technical personnel, or an existing network model can be used, such as VSN (View Synthesis Networks).

[0058] It is understandable that in order to simplify the synthesis process of photometric stereo images without using a large amount of historical image data under different light source angles as samples for model training, as well as subsequent maintenance and optimization of the model, the following embodiment can be used for photometric stereo image synthesis.

[0059] In an optional embodiment, at least one photometric stereo composite image for representing image stereo information is generated based on image grayscale information of the images of the objects under test at different light source angles, including: generating an image intensity matrix based on a preset image decomposition method according to the image grayscale information of each image of the objects under test; performing singular value decomposition on the image intensity matrix to obtain a left singular matrix and a singular value matrix; generating a gradient composite image for representing image gradient information based on the left singular matrix and the singular value matrix; and generating a photometric stereo composite image including the gradient composite image.

[0060] Exemplarily, the image pixels of each measured object image can be re-stitched to obtain an image intensity matrix. For example, if the image sizes of each measured object image are the same, and the number of image pixels is P, and the number of measured object images is F, then the F measured object images are spliced ​​to generate a composite image I of size P*F, which is converted into a matrix form to be the image intensity matrix I.

[0061] Among them, the image decomposition method can be a singular value decomposition method. Exemplarily, SVD (Singular Value Decomposition) is performed on the image intensity matrix to obtain a left singular matrix U and a singular value matrix ∑. According to the left singular matrix and the singular value matrix, a pseudo surface normal vector matrix for representing different directions in three-dimensional space can be generated, and according to the pseudo surface normal vector field matrix, a gradient image in the X direction and the Y direction is generated, and the image pixels of the gradient image in the X direction and the gradient image in the Y direction are summed as a gradient composite image.

[0062] When the light source angle is unknown, the light source information and the surface image information of the object under test can be transformed into a bilinear constraint problem. By performing singular value decomposition on the image intensity matrix, the gradient information of each pixel point in the three coordinate directions in the three-dimensional space dimension can be determined, thereby generating a gradient composite image, simplifying the image synthesis operation process and improving the image synthesis efficiency and synthesis accuracy.

[0063] The technical solution of the embodiment of the present invention solves the problem that multiple line scan cameras need to be used to stitch the captured images in wide scenes, which has high requirements for on-site installation accuracy and poor image consistency and anti-vibration ability, and simplifies the system composition structure. At least one photometric stereo synthesis image for representing the stereo information of the image is generated according to the image grayscale information of the image of the measured object under different light source angles. During the generation process, there is no need to calibrate the light source angle, nor is there a need to limit whether there is a linear relationship between the light source angles. The image pixel data in different dimensional directions can be obtained by processing the data of different light source angles in a preset image decomposition method, thereby obtaining at least one photometric stereo synthesis image for representing the stereo information of the image, thereby improving the synthesis quality and stability of the photometric stereo image, simplifying the operation process during the image synthesis process, and improving the image synthesis efficiency.

[0064] Embodiment 2

[0065] Figure 2 This is a flow chart of a photometric stereo image synthesis method provided in the second embodiment of the present invention. This embodiment is optimized and improved on the basis of the above technical solutions.

[0066] Furthermore, the step of "generating at least one photometric stereo composite image for representing the image stereo information based on the image grayscale information of the image of the object under test at different light source angles and based on a preset image decomposition method" is refined into "generating an image intensity matrix based on the image grayscale information of each image of the object under test; performing singular value decomposition on the image intensity matrix to obtain a left singular matrix and a singular value matrix; generating a gradient composite image for representing the image gradient information based on the left singular matrix and the singular value matrix; generating a photometric stereo composite image including the gradient composite image." to improve the method for generating photometric stereo composite images.

[0067] The step "generating a gradient composite image for representing the image gradient information according to the left singular matrix and the singular value matrix" is refined into "generating a pseudo-surface normal vector field matrix according to the left singular matrix and the singular value matrix; generating first direction shape information and second direction shape information according to the pseudo-surface normal vector field matrix; generating a gradient composite image for representing the image gradient information according to the first direction shape information and the second direction shape information." to improve the generation method of the gradient composite image.

[0068] It should be noted that for the parts not described in detail in the embodiments of the present invention, reference can be made to the descriptions of other embodiments. Figure 2 As shown, the method comprises the following specific steps:

[0069] S210, acquiring a movement pulse signal obtained by the trigger input circuit collecting the detected object.

[0070] S220: Process the moving pulse signal to obtain a collection driving signal and a light source driving signal.

[0071] S230: Send the light source driving signal to the light source driving circuit, so that the light source driving circuit generates at least two light source angles for illuminating the measured object with light source brightness according to the light source driving signal.

[0072] S240, sending an acquisition driving signal to the image sensor, so that the image sensor can acquire images of the object under at least two light source angles according to the acquisition driving signal, and obtain and feed back at least two images of the object under different light source angles.

[0073] S250 , generating an image intensity matrix according to the image grayscale information of each measured object image.

[0074] S260 , performing singular value decomposition on the image intensity matrix to obtain a left singular matrix and a singular value matrix.

[0075] S270. Generate a pseudo surface normal vector field matrix according to the left singular matrix and the singular value matrix.

[0076] According to the left singular matrix U and the singular value matrix ∑, a pseudo surface normal vector field matrix S is generated. Specifically, the pseudo surface normal vector field matrix S is generated as follows:

[0077]

[0078] S280 . Generate first direction shape information and second direction shape information according to the pseudo surface normal vector field matrix.

[0079] Exemplarily, the shape information of each pixel position in the first direction, that is, the X direction in the three-dimensional space, and the shape information of each pixel in the second direction, that is, the Y direction in the three-dimensional space, can be determined based on the pseudo-surface normal vector field matrix. For example, if the pseudo-surface normal vector field matrix is ​​a vector matrix of F*P dimensions, using existing decomposition techniques such as eigenvalue decomposition, a vector matrix of a decomposed matrix size of P*3 can be obtained, wherein each column in the P*3 vector matrix represents the shape information in the X direction, the Y direction, and the Z direction, respectively, and is represented in the form of a matrix of size P*1.

[0080] S290 . Generate a gradient composite image for representing image gradient information according to the first directional shape information and the second directional shape information.

[0081] Exemplarily, the gradient information in different dimensional directions can be determined according to the first direction shape information and the second direction information based on the convolution kernel in the first direction (X direction) and the convolution kernel in the second direction (Y direction).

[0082] In an optional embodiment, a gradient composite image for representing image gradient information is generated based on first directional shape information and second directional shape information, including: generating first directional gradient information based on the first directional shape information and a first directional convolution kernel; and generating second directional gradient information based on the second directional shape information and the second directional convolution kernel; generating a gradient composite image for representing image gradient information based on the first directional gradient information and the second directional gradient information.

[0083] The first direction convolution kernel, i.e., the X direction convolution kernel, and the second direction convolution kernel, i.e., the Y direction convolution kernel, can be preset by relevant technicians according to actual needs. For example, the convolution kernels of the X direction convolution kernel wx and the Y direction convolution kernel wy are set as follows:

[0084] wx=[-1,0,1]

[0085]

[0086] The matrix of the first direction shape information is multiplied by the first direction convolution kernel to obtain the first direction gradient information, which is also expressed in the form of a matrix; the matrix of the second direction shape information is multiplied by the second direction convolution kernel to obtain the second direction gradient information, which is also expressed in the form of a matrix. The first direction gradient information (matrix) and the second direction gradient information (matrix) are summed to obtain a gradient composite image for representing the image gradient information.

[0087] It should be noted that different reference plane gradient composite images can be generated based on different reference planes, wherein the reference plane can be preset by relevant technical personnel, for example, it can be 0 or 128.

[0088] S2100 , generate a photometric stereo composite image including a gradient composite image.

[0089] It should be noted that the photometric stereo composite image used to represent the stereo information of the image may include a height composite image in addition to the gradient composite image.

[0090] In an optional embodiment, a pixel composite image is generated based on the image grayscale information of each measured object image; for any pixel point in the pixel composite image, a unit vector between the pixel point and its two adjacent direction pixels is determined; based on a preset vector association relationship, a height composite image for representing the image height information is generated based on the surface normal vector of the pixel point and each unit vector; and a photometric stereo composite image including the height composite image is generated.

[0091] Among them, exemplarily, image stitching can be performed according to the image grayscale information of each measured object image to obtain a pixel composite image. For details, reference can be made to the generation method of the aforementioned image intensity matrix, which is the matrix expression form of the pixel composite image.

[0092] Exemplarily, for any pixel point in the pixel synthesis image, the unit vector between the pixel point and its two adjacent pixel points is determined. For example, if the pixel point is (x, y), the two adjacent pixel points can be (x+1, y) and (x, y+1) respectively.

[0093] Specifically, the coordinate point of another pixel point (x, y) in the third direction (Z-axis direction) is z x,y , the unit vector between the pixel point (x, y) and its two adjacent pixels (x+1, y) and (x, y+1) and They are as follows:

[0094]

[0095]

[0096] The surface normal vector of the pixel point (x, y) is (n x ,n y ,n z ), then the vector relationship between the pixel point (x, y) and its two adjacent pixel points (x+1, y) and (x, y+1) is as follows:

[0097]

[0098] According to the above vector relationship, the following set of equations can be sorted and constructed:

[0099]

[0100] The conjugate gradient least squares method can be used to solve the equation and finally obtain the height information z of the pixel point (x, y) x,y , so that the height information of any pixel on the surface of the object can be determined and a highly synthesized image with height information can be obtained.

[0101] It should be noted that the photometric stereo composite image used to represent the stereo information of the image may include a height composite image in addition to the gradient composite image and the height composite image.

[0102] In an optional embodiment, third direction shape information is generated according to the pseudo-surface normal vector field matrix; a color composite image is generated based on a preset color image channel according to the first direction shape information, the second direction shape information and the third direction information; and a photometric stereo composite image including the color composite image is generated.

[0103] Among them, the third direction shape information, that is, the shape information in the Z direction in three-dimensional space, can be obtained by decomposing the pseudo-surface normal vector field matrix, which is the same as the above-mentioned method of determining the second direction shape information and the third direction shape information, and will not be repeated in this embodiment.

[0104] Exemplarily, the determined first direction shape information, second direction shape information and third direction information are respectively placed into preset color image channels, wherein the color image channels can be preset by relevant technical personnel, for example, the color image channels can be RGB (Red, Green, Blue) color channels. Specifically, the first direction (X direction) shape information can be placed into the R color channel, the second direction (Y direction) shape information can be placed into the G color channel, and the third direction (Z direction) shape information can be placed into the B color channel, thereby obtaining a color composite image.

[0105] The technical solution of this embodiment generates an image intensity matrix according to the image grayscale information of each measured object image; performs singular value decomposition on the image intensity matrix to obtain a left singular matrix and a singular value matrix; generates a gradient synthesis image for representing the image gradient information according to the left singular matrix and the singular value matrix; and generates a photometric stereo synthesis image including the gradient synthesis image. The above technical solution processes each measured object image under different light source angles by adopting singular value decomposition to obtain at least one photometric stereo synthesis image for representing the image stereo information, without the need to input the light source direction and angle information, breaking the limitation of the number of images with at least three different lighting directions required in the traditional photometric stereo algorithm. For the light sources in the line scan system, even if they are integrated parallel to the camera, the traditional photometric stereo algorithm can be applied for image synthesis, without considering the linear relationship between the light source angles, thereby improving the synthesis quality and stability of the photometric stereo image, simplifying the operation flow in the image synthesis process, and improving the image synthesis efficiency.

[0106] Embodiment 3

[0107] Figure 3The structure diagram of a photometric stereo image synthesis device provided in the third embodiment of the present invention. The photometric stereo image synthesis device provided in the embodiment of the present invention can be applied to the case of performing photometric stereo image synthesis on images of a measured object under different light source angles. The photometric stereo image synthesis device can be implemented in the form of hardware and / or software, such as Figure 3 As shown, the device specifically includes: a pulse signal acquisition module 301, a signal processing module 302, a light source driving signal sending module 303, a collection driving signal sending module 304 and a stereo image synthesis module 305. Among them,

[0108] The pulse signal acquisition module 301 is used to acquire the movement pulse signal obtained by the trigger input circuit collecting the detected object;

[0109] A signal processing module 302 is used to perform signal processing on the moving pulse signal to obtain a collection drive signal and a light source drive signal;

[0110] A light source driving signal sending module 303 is used to send the light source driving signal to the light source driving circuit, so that the light source driving circuit generates at least two light source angles for the light source brightness irradiating the measured object according to the light source driving signal;

[0111] The acquisition driving signal sending module 304 is used to send the acquisition driving signal to the image sensor, so that the image sensor can acquire images of the object under at least two light source angles according to the acquisition driving signal, and obtain and feed back at least two images of the object under different light source angles;

[0112] The stereo image synthesis module 305 is used to generate at least one photometric stereo synthesis image for representing the stereo information of the image according to the image grayscale information of the image of the object under test at each of the different light source angles.

[0113] The technical solution of the embodiment of the present invention solves the problem that multiple line scan cameras need to be used to stitch the captured images in wide scenes, which has high requirements for on-site installation accuracy and poor image consistency and anti-vibration ability, and simplifies the system composition structure. At least one photometric stereo synthesis image for representing the stereo information of the image is generated according to the image grayscale information of the image of the measured object under different light source angles. During the generation process, there is no need to calibrate the light source angle, nor is there a need to limit whether there is a linear relationship between the light source angles. The image pixel data in different dimensional directions can be obtained by processing the data of different light source angles in a preset image decomposition method, thereby obtaining at least one photometric stereo synthesis image for representing the stereo information of the image, thereby improving the synthesis quality and stability of the photometric stereo image, simplifying the operation process during the image synthesis process, and improving the image synthesis efficiency.

[0114] Optionally, the stereoscopic image synthesis module 305 includes:

[0115] An image intensity matrix generating unit, used for generating an image intensity matrix according to the image grayscale information of each of the images of the measured objects;

[0116] A singular value decomposition unit, used for performing singular value decomposition on the image intensity matrix to obtain a left singular matrix and a singular value matrix;

[0117] A gradient synthesis image generating unit, used for generating a gradient synthesis image for representing image gradient information according to the left singular matrix and the singular value matrix;

[0118] The stereo image synthesis unit is used to generate a photometric stereo synthesis image including the gradient synthesis image.

[0119] Optionally, the gradient composite image generating unit includes:

[0120] A normal vector field matrix generating subunit, used for generating a pseudo surface normal vector field matrix according to the left singular matrix and the singular value matrix;

[0121] A shape information generating subunit, configured to generate first direction shape information and second direction shape information according to the pseudo surface normal vector field matrix;

[0122] The gradient synthesis image generation subunit is used to generate a gradient synthesis image for representing image gradient information according to the first direction shape information and the second direction shape information.

[0123] Optionally, the gradient composite image generating subunit is specifically used for:

[0124] Generate first directional gradient information according to the first directional shape information and the first directional convolution kernel; and

[0125] Generate second direction gradient information according to the second direction shape information and the second direction convolution kernel;

[0126] A gradient composite image for representing image gradient information is generated according to the first directional gradient information and the second directional gradient information.

[0127] Optionally, the device further comprises:

[0128] A pixel synthesis image generation module, which is used for generating a pixel synthesis image based on the image grayscale information of each of the images of the measured object;

[0129] A unit vector determination module, used for determining, for any pixel point in the pixel synthesis image, a unit vector between the pixel point and its two adjacent pixel points in a direction;

[0130] A height composite image generation module, used to generate a height composite image for representing image height information based on a surface normal vector of the pixel point and each unit vector and a preset vector association relationship;

[0131] The second stereo composite image generation module is used to generate a photometric stereo composite image including the height composite image.

[0132] Optionally, the device further comprises:

[0133] A third direction state information generating module, used for generating third direction shape information according to the pseudo surface normal vector field matrix;

[0134] A color composite image generation module, configured to generate a color composite image based on a preset color image channel according to the first direction shape information, the second direction shape information and the third direction information;

[0135] The third stereo composite image generation module is used to generate a photometric stereo composite image including the color composite image.

[0136] The photometric stereo image synthesis device provided in the embodiment of the present invention can execute the photometric stereo image synthesis method provided in any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of the execution method.

[0137] Embodiment 4

[0138] Figure 4 A schematic diagram of the structure of an electronic device 40 that can be used to implement an embodiment of the present invention is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices (such as helmets, glasses, watches, etc.) and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present invention described and / or required herein.

[0139] like Figure 4As shown, the electronic device 40 includes at least one processor 41, and a memory connected to the at least one processor 41, such as a read-only memory (ROM) 42, a random access memory (RAM) 43, etc., wherein the memory stores a computer program that can be executed by at least one processor, and the processor 41 can perform various appropriate actions and processes according to the computer program stored in the read-only memory (ROM) 42 or the computer program loaded from the storage unit 48 to the random access memory (RAM) 43. In the RAM 43, various programs and data required for the operation of the electronic device 40 can also be stored. The processor 41, the ROM 42, and the RAM 43 are connected to each other through a bus 44. An input / output (I / O) interface 45 is also connected to the bus 44.

[0140] A number of components in the electronic device 40 are connected to the I / O interface 45, including: an input unit 46, such as a keyboard, a mouse, etc.; an output unit 47, such as various types of displays, speakers, etc.; a storage unit 48, such as a disk, an optical disk, etc.; and a communication unit 49, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 49 allows the electronic device 40 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0141] The processor 41 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the processor 41 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any appropriate processor, controller, microcontroller, etc. The processor 41 performs the various methods and processes described above, such as a photometric stereo image synthesis method.

[0142] In some embodiments, the photometric stereo image synthesis method may be implemented as a computer program, which is tangibly contained in a computer-readable storage medium, such as a storage unit 48. In some embodiments, part or all of the computer program may be loaded and / or installed on the electronic device 40 via the ROM 42 and / or the communication unit 49. When the computer program is loaded into the RAM 43 and executed by the processor 41, one or more steps of the photometric stereo image synthesis method described above may be performed. Alternatively, in other embodiments, the processor 41 may be configured to perform the photometric stereo image synthesis method in any other appropriate manner (e.g., by means of firmware).

[0143] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0144] Computer programs for implementing the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that when the computer program is executed by the processor, the functions / operations specified in the flow chart and / or block diagram are implemented. The computer program may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0145] In the context of the present invention, a computer-readable storage medium may be a tangible medium that may contain or store a computer program for use by or in combination with an instruction execution system, device or equipment. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices or equipment, or any suitable combination of the foregoing. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0146] To provide interaction with a user, the systems and techniques described herein may be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and a pointing device (e.g., a mouse or trackball) through which the user can provide input to the electronic device. Other types of devices may also be used to provide interaction with the user; for example, the feedback provided to the user may be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user may be received in any form (including acoustic input, voice input, or tactile input).

[0147] The systems and techniques described herein may be implemented in a computing system that includes backend components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes frontend components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such backend components, middleware components, or frontend components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), a blockchain network, and the Internet.

[0148] A computing system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The client and server relationship is generated by computer programs running on the corresponding computers and having a client-server relationship with each other. The server may be a cloud server, also known as a cloud computing server or cloud host, which is a host product in the cloud computing service system to solve the defects of difficult management and weak business scalability in traditional physical hosts and VPS services.

[0149] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps described in the present invention can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solution of the present invention can be achieved, and this document does not limit this.

[0150] The above specific implementations do not constitute a limitation on the protection scope of the present invention. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A photometric stereo image synthesis method, characterized in that: Applied to a photometric stereo image synthesis system, the system comprises: a processor, a trigger input circuit, an image sensor and a light source driving circuit; wherein the processor is respectively connected to the trigger input circuit, the image sensor and the light source driving circuit for communication; the method is executed by the processor, comprising: Acquire the movement pulse signal obtained by the trigger input circuit collecting the measured object; Processing the moving pulse signal to obtain a collection drive signal and a light source drive signal; Sending the light source driving signal to the light source driving circuit, so that the light source driving circuit generates at least two light source angles of the light source brightness irradiating the measured object according to the light source driving signal; The acquisition driving signal is sent to the image sensor, so that the image sensor can acquire images of the object under at least two light source angles according to the acquisition driving signal, and obtain and feed back at least two images of the object under different light source angles; According to the image grayscale information of the image of the object under test at each of the different light source angles, based on a preset image decomposition method, at least one photometric stereo composite image for representing the image stereo information is generated; The step of generating at least one photometric stereo composite image for representing stereo information of the image based on the image grayscale information of the image of the object under each of the different light source angles and based on a preset image decomposition method comprises: Generate an image intensity matrix according to the image grayscale information of each of the images of the measured objects; Performing singular value decomposition on the image intensity matrix to obtain a left singular matrix and a singular value matrix; Generating a gradient composite image for representing image gradient information according to the left singular matrix and the singular value matrix; A photometric stereo composite image is generated including the gradient composite image.

2. The method according to claim 1, characterized in that The step of generating a gradient composite image for representing image gradient information according to the left singular matrix and the singular value matrix comprises: Generate a pseudo surface normal vector field matrix according to the left singular matrix and the singular value matrix; Generating first direction shape information and second direction shape information according to the pseudo surface normal vector field matrix; A gradient composite image for representing image gradient information is generated according to the first directional shape information and the second directional shape information.

3. The method according to claim 2, characterized in that The step of generating a gradient composite image for representing image gradient information according to the first directional shape information and the second directional shape information includes: Generate first directional gradient information according to the first directional shape information and the first directional convolution kernel; and Generate second direction gradient information according to the second direction shape information and the second direction convolution kernel; A gradient composite image for representing image gradient information is generated according to the first directional gradient information and the second directional gradient information.

4. The method according to claim 1, characterized in that: The method further comprises: Generating a pixel composite image according to the image grayscale information of each of the images of the measured object; For any pixel point in the pixel composite image, determine the unit vector between the pixel point and its two adjacent pixel points in the direction; According to the surface normal vector of the pixel point and each unit vector, based on a preset vector association relationship, a highly synthesized image for representing image height information is generated; A photometric stereo composite image is generated including the height composite image.

5. The method according to claim 2, characterized in that: The method further comprises: generating shape information in a third direction according to the pseudo surface normal vector field matrix; Generate a color composite image based on a preset color image channel according to the first direction shape information, the second direction shape information and the third direction information; A photometric stereo composite image is generated including the color composite image.

6. A photometric stereo image synthesis system, characterized in that: The system comprises: a processor, a trigger input circuit, an image sensor and a light source driving circuit; wherein the processor is respectively connected to the trigger input circuit, the image sensor and the light source driving circuit for communication; The trigger input circuit is used to collect the movement pulse signal of the object under test and send the movement pulse signal to the processor; The processor is used to perform signal processing on the acquired moving pulse signal to obtain an acquisition driving signal and a light source driving signal, and send the light source driving signal to the light source driving circuit, and send the acquisition driving signal to the image sensor; The light source driving circuit is used to generate at least two light source angles of the light source brightness irradiating the object to be measured according to the light source driving signal; The image sensor is used to collect images of the object under test under at least two light source angles according to the acquisition drive signal, and obtain and feed back at least two images of the object under test under different light source angles; The processor is further configured to generate at least one photometric stereo composite image for representing stereo information of the image according to the image grayscale information of the image of the object under test at each of the different light source angles; The step of generating at least one photometric stereo composite image for representing stereo information of the image based on the image grayscale information of the image of the object under each of the different light source angles and based on a preset image decomposition method comprises: Generate an image intensity matrix according to the image grayscale information of each of the images of the measured objects; Performing singular value decomposition on the image intensity matrix to obtain a left singular matrix and a singular value matrix; Generating a gradient composite image for representing image gradient information according to the left singular matrix and the singular value matrix; A photometric stereo composite image is generated including the gradient composite image.

7. A photometric stereo image synthesis device, characterized in that: Applied to a photometric stereo image synthesis system, the system comprises: a processor, a trigger input circuit, an image sensor and a light source driving circuit; wherein the processor is respectively connected to the trigger input circuit, the image sensor and the light source driving circuit for communication; the device is configured on the processor, comprising: A pulse signal acquisition module, used to acquire the movement pulse signal obtained by the trigger input circuit collecting the measured object; A signal processing module, used for performing signal processing on the moving pulse signal to obtain a collection driving signal and a light source driving signal; A light source driving signal sending module, used for sending the light source driving signal to the light source driving circuit, so that the light source driving circuit generates at least two light source angles for the light source brightness irradiating the measured object according to the light source driving signal; An acquisition drive signal sending module is used to send the acquisition drive signal to the image sensor, so that the image sensor can acquire images of the object under at least two light source angles according to the acquisition drive signal, and obtain and feed back at least two images of the object under different light source angles; A stereo image synthesis module, used to generate at least one photometric stereo synthesis image for representing the stereo information of the image according to the image grayscale information of the image of the object under test at each of the different light source angles; The stereo image synthesis module comprises: An image intensity matrix generating unit, used for generating an image intensity matrix according to the image grayscale information of each of the images of the measured objects; A singular value decomposition unit, used for performing singular value decomposition on the image intensity matrix to obtain a left singular matrix and a singular value matrix; A gradient synthesis image generating unit, used for generating a gradient synthesis image for representing image gradient information according to the left singular matrix and the singular value matrix; The stereo image synthesis unit is used to generate a photometric stereo synthesis image including the gradient synthesis image.

8. An electronic device, characterized in that: The electronic device comprises: at least one processor; and a memory communicatively connected to the at least one processor; wherein, The memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can perform the photometric stereo image synthesis method according to any one of claims 1 to 5.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the photometric stereo image synthesis method according to any one of claims 1 to 5 when executed.

Images