Image decoding method, apparatus, device, and storage medium

Abstract

The application discloses an image decoding method and belongs to the technical field of artificial intelligence. The method comprises the following steps: acquiring N measured object encoding images corresponding to a measured object, wherein N is a positive integer; inputting the N measured object encoding images into a trained structured light decoding model to obtain a decoding result of each pixel point in the N measured object encoding images. The decoding of the pixel points in the measured object encoding images in the three-dimensional reconstruction based on the structured light can be realized.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of three-dimensional imaging, and in particular to an image decoding method and device, equipment and a storage medium. BACKGROUND

[0002] The three-dimensional imaging technology based on structured light is a non-contact measurement technology that is widely used at present, and is widely used in manufacturing, medical treatment, robots and other fields.

[0003] In a structured light imaging system, a projector and a camera are included. When the structured light imaging system is used to perform three-dimensional reconstruction of an object, the projector projects N encoded projection images onto the measured object. When each projection image is projected, the camera captures the object on which the projection image is projected to obtain a corresponding measured object encoding image. In this way, N measured object encoding images can be obtained. Each pixel point of each measured object encoding image is decoded to obtain a decoding result of the pixel point, and the decoding result is used to indicate whether the pixel point is a bright fringe or a dark fringe in the corresponding projection image. Furthermore, for the same pixel points in the N measured object encoding images, the three-dimensional coordinates of the corresponding points on the object can be obtained according to the decoding results of the N same pixel points.

[0004] It can be seen that how to realize the decoding of the pixel points is a key in the three-dimensional reconstruction process. SUMMARY

[0005] Embodiments of the present application provide an image decoding method, device, equipment and storage medium, which can realize the decoding of pixel points in a measured object encoding image in three-dimensional reconstruction based on structured light. The technical solution is as follows:

[0006] In a first aspect, an image decoding method is provided, and the method comprises:

[0007] Obtaining N measured object encoding images corresponding to a measured object, wherein N is a positive integer;

[0008] Inputting the N measured object encoding images into a trained structured light decoding model to obtain a decoding result of each pixel point in the N measured object encoding images.

[0009] In a possible implementation, the method further comprises:

[0010] Obtaining a full-bright image and a full-dark image corresponding to the measured object, wherein the full-bright image is obtained by capturing the measured object by a camera when a projection device projects a full-bright projection image onto the measured object, and the full-dark image is obtained by capturing the measured object by the camera when the projection device projects a full-dark projection image onto the measured object;

[0011] The N measured object encoding images are input into the trained structured light decoding model to obtain an encoding result of each pixel point in the N measured object encoding images.

[0012] The N measured object encoding images, the full-brightness image, and the full-dark image are input into the trained structured light decoding model as three-way input data to obtain a decoding result of each pixel point in the N measured object encoding images.

[0013] In a possible implementation, the method further includes:

[0014] A plurality of groups of training samples are obtained, each group of training samples including a full-brightness image corresponding to a sample measured object, a full-dark object corresponding to the sample measured object, N sample measured object encoding images corresponding to the sample measured object, and a decoding result true value of each pixel point in the N sample measured object encoding images, where N is a positive integer, and the material of the sample measured object or the measured object encoding image is different in different groups of training samples.

[0015] Based on the plurality of groups of training samples, an initial structured light decoding model is trained to obtain the trained structured light decoding model.

[0016] In a possible implementation, the obtaining of the plurality of groups of training samples includes:

[0017] The plurality of groups of training samples are generated by a 3D structured light imaging simulation system.

[0018] In a possible implementation, the structured light decoding model is an image segmentation neural network model.

[0019] In a second aspect, an image decoding apparatus is provided, and the apparatus includes:

[0020] An obtaining module is configured to obtain N measured object encoding images corresponding to a measured object, where N is a positive integer.

[0021] A decoding module is configured to input the N measured object encoding images into a trained structured light decoding model to obtain a decoding result of each pixel point in the N measured object encoding images.

[0022] In a possible implementation, the obtaining module is further configured to:

[0023] The full-brightness image and the full-dark image corresponding to the measured object are obtained, where the full-brightness image is obtained by a camera shooting the measured object when a projection device projects a full-brightness projection image onto the measured object, and the full-dark image is obtained by the camera shooting the measured object when the projection device projects a full-dark projection image onto the measured object.

[0024] The decoding module is configured to:

[0025] The N measured object encoding images, the full-brightness image and the full-dark image are input as three-way input data into the trained structured light decoding model to obtain a decoding result of each pixel point in the N measured object encoding images.

[0026] In a possible implementation, the apparatus further includes a training module configured to:

[0027] A plurality of groups of training samples are obtained, wherein each group of training samples includes a full-brightness image corresponding to a sample measured object, a full-dark object corresponding to the sample measured object, N sample measured object encoding images corresponding to the sample measured object, and a decoding result true value of each pixel point in the N sample measured object encoding images, where N is a positive integer, and the material or measured object encoding image of the sample measured object in different groups of training samples is different.

[0028] Based on the plurality of groups of training samples, an initial structured light decoding model is trained to obtain the trained structured light decoding model.

[0029] In a possible implementation, the obtaining module is configured to:

[0030] The plurality of groups of training samples are generated by a 3D structured light imaging simulation system.

[0031] In a possible implementation, the structured light decoding model is an image segmentation neural network model.

[0032] In a third aspect, an electronic device is provided, which includes a processor and a memory, and the memory stores at least one instruction, which is loaded and executed by the processor to implement the operations performed by the image decoding method according to the first aspect and possible implementation manners thereof.

[0033] In a fourth aspect, a computer-readable storage medium is provided, which stores at least one instruction, which is loaded and executed by a processor to implement the operations performed by the image decoding method according to the first aspect and possible implementation manners thereof.

[0034] In a fifth aspect, a computer program product is provided, which includes at least one instruction, which is loaded and executed by a processor to implement the operations performed by the image decoding method according to the first aspect and possible implementation manners thereof.

[0035] The technical scheme provided by the embodiments of the present application has the following beneficial effects:

[0036] After obtaining the measured object code image corresponding to the measured object, the trained structured light decoding model is used to infer the decoding result of each pixel point in the measured object code image, so as to realize decoding of each pixel point in the measured object code image and obtain the decoding result. BRIEF DESCRIPTION OF DRAWINGS

[0037] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings needed in the embodiment description will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0038] Figure 1 is a schematic diagram of a binocular vision system provided by the embodiments of the present application;

[0039] Figure 2 is a schematic diagram of a coded projection image provided by the embodiments of the present application;

[0040] Figure 3 is a method flowchart of image decoding provided by the embodiments of the present application;

[0041] Figure 4 is a structural schematic diagram of an image decoding device provided by the embodiments of the present application;

[0042] Figure 5 is a structural schematic diagram of an electronic device provided by the embodiments of the present application. DETAILED DESCRIPTION

[0043] In order to facilitate the understanding of the embodiments of the present application, the following will first explain some terms related to the embodiments of the present application.

[0044] 1. Three-dimensional reconstruction based on structured light

[0045] In the three-dimensional reconstruction system based on structured light, a camera and a projector are included. In some theories, the projector can be regarded as an inverse camera. In this way, as shown in Figure 1 , in the case of regarding the projector as an inverse camera, the three-dimensional reconstruction system based on structured light can be regarded as a binocular vision system. In this system, it is assumed that there is only one camera on the left side, and one pixel point on the camera can correspond to countless points on the line O L connecting the optical center and the imaging point of the camera in three-dimensional space, such as Q, Q1, Q2, etc. in the figure. In this way, only one camera cannot obtain the three-dimensional information of the object in space. On this basis, a right camera is introduced. Because objects at different distances from the left camera have different imaging positions on the right camera, if the imaging position of the object on the left camera QL and the imaging position Q of the camera on the right side of the object R , the three-dimensional position P of the object can be uniquely determined.

[0046] The camera is essentially to collect the light reflected by the object, while the projector is to project light onto the object. If the light of each pixel point projected by the projector is encoded, the encoded light is projected onto the surface of the object, and the light reflected by the surface of the object collected by the camera is decoded, it can be known that the light projected on the surface of the object is emitted by which pixel point of the projector, that is, the imaging position of the surface of the object on the virtual projector image can be known.

[0047] 2. Encoding and decoding of structured light

[0048] The encoding method of the light projected by the projector can be various, and the embodiment of the present application only takes the Gray code encoding as an example for description.

[0049] Taking a four-bit Gray code as an example, as shown in the figure, it is four projection images encoded by Gray code, and the black stripes represent 0 and the white stripes represent 1. The projector projects the four projection images onto the measured object in turn, and the camera takes a picture of the measured object on which the projection image is projected every time the projector projects a projection image, obtaining a corresponding measured object encoding image. In this way, four measured object encoding images can be obtained corresponding to the four projection images encoded by Gray code. Further, each pixel point in each projection image is decoded to obtain the decoding result of the pixel point, which is 0 or 1. Figure 2 In the related art, decoding is a binary operation on the pixel point. The common method can be as follows:

[0050] Method one,

[0051] A global gray threshold is set, and for each pixel point in each projection image, if the gray value of the pixel point is greater than the global gray threshold, the decoding result of the pixel point is determined to be 1, and if the gray value of the pixel point is greater than the global gray threshold, the decoding result of the pixel point is determined to be 0.

[0052] Method two,

[0053] Taking

[0054] Figure 2 ​For example, for the pixel point in the first row and the first column of any measured object code image, the pixel point in the first row and the first column of each of the remaining three measured object code images is the homologous pixel point of the pixel point.

[0055] Then, for each pixel point on each measured object code image, the maximum gray value of the gray value of the pixel point and the gray value of the homologous pixel point of the pixel point is determined, denoted as l max The minimum gray value of the gray value of the pixel point and the gray value of the homologous pixel point of the pixel point is determined, denoted as l min Then, the normalized gray value of the pixel point can be calculated, denoted as l n . n The calculation formula of l

[0056]

[0057] Wherein, l is the gray value of the pixel point.

[0058] After obtaining the normalized gray value of the pixel point, the size between the normalized gray value of the pixel point and the global gray threshold value is compared. If the normalized gray value of the pixel point is greater than the global gray threshold value, it is determined that the decoding result of the pixel point is 1, and if the normalized gray value of the pixel point is greater than the global gray threshold value, it is determined that the decoding result of the pixel point is 0.

[0059] To make the purpose, technical scheme and advantages of the present application clearer, the embodiments of the present application will be further described in detail below with reference to the drawings.

[0060] Figure 3 is a flowchart of an image decoding method provided by an embodiment of the present application. The method can be realized by an electronic device, and the embodiment of the present application can include the following steps: Figure 3

[0061] Step 301, obtaining N measured object code images corresponding to the measured object.

[0062] Wherein, N is a positive integer.

[0063] ​In implementation, in the case that the projector projects N encoded projection images to the measured object, for each encoded projection image, the camera can capture the measured object encoding image corresponding to the projection image, so that a total of N measured object encoding images can be captured. Further, the electronic device obtains the N measured object encoding images captured by the camera.

[0064] It should be noted that the N encoded projection images can be encoded using Gray code or binary code, and the application embodiments do not limit the encoding method.

[0065] Step 302, input the N measured object encoding images into the trained structured light decoding model to obtain the decoding result of each pixel point in the N measured object encoding images.

[0066] In implementation, after obtaining the N measured object encoding images corresponding to the measured object, the N measured object encoding images are sequentially input into the trained structured light decoding model according to the capture time sequence of the N measured object encoding images. The structured light decoding model outputs the decoding result of each pixel point in each measured object encoding image in the N measured object encoding images, and the decoding result of each pixel point is 0 or 1.

[0067] In a possible implementation, in order to improve the decoding accuracy, in addition to obtaining the N measured object encoding images corresponding to the measured object, the full-bright image and the full-dark image corresponding to the measured object can also be obtained, wherein the full-bright image is obtained by the camera capturing the measured object when the projector projects the full-bright projection image to the measured object, and the full-dark image is obtained by the camera capturing the measured object when the projector projects the full-dark projection image to the measured object. Correspondingly, when the trained structured light decoding model is used to decode the measured object encoding image, the processing can be as follows:

[0068] The N measured object encoding images, the full-bright image and the full-dark image are input into the trained structured light decoding model as three input data to obtain the decoding result of each pixel point in the N measured object encoding images.

[0069] In implementation, in order to ensure that each position on the measured object can be changed from bright fringe projection to dark fringe projection or from dark fringe projection to bright fringe projection at least once, the projector can project a full-bright projection image and a full-dark projection image to the measured object in sequence before projecting the N encoded projection images, wherein the full-bright projection image is an image with only bright fringes and no dark fringes, and the full-dark projection image is an image with only dark fringes and no bright fringes. Alternatively, the projector can project a full-bright projection image and a full-dark projection image to the measured object in sequence after projecting the N encoded projection images.

[0070] After the projector projects a full-bright projection image to the measured object, the camera captures the measured object on which the full-bright projection image is projected to obtain a full-bright image corresponding to the measured object. After the projector projects a full-dark projection image to the measured object, the camera captures the measured object on which the full-dark projection image is projected to obtain a full-dark image corresponding to the measured object. Correspondingly, the electronic device can obtain the full-dark image corresponding to the measured object and the full-bright image corresponding to the measured object in addition to the N encoded images of the measured object captured by the camera. Then, the electronic device inputs the N encoded images of the measured object, the full-bright image and the full-dark image as three-way input data into the trained structured light decoding model to obtain the decoding result of each pixel point in the N encoded images of the measured object.

[0071] The above structured light decoding model is described below.

[0072] The structured light decoding model in the embodiments of the present application is a kind of neural network model, specifically, it can be an image segmentation neural network model capable of realizing pixel-level segmentation, for example, U-Net model. The pixel-level image segmentation neural network model can output the segmentation result of each pixel in the image, and in the embodiments of the present application, the segmentation result is the decoding result, including two decoding results of 0 and 1. When the decoding result of the pixel point is 0, it indicates that the position of the measured object corresponding to the pixel point is projected by the dark fringe in the projection image corresponding to the measured object encoding image to which the pixel point belongs. When the decoding result of the pixel point is 1, it indicates that the position of the measured object corresponding to the pixel point is projected by the bright fringe in the projection image corresponding to the measured object encoding image to which the pixel point belongs.

[0073] In a possible implementation, before using the structured light decoding model for decoding, the structured light decoding model needs to be trained first. The training method can be as follows:

[0074] A plurality of groups of training samples are obtained, each group of training samples including a sample and a decoding result true value of each pixel point in N sample measured object coded images, and the material of the sample measured object or the sample measured object coded image in different groups of training samples is different. Based on the plurality of groups of training samples, the initial structured light decoding model is trained to obtain the trained structured light decoding model.

[0075] In implementation, the N sample measured object coded images corresponding to the sample measured object can be obtained from public resources on the Internet or obtained by a user through a structured light-based three-dimensional reconstruction system. In order to make the training samples diverse and thus make the inference ability of the structured light decoding model better, a plurality of groups of training samples can be used, the material of the sample measured object corresponding to different groups of training samples can be different, and the coding of the sample measured object coded image in different groups of training samples can also be different. After obtaining the sample measured object coded image, the sample measured object coded image can be labeled to label the decoding result true value of each pixel point. Labeling can be performed by using a labeling model, manual labeling, or pre-labeling by a labeling model and then correcting the pre-labeling result manually to obtain the decoding result true value of each pixel point in the sample measured object coded image.

[0076] The material of the sample measured object is different in order to make the reflectivity of the surface of the sample measured object different, so that the trained structured light decoding model can have good decoding effect on measured objects with different reflectivity. The coding of the sample measured object coded image is different in order to make the trained structured light decoding model have good decoding effect on measured object coded images with different coding.

[0077] After obtaining the plurality of groups of training samples, for each group of training samples, the N sample measured object coded images corresponding to the sample measured object in the training sample are used as input samples, and the decoding result true value of each pixel point in the N sample measured object coded images is used as output samples. Then, the input samples and the output samples are used to supervise the training of the structured light decoding model until the training stop condition is met, the training is ended, and the trained structured light decoding model is obtained. The training stop condition can be that the number of training times reaches a number threshold, or the model loss value is less than a loss value threshold.

[0078] When the structure light decoding model is used for decoding, if the full-bright image and the full-dark image corresponding to the measured object are input in addition to the measured object encoding image corresponding to the measured object, the full-dark image and the full-bright image corresponding to the sample measured object also need to be included in the training sample used when the structure light decoding model is trained. That is, in this case, the full-dark image corresponding to the sample measured object, the full-bright image corresponding to the sample measured object, N sample measured object encoding images corresponding to the sample measured object, and the decoding result true value of each pixel point in the N sample measured object encoding images are included in each set of training samples. Correspondingly, when training, the full-dark image corresponding to the sample measured object, the full-bright image corresponding to the sample measured object, and the N sample measured object encoding images corresponding to the sample measured object are used as three-way input data, the three-way input data are used as input samples, the decoding result true value of each pixel point in the N sample measured object encoding images is used as output samples, and then the input samples and the output samples are used to supervise the training of the structure light decoding model until the training stop condition is met, the training is ended, and the trained structure light decoding model is obtained.

[0079] In a possible implementation, the training sample can be obtained by simulation using a 3D structure light imaging simulation system. The 3D structure light imaging simulation system includes a light encoding projection unit mathematical model and an acquisition unit model. The light encoding projection unit mathematical model is established based on an inverse pinhole camera model, and the acquisition unit mathematical model is established based on a pinhole camera model. The light encoding projection unit mathematical model is the mathematical model of the camera, and the acquisition unit mathematical model is the mathematical model of the projector. After the light encoding projection unit mathematical model and the acquisition unit mathematical model are established, the light encoding projection parameters, the optical imaging parameters, and the optical distortion parameters of the light encoding projection unit mathematical model are set, the optical imaging parameters and the optical distortion parameters of the acquisition unit mathematical model are set, and the spatial position parameters between the light encoding projection unit mathematical model and the acquisition unit mathematical model are set. Based on the 3D structure light imaging simulation system, a computer graphics modeling and rendering tool is used to model the sample measured object, generate a plurality of sample measured objects with different materials, add each sample measured object to the 3D structure light imaging simulation system, simulate, and obtain the training sample. During simulation, the encoding of the projection image can be adjusted by adjusting the light encoding projection parameters. In this way, for each sample measured object with a material, a plurality of sets of training samples can be generated.

[0080] The technical scheme provided in the embodiments of the present application can be used to obtain the decoding result of each pixel point in the measured object encoding image after obtaining the measured object encoding image corresponding to the measured object, and to infer the decoding result of each pixel point in the measured object encoding image by using the trained structure light decoding model, so as to realize the decoding of each pixel point in the measured object encoding image and obtain the decoding result.

[0081] All the optional technical solutions described above can be combined to form optional embodiments of the present application, which will not be described again here.

[0082] The embodiments of the present application also provide an image decoding apparatus, which can be an electronic device, such as a mobile phone, a tablet computer, a computer, a television, a camera, a display, a wearable device, a vehicle, a robot, a home appliance, a medical device, a security device, a financial device, a network device, a server, a cloud, or the like. Figure 4 As shown in the figure, the apparatus can include an acquisition module 410 and a decoding module 420, wherein:

[0083] The acquisition module 410 is configured to acquire N measured object encoding images corresponding to a measured object, wherein N is a positive integer.

[0084] The decoding module 420 is configured to input the N measured object encoding images into a trained structured light decoding model to obtain a decoding result of each pixel point in the N measured object encoding images.

[0085] In a possible implementation, the acquisition module 410 is further configured to:

[0086] acquire a full-brightness image and a full-dark image corresponding to the measured object, wherein the full-brightness image is obtained by a camera shooting the measured object in a case where a projection device projects a full-brightness projection image to the measured object, and the full-dark image is obtained by the camera shooting the measured object in a case where the projection device projects a full-dark projection image to the measured object.

[0087] The decoding module 420 is configured to:

[0088] input the N measured object encoding images, the full-brightness image, and the full-dark image as three-way input data into the trained structured light decoding model to obtain the decoding result of each pixel point in the N measured object encoding images.

[0089] In a possible implementation, the apparatus further includes a training module configured to:

[0090] acquire a plurality of groups of training samples, wherein each group of training samples includes a full-brightness image corresponding to a sample measured object, a full-dark object corresponding to the sample measured object, N sample measured object encoding images corresponding to the sample measured object, and a decoding result true value of each pixel point in the N sample measured object encoding images, wherein N is a positive integer, and the material of the sample measured object or the sample measured object encoding image is different in different groups of training samples.

[0091] Based on the plurality of groups of training samples, an initial structured light decoding model is trained to obtain the trained structured light decoding model.

[0092] In a possible implementation, the acquisition module is configured to:

[0093] The multiple groups of training samples are generated through a 3D structured light imaging simulation system.

[0094] In a possible implementation, the structured light decoding model is an image segmentation neural network model.

[0095] The technical scheme provided by the embodiments of the present application, after obtaining the measured object coding image corresponding to the measured object, uses the trained structured light decoding model to infer the decoding result of each pixel point in the measured object coding image, so as to realize the decoding of each pixel point in the measured object coding image and obtain the decoding result.

[0096] It should be noted that: the image decoding apparatus provided by the above embodiments in the image decoding process, only the above-mentioned each functional module is divided into example, in actual application, can be needed to complete the above-mentioned function distribution by different functional module, namely the internal structure of electronic equipment is divided into different functional module, to complete the above description of all or part of the function. In addition, the image decoding apparatus and the image decoding method provided by the above embodiments belong to the same concept, and the specific implementation process is described in the method embodiment, which will not be repeated here.

[0097] Figure 5 The structural block diagram of the electronic device 500 provided by an example embodiment of the present application is shown. The electronic device 500 can be a portable mobile terminal, such as a smart phone, a tablet computer, an MP3 player (Moving Picture Experts Group Audio Layer III), an MP4 player (Moving Picture Experts Group Audio Layer IV), a notebook computer or a desktop computer. The electronic device 500 can also be referred to as a user equipment, a portable terminal, a laptop terminal, a desktop terminal, and other names.

[0098] Generally, the electronic device 500 includes a processor 501 and a memory 502.

[0099] The processor 501 can include one or more processing cores, such as a 4-core processor, an 8-core processor, and the like. The processor 501 can be implemented in at least one of a hardware form of a DSP (Digital Signal Processing), an FPGA (Field-Programmable Gate Array), a PLA (Programmable Logic Array). The processor 501 can also include a main processor and a coprocessor. The main processor is a processor for processing data in an awake state, also referred to as a CPU (Central Processing Unit). The coprocessor is a low-power processor for processing data in a standby state. In some embodiments, the processor 501 can be integrated with a GPU (Graphics Processing Unit) for rendering and drawing content required to be displayed by the display screen. In some embodiments, the processor 501 can further include an AI (Artificial Intelligence) processor for processing machine learning related computing operations.

[0100] The memory 502 can include one or more computer-readable storage media, which can be non-transitory. The memory 502 can also include a high-speed random access memory, and a nonvolatile memory such as one or more disk storage devices, flash storage devices. In some embodiments, the non-transitory computer-readable storage medium in the memory 502 is used to store at least one instruction for being executed by the processor 501 to implement the image decoding method provided by the method embodiments in the present application.

[0101] In some embodiments, the electronic device 500 can also optionally include a peripheral device interface 503 and at least one peripheral device. The processor 501, the memory 502, and the peripheral device interface 503 can be connected through a bus or a signal line. Each peripheral device can be connected to the peripheral device interface 503 through a bus, a signal line, or a circuit board. Specifically, the peripheral device includes at least one of a radio frequency circuit 504, a display screen 505, a camera assembly 506, an audio circuit 507, a positioning assembly 508, and a power supply 509.

[0102] The peripheral interface 503 can be used to connect at least one I / O (Input / Output) related peripheral device to the processor 501 and the memory 502. In some embodiments, the processor 501, the memory 502 and the peripheral interface 503 are integrated on the same chip or circuit board; in some other embodiments, any one or two of the processor 501, the memory 502 and the peripheral interface 503 can be implemented on a separate chip or circuit board, and the present embodiments are not limited in this regard.

[0103] The radio frequency circuit 504 is configured to receive and send RF (Radio Frequency) signals, also known as electromagnetic signals. The radio frequency circuit 504 communicates with communication networks and other communication devices through electromagnetic signals. The radio frequency circuit 504 converts electrical signals into electromagnetic signals for transmission, or converts received electromagnetic signals into electrical signals. Optionally, the radio frequency circuit 504 includes an antenna system, an RF transceiver, one or more amplifiers, a tuner, an oscillator, a digital signal processor, a codec chipset, a subscriber identity module card, and the like. The radio frequency circuit 504 can communicate with other terminals through at least one wireless communication protocol. The wireless communication protocol includes but is not limited to the World Wide Web, a metropolitan area network, an intranet, various generations of mobile communication networks (2G, 3G, 4G and 5G), a wireless local area network and / or a WiFi (Wireless Fidelity) network. In some embodiments, the radio frequency circuit 504 can also include NFC (Near Field Communication) related circuitry, and the present application is not limited in this regard.

[0104] The display screen 505 is configured to display a UI (User Interface). The UI can include graphics, text, icons, video, and any combination thereof. When the display screen 505 is a touch display screen, the display screen 505 is further configured to capture touch signals on or above the surface of the display screen 505. The touch signals can be input to the processor 501 as control signals for processing. In this case, the display screen 505 can also be configured to provide virtual buttons and / or virtual keyboard, also known as soft buttons and / or soft keyboard. In some embodiments, the display screen 505 can be one, disposed on the front panel of the electronic device 500; in other embodiments, the display screen 505 can be at least two, respectively disposed on different surfaces of the electronic device 500 or in a folding design; in other embodiments, the display screen 505 can be a flexible display screen, disposed on a curved surface or a folding surface of the electronic device 500. Even, the display screen 505 can also be disposed in an irregular shape, i.e., a special-shaped screen. The display screen 505 can be made of LCD (Liquid Crystal Display), OLED (Organic Light-Emitting Diode), etc.

[0105] The camera assembly 506 is configured to capture images or videos. Optionally, the camera assembly 506 includes a front camera and a rear camera. Typically, the front camera is disposed on the front panel of the terminal, and the rear camera is disposed on the back of the terminal. In some embodiments, the rear camera is at least two, which are any one of a main camera, a depth-of-field camera, a wide-angle camera, and a telephoto camera, to realize the background blur function by fusing the main camera and the depth-of-field camera, the panoramic shooting and VR (Virtual Reality) shooting function by fusing the main camera and the wide-angle camera, or other fusion shooting functions. In some embodiments, the camera assembly 506 can further include a flash. The flash can be a single-color-temperature flash or a dual-color-temperature flash. The dual-color-temperature flash refers to the combination of a warm light flash and a cold light flash, which can be used for light compensation under different color temperatures.

[0106] The audio circuit 507 can include a microphone and a speaker. The microphone is used to collect sound waves of the user and the environment, and convert the sound waves into an electrical signal input to the processor 501 for processing, or input to the radio frequency circuit 504 to realize voice communication. For the purpose of stereo sound collection or noise reduction, the microphone can be multiple, respectively arranged at different parts of the electronic device 500. The microphone can also be an array microphone or an omnidirectional collection type microphone. The speaker is used to convert the electrical signal from the processor 501 or the radio frequency circuit 504 into sound waves. The speaker can be a conventional diaphragm speaker, or a piezoelectric ceramic speaker. When the speaker is a piezoelectric ceramic speaker, not only can it convert electrical signals into sound waves that humans can hear, but it can also convert electrical signals into sound waves that humans cannot hear for ranging purposes. In some embodiments, the audio circuit 507 can also include a headphone jack.

[0107] The positioning component 508 is used to position the current geographic location of the electronic device 500 to realize navigation or LBS (Location Based Service). The positioning component 508 can be a positioning component based on the GPS (Global Positioning System) of the United States, the Beidou system of China, or the Galileo system of Russia.

[0108] The power supply 509 is used to supply power to various components in the electronic device 500. The power supply 509 can be alternating current, direct current, disposable battery or rechargeable battery. When the power supply 509 includes a rechargeable battery, the rechargeable battery can be a wired charging battery or a wireless charging battery. The wired charging battery is a battery charged through a wired line, and the wireless charging battery is a battery charged through a wireless coil. The rechargeable battery can also be used to support fast charging technology.

[0109] In some embodiments, the electronic device 500 further includes one or more sensors 510. The one or more sensors 510 include but are not limited to: an acceleration sensor 511, a gyroscope sensor 512, a pressure sensor 513, a fingerprint sensor 514, an optical sensor 515, and a proximity sensor 516.

[0110] The acceleration sensor 511 can detect the acceleration in three coordinate axes of the coordinate system established by the electronic device 500. For example, the acceleration sensor 511 can be used to detect the components of gravitational acceleration in three coordinate axes. The processor 501 can control the display screen 505 to display the user interface in a landscape view or a portrait view according to the gravitational acceleration signal collected by the acceleration sensor 511. The acceleration sensor 511 can also be used for gaming or user motion data collection.

[0111] The gyroscope sensor 512 can detect the body direction and rotation angle of the electronic device 500, and can collect 3D motions of the user on the electronic device 500 in cooperation with the acceleration sensor 511. The processor 501 can realize the following functions according to the data collected by the gyroscope sensor 512: motion sensing (such as changing the UI according to the tilt operation of the user), image stabilization during shooting, game control, and inertial navigation.

[0112] The pressure sensor 513 can be arranged on the side frame of the electronic device 500 and / or the lower layer of the display screen 505. When the pressure sensor 513 is arranged on the side frame of the electronic device 500, the holding signal of the user on the electronic device 500 can be detected, and the left-hand or right-hand recognition or shortcut operation can be performed by the processor 501 according to the holding signal collected by the pressure sensor 513. When the pressure sensor 513 is arranged on the lower layer of the display screen 505, the controllable control on the UI interface can be controlled by the processor 501 according to the pressure operation of the user on the display screen 505. The controllable control includes at least one of a button control, a scroll bar control, an icon control, and a menu control.

[0113] The fingerprint sensor 514 is used to collect the fingerprint of the user, and the identity of the user can be recognized by the processor 501 according to the fingerprint collected by the fingerprint sensor 514, or by the fingerprint sensor 514 according to the collected fingerprint. When the identity of the user is recognized as a trusted identity, the processor 501 authorizes the user to perform related sensitive operations, including unlocking the screen, viewing encrypted information, downloading software, payment, and changing settings. The fingerprint sensor 514 can be arranged on the front, back or side of the electronic device 500. When the physical button or the manufacturer's logo is arranged on the electronic device 500, the fingerprint sensor 514 can be integrated with the physical button or the manufacturer's logo.

[0114] The optical sensor 515 is used to collect the ambient light intensity. In one embodiment, the processor 501 can control the display brightness of the display screen 505 according to the ambient light intensity collected by the optical sensor 515. Specifically, when the ambient light intensity is high, the display brightness of the display screen 505 is increased; when the ambient light intensity is low, the display brightness of the display screen 505 is decreased. In another embodiment, the processor 501 can also dynamically adjust the shooting parameters of the camera assembly 506 according to the ambient light intensity collected by the optical sensor 515.

[0115] The proximity sensor 516, also referred to as a distance sensor, is usually arranged on the front panel of the electronic device 500. The proximity sensor 516 is used to collect the distance between the user and the front of the electronic device 500. In an embodiment, when the proximity sensor 516 detects that the distance between the user and the front of the electronic device 500 gradually decreases, the display screen 505 is switched from the bright screen state to the screen-off state under the control of the processor 501; when the proximity sensor 516 detects that the distance between the user and the front of the electronic device 500 gradually increases, the display screen 505 is switched from the screen-off state to the bright screen state under the control of the processor 501.

[0116] Those skilled in the art can understand that the structure shown in the above embodiments does not constitute a limitation on the electronic device 500, and can include more or fewer components than shown, or combine certain components, or adopt a different component arrangement. Figure 5 Those skilled in the art can understand that the structure shown in the above embodiments does not constitute a limitation on the electronic device 500, and can include more or fewer components than shown, or combine certain components, or adopt a different component arrangement.

[0117] In an example embodiment, a computer readable storage medium, such as a memory including instructions executable by a processor in a terminal to perform the method of image decoding in the above embodiments, is also provided. The computer readable storage medium can be non-transitory. For example, the computer readable storage medium can be a ROM (Read-Only Memory), a RAM (Random Access Memory), a CD-ROM (Compact Disc Read-Only Memory), a magnetic tape, a floppy disk, and an optical data storage device, etc.

[0118] It should be noted that the information (including but not limited to user device information, user personal information, etc.), data (including but not limited to data for analysis, stored data, displayed data, etc.), and signals (including but not limited to signals transmitted between user terminals and other devices, etc.) involved in the present application are all authorized by the user or fully authorized by all parties, and the collection, use, and processing of related data need to comply with relevant laws, regulations, and standards of relevant countries and regions. For example, the measured object coded image and the sample measured object coded image involved in the present application are obtained under full authorization.

[0119] Those of ordinary skill in the art can understand that all or part of the steps of the above embodiments can be completed by hardware, or by a program instructing related hardware, and the program can be stored in a computer readable storage medium, and the storage medium mentioned above can be a read-only memory, a magnetic disk or an optical disk, etc.

[0120] The above merely provides the optional embodiments of the present application, and is not intended to limit the present application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. An image decoding method, characterized in that, The method includes: Multiple sets of training samples are obtained. Each set of training samples includes a fully bright image corresponding to the sample tested object, a fully dark object corresponding to the sample tested object, N encoded images of the sample tested object corresponding to the sample tested object, and the true value of the decoding result of each pixel in the N encoded images of the sample tested object. The material of the sample tested object or the encoded image of the sample tested object in different sets of training samples are different, where N is a positive integer. Based on the multiple sets of training samples, the initial structured light decoding model is trained to obtain the trained structured light decoding model. N coded images of the tested object are obtained. The method for obtaining the coded images of the tested object is as follows: the projection device sequentially projects N Gray code-encoded projection images onto the tested object. For each projection image, the camera takes a picture of the tested object with the projected projection image to obtain a coded image of the tested object. Acquire a full-brightness image and a full-darkness image corresponding to the object under test, wherein the full-brightness image is obtained by the camera taking a picture of the object under test when the projection device projects a full-brightness projection image onto the object under test, and the full-darkness image is obtained by the camera taking a picture of the object under test when the projection device projects a full-darkness projection image onto the object under test; The N encoded images of the tested object, the fully bright image, and the fully dark image are used as three input data streams and input into the trained structured light decoding model to obtain the decoding result of each pixel in the N encoded images of the tested object.

2. The method according to claim 1, characterized in that, The acquisition of multiple sets of training samples includes: The multiple sets of training samples are generated using a three-dimensional structured light imaging simulation system.

3. The method according to claim 1 or 2, characterized in that, The structured light decoding model is an image segmentation neural network model.

4. A structured light decoding device, characterized in that, The device includes: The training module is used to acquire multiple sets of training samples. Each set of training samples includes a fully bright image corresponding to the sample object being tested, a fully dark image corresponding to the sample object being tested, N encoded images of the sample object being tested, and the ground truth value of the decoding result of each pixel in the N encoded images of the sample object being tested. The material of the sample object being tested or the encoded images of the sample object being tested are different in different sets of training samples, where N is a positive integer. Based on the multiple sets of training samples, the initial structured light decoding model is trained to obtain the trained structured light decoding model. The acquisition module is used to acquire N coded images of the object under test. The method for acquiring the coded images of the object under test is as follows: the projection device sequentially projects N Gray code-encoded projection images onto the object under test. For each projection image, the camera takes a picture of the object under test with the projected projection image, thus obtaining a coded image of the object under test; the module also acquires a fully bright image and a fully dark image corresponding to the object under test. The fully bright image is obtained by taking a picture of the object under test when the projection device projects a fully bright projection image onto the object under test, and the fully dark image is obtained by taking a picture of the object under test when the projection device projects a fully dark projection image onto the object under test. The decoding module is used to input the N encoded images of the tested object, the fully bright image, and the fully dark image as three input data into the trained structured light decoding model to obtain the decoding result of each pixel in the N encoded images of the tested object.

5. The apparatus according to claim 4, characterized in that, The acquisition module is used for: The multiple sets of training samples are generated using a three-dimensional structured light imaging simulation system.

6. The apparatus according to claim 4 or 5, characterized in that, The structured light decoding model is an image segmentation neural network model.

7. An electronic device, characterized in that, The electronic device includes a processor and a memory, the memory storing at least one instruction, which is loaded and executed by the processor to perform the operation of the image decoding method as described in any one of claims 1 to 3.

8. A computer-readable storage medium, characterized in that, The storage medium stores at least one instruction, which is loaded and executed by a processor to perform the operation of the image decoding method as described in any one of claims 1 to 3.

Images