Non-vision field imaging system and reconstruction method based on TOF sensor and laser

Abstract

The invention discloses a non-vision-field imaging system and reconstruction method based on a TOF sensor and a laser, and the method comprises the steps: building a non-vision-field imaging scene which is composed of the TOF sensor, the laser, an intermediate surface, and an imaging target; collecting a non-vision field data set, wherein the non-vision field data set is composed of an unprocessed original depth degraded image and a corresponding depth clear image; according to the non-vision field data set, a three-dimensional reconstruction model of a variable-scale coding-decoding architecture based on a deep neural network is built for reconstructing a non-vision field target, and the preprocessed non-vision field data set is adopted to complete training of the three-dimensional reconstruction model; and inputting a to-be-reconstructed depth degradation image into the three-dimensional reconstruction model for three-dimensional reconstruction, and outputting a depth reconstruction image of the non-vision field scene. The method is good in reconstruction effect, high in applicability and low in cost, can reconstruct the contour of the imaging target under the condition that the depth degradation image is completely fuzzy, is less in shadow, and can be widely applied to the technical field of computers.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a non-line-of-sight imaging system and a reconstruction method based on a TOF sensor and a laser. Background Art

[0002] Non-line-of-sight imaging refers to a technology in which the detector collects the target light information outside the field of view with the help of an intermediate reflective surface and then performs computational imaging. The purpose of non-line-of-sight imaging is to achieve imaging or three-dimensional reconstruction of non-line-of-sight targets by performing optical inverse operations on the speckle light information on the intermediate surface or collecting the target return light information. This technology can play a key role in information collection in scenes such as cave detection and fire rescue, and has important research significance and application value. The light signal of the non-line-of-sight target enters the imaging system through reflection from the intermediate surface. Due to the scattering characteristics of the intermediate surface, the target light signal undergoes different degrees of random scattering, resulting in serious degradation of the image information. During the detector acquisition process, in order to ensure that the target light signal enters the sensor, it is necessary to analyze the propagation path of the light signal. At the same time, in order to reduce the interference of stray light, it is necessary to avoid the backward stray light caused by the first incident light of the lighting system on the intermediate surface. At present, the methods of non-line-of-sight imaging technology usually include active imaging and passive imaging. Among them, active imaging methods are divided into transient imaging and range-gated imaging. The transient imaging method uses high-speed detectors and high-speed lighting to coordinate work, collect the target's return light information for accurate reconstruction of time and intensity. The system cost and maintenance cost are high, and the imaging speed is slow. The range-gated method adds a range-gated gate in front of the high-sensitivity detector, and receives the characteristic distance light signal through precise time control, thereby avoiding the primary scattering of the intermediate surface and only receiving the target light information to achieve the purpose of non-line-of-sight imaging. This method is complex to control, has poor imaging quality and slow speed. The passive imaging method collects the target speckle light information on the intermediate surface and performs optical inverse operations to reconstruct the target. However, since the scattering characteristics are unknown, the inverse operation is usually complex to calculate and the boundary conditions are insufficient, with great limitations and poor imaging results. Therefore, it is urgent to design a simple, easy and effective method to meet the application needs of non-line-of-sight imaging scenarios.

[0003] Non-line-of-sight imaging usually consists of three parts: imaging equipment, intermediary surface, and non-line-of-sight scene. The basic method of non-line-of-sight imaging is to collect non-line-of-sight target information through the intermediary surface, and then use the corresponding processing method to achieve target reconstruction. Existing non-line-of-sight imaging technologies can be divided into the following types according to the implementation method:

[0004] (1) Transient non-line-of-sight imaging technology based on ultrafast illumination and detection. Use femtosecond lasers to emit ultrashort pulses to directly detect and capture non-line-of-sight target light information; or use single-photon detectors to count the number of photons through scanning mechanisms to achieve non-line-of-sight target identification. The ultrashort high-energy pulses of femtosecond pulse lasers provide a wider range of conditions for measuring spatial information, and the streak tube camera has the acquisition resolution of femtosecond light signals. When the camera receives the continuously emitted pulse light signals, it can calculate the time difference between the light signal going back and forth between the target and the detector, and then distinguish the optical path experienced by the femtosecond pulse to achieve three-dimensional reconstruction of the target. However, the pulse point light source requires multiple scanning illumination, which makes the calculation process very time-consuming, and the cost of devices for nanosecond processing and calculation is relatively high and complex. In recent years, although this method has reduced costs and imaging speeds through confocal optical systems and single-photon detection equipment, this method still has the problems of high equipment maintenance costs and slow imaging speeds.

[0005] (2) Distance-gated imaging technology. When the effective signal returns, the gate is opened, and the detector only collects the return light signal at a specific distance, thereby effectively avoiding the influence of stray light from the reflective surface and realizing non-field of view target imaging. However, the collected data is still affected by the scattering of the intermediate surface, and it is necessary to perform inverse operations based on physical characteristics. There are problems such as long calculation time and complex inverse operation model. In addition, the illumination pulse width has a direct impact on the target depth measurement accuracy and the shortest measurement distance.

[0006] (3) Passive non-line-of-sight imaging technology and other methods. By collecting the speckle light information on the reflective intermediate surface and combining it with the inverse model of light path propagation, the target detection on the other side is realized. This type of method lacks information about the light propagation time and it is difficult to achieve three-dimensional reconstruction of non-line-of-sight targets. However, this type of imaging system has a simpler maintenance method than the above two methods. Combined with the non-Lambertian scattering intermediate surface and artificial intelligence related technologies in recent years, it has become one of the research hotspots in recent years. Summary of the invention

[0007] The main purpose of the embodiments of the present invention is to propose a non-line-of-sight imaging system and reconstruction method based on TOF sensor and laser with good reconstruction effect, strong applicability and low cost, which can reconstruct the outline of the imaging target from the completely blurred depth degradation map with fewer shadows.

[0008] To achieve the above object, an embodiment of the present invention provides a non-line-of-sight imaging system and reconstruction method based on a TOF sensor and a laser, comprising the following steps:

[0009] Building a non-line-of-sight imaging scene, wherein the non-line-of-sight imaging scene is composed of a TOF sensor, a laser, an intermediate surface, and an imaging target;

[0010] Collecting a non-viewing area data set, wherein the non-viewing area data set consists of an unprocessed original depth-degraded image and a corresponding depth-clear image;

[0011] According to the non-line-of-sight data set, a three-dimensional reconstruction model based on a variable-scale encoding-decoding architecture of a deep neural network is constructed for reconstructing non-line-of-sight targets, and the preprocessed non-line-of-sight data set is used to complete the training of the three-dimensional reconstruction model;

[0012] After the training of the 3D reconstruction model is completed, the depth-degraded image to be reconstructed is input into the 3D reconstruction model for 3D reconstruction, and a depth-reconstructed image of the non-viewing area scene is output.

[0013] In some embodiments, the step of setting up a non-line-of-sight imaging scene comprises the following steps:

[0014] The target is fully illuminated by the modulated light emitted by the illumination light source, and the TOF sensor realizes depth calculation imaging through the phase difference between the illumination light and the received light; wherein, the non-line-of-sight imaging system based on the TOF sensor is mainly composed of an illumination unit, a detector, a control unit, and a processor;

[0015] The imaging step of the non-line-of-sight imaging system comprises the following steps:

[0016] Driven by the synchronization signal, the laser emits high-frequency modulated light to illuminate the target, and the light beam hits the target object and is reflected;

[0017] The detector receives the reflected light signal and forms an image on the TOF sensor;

[0018] The memory temporarily stores the captured image and depth data;

[0019] The processor processes the image and depth data, connects to the external computing device through the data interface, and transmits the data to the computer device for processing and storage.

[0020] In some embodiments, the collecting of the non-viewing area dataset comprises the following steps:

[0021] Use TOF sensor and laser to image the target and obtain the depth information of each pixel of the target;

[0022] The non-viewing area imaging scene is used to collect depth image data, and during the collection process, the interface between the intermediate surface and the imaging target is replaced with a mirror;

[0023] A rotating stage is placed under the imaging target, and the features of the imaging target at different angles can be obtained by rotating it one circle.

[0024] In some embodiments, building a 3D reconstruction model based on a variable scale encoding-decoding architecture of a deep neural network for reconstructing non-line-of-sight targets according to the non-line-of-sight dataset, and using the preprocessed non-line-of-sight dataset to complete the training of the 3D reconstruction model comprises the following steps:

[0025] On the encoder side, ResNet-50 is used as the feature extraction network, and the feature extraction network introduces the bottleneck residual module to construct the residual relationship between the input and the output; wherein, the feature extraction network is divided into 5 stages, wherein the first stage inputs the preprocessed deep degraded image into the feature extraction network to extract multi-layer features, and the next four stages contain 3, 4, 6, and 3 bottleneck residual modules respectively;

[0026] At the decoder end, the channel attention module is used to process the effective features obtained in each stage respectively; the specific process is as follows: the features extracted in the previous stage are upsampled after passing through the channel attention module, and finally added with the features extracted in the next stage as the input of the corresponding channel attention module in the next stage, and skip connections are used to achieve multi-scale effective feature fusion; at the end of the decoder, the features decoded by channel attention in each stage are upsampled, and then the features are fused to output the final deep reconstructed image.

[0027] In some embodiments, the method further comprises:

[0028] The 3D reconstruction model performs depth data preprocessing on the non-viewing area dataset before training, specifically: inputting a depth-degraded image with a size of 192×192 and outputting a depth-reconstructed image with a size of 96×96;

[0029] The MATLAB programming language is used to implement the non-line-of-sight 3D reconstruction model. The loss function is set as the mean square error. The Adam optimizer is used to backpropagate the loss function to minimize the loss function value. Finally, the depth-degraded images of the test set are sent to the trained model for 3D reconstruction, achieving high-quality 3D reconstruction in data-driven non-line-of-sight scenarios.

[0030] Another aspect of the embodiments of the present invention further provides a non-line-of-sight imaging system using a TOF sensor and a laser, comprising:

[0031] The first module is used to build a non-line-of-sight imaging scene, which consists of a TOF sensor, a laser, an intermediate surface, and an imaging target;

[0032] The second module is used to collect a non-viewing area dataset, where the non-viewing area dataset consists of an unprocessed original depth-degraded image and a corresponding depth-clear image;

[0033] The third module is used to build a 3D reconstruction model based on a variable scale coding-decoding architecture of a deep neural network according to the non-line-of-sight data set for reconstructing non-line-of-sight targets, and use the preprocessed non-line-of-sight data set to complete the training of the 3D reconstruction model;

[0034] The fourth module is used to input the depth-degraded image to be reconstructed into the three-dimensional reconstruction model for three-dimensional reconstruction after completing the training of the three-dimensional reconstruction model, and output the depth-reconstructed image of the non-viewing area scene.

[0035] In some embodiments, the second module is specifically used to:

[0036] Use TOF sensor and laser to image the target and obtain the depth information of each pixel of the target;

[0037] The non-viewing area imaging scene is used to collect depth image data, and during the collection process, the interface between the intermediate surface and the imaging target is replaced with a mirror;

[0038] A rotating stage is placed under the imaging target, and the features of the imaging target at different angles can be obtained by rotating it one circle.

[0039] To achieve the above objective, another aspect of an embodiment of the present invention provides an electronic device, the electronic device comprising a memory and a processor, the memory storing a computer program, and the processor implementing the above-mentioned method when executing the computer program.

[0040] To achieve the above objective, another aspect of an embodiment of the present invention provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the above-mentioned method is implemented.

[0041] The embodiment of the present invention also discloses a computer program product or a computer program, which includes a computer instruction stored in a computer-readable storage medium. A processor of a computer device can read the computer instruction from the computer-readable storage medium, and the processor executes the computer instruction, so that the computer device executes the above method.

[0042] The embodiments of the present invention include at least the following beneficial effects: the present invention provides a non-line-of-sight imaging system and reconstruction method based on a TOF sensor and a laser, the scheme first builds a non-line-of-sight imaging scene, the non-line-of-sight imaging scene is composed of a TOF sensor, a laser, an intermediate surface, and an imaging target; then a non-line-of-sight data set is collected, the non-line-of-sight data set is composed of an unprocessed original depth-degraded image and a corresponding depth-clear image; then, according to the non-line-of-sight data set, a three-dimensional reconstruction model based on a variable-scale encoding-decoding architecture of a deep neural network is built for reconstructing non-line-of-sight targets, and the pre-processed non-line-of-sight data set is used to complete the training of the three-dimensional reconstruction model; after completing the training of the three-dimensional reconstruction model, the depth-degraded image to be reconstructed is input into the three-dimensional reconstruction model for three-dimensional reconstruction, and a depth-reconstructed image of the non-line-of-sight scene is output. The present invention has good reconstruction effect, strong applicability and low cost, and can reconstruct the outline of the imaging target from the completely blurred depth degradation image, and there are fewer shadows. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] Figure 1 It is a schematic diagram of an implementation environment provided by an embodiment of the present invention;

[0044] Figure 2 is a flow chart of the overall steps provided by an embodiment of the present invention;

[0045] Figure 3 is a non-viewing-area imaging equivalent diagram provided by an embodiment of the present invention;

[0046] Figure 4 is a schematic diagram of a non-line-of-sight imaging scenario provided by an embodiment of the present invention;

[0047] Figure 5 is a schematic diagram of a neural network SMM model provided by an embodiment of the present invention;

[0048] Figure 6 is an image reconstruction result provided by an embodiment of the present invention where the intermediate surface is a Lambertian surface;

[0049] Figure 7 is a schematic diagram of the system structure provided by an embodiment of the present invention;

[0050] Figure 8 It is a schematic diagram of the hardware structure of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0051] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention. When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the embodiments of the present invention. They are only examples of devices and methods consistent with some aspects of the embodiments of the present invention as detailed in the attached claims.

[0052] It is understood that the terms "first", "second", etc. used in the present invention may be used herein to describe various concepts, but unless otherwise specified, these concepts are not limited by these terms. These terms are only used to distinguish one concept from another. For example, without departing from the scope of the embodiments of the present invention, the first information may also be referred to as the second information, and similarly, the second information may also be referred to as the first information. Depending on the context, the words "if" and "if" as used herein may be interpreted as "at the time of" or "when" or "in response to determining".

[0053] The terms "at least one", "multiple", "each", "any", etc. used in the present invention, at least one includes one, two or more, multiple includes two or more, each refers to each of the corresponding multiple, and any refers to any one of the multiple.

[0054] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which the present invention belongs. The terms used herein are only for the purpose of describing the embodiments of the present invention and are not intended to limit the present invention.

[0055] Before describing the embodiments of the present invention in detail, some related technologies involved in the embodiments of the present invention are first described as follows:

[0056] TOF camera: Time of flight camera.

[0057] TOF sensor: A distance measuring device based on the time of flight principle, which calculates the distance by emitting light or sound waves of a specific wavelength and measuring the time it takes for the signal to be reflected back to the receiver.

[0058] Laser: An optical device that converts electrical or other forms of energy into a laser beam.

[0059] Non-line-of-sight imaging: A technique for implementing computational imaging of targets that are not directly visible, such as those behind obstacles or around corners.

[0060] Intermediary surface: The reflective surface used for non-line-of-sight imaging.

[0061] ResNet-50: Deep Residual Networks.

[0062] The non-line-of-sight imaging system and reconstruction method based on TOF sensor and laser provided in the embodiment of the present invention relate to the field of computer technology. The non-line-of-sight imaging system and reconstruction method based on TOF sensor and laser provided in the embodiment of the present invention can be applied to a terminal, can also be applied to a server, and can also be software running in a terminal or a server. In some embodiments, the terminal can be a smart phone, a tablet computer, a laptop computer, a desktop computer, a smart speaker, a smart watch, and a car terminal, etc., but is not limited to this; the server side can be configured as an independent physical server, or a server cluster or distributed system composed of multiple physical servers, and can also be configured as a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms. The server can also be a node server in a blockchain network; the software can be an application that implements a non-line-of-sight imaging system and reconstruction method based on TOF sensors and lasers, etc., but is not limited to the above forms.

[0063] The present invention can be used in many general or special computer system environments or configurations. For example: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronic devices, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, etc. The present invention can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. The present invention can also be practiced in distributed computing environments, in which tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media including storage devices.

[0064] like Figure 1 FIG. 1 is a schematic diagram of an implementation environment provided by an embodiment of the present invention. Figure 1 The implementation environment includes at least one terminal 102 and a server 101. The terminal 102 and the server 101 can be connected to a network wirelessly or wired to complete data transmission and exchange.

[0065] Server 101 can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers. It can also be a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN (Content Delivery Network), as well as big data and artificial intelligence platforms.

[0066] In addition, the server 101 can also be a node server in the blockchain network. Blockchain is a new application model of computer technologies such as distributed data storage, peer-to-peer transmission, consensus mechanism, and encryption algorithm.

[0067] The terminal 102 may be a smart phone, a tablet computer, a laptop computer, a desktop computer, a smart speaker, a smart watch, etc. The terminal 102 may also be a vehicle-mounted terminal of various device types described above, but is not limited thereto. The terminal 102 and the server 101 may be directly or indirectly connected via wired or wireless communication, which is not limited in this embodiment of the present invention.

[0068] Based on the example Figure 1 In the implementation environment shown, an embodiment of the present invention provides a non-line-of-sight imaging system and reconstruction method based on a TOF sensor and a laser. The following is an example of the non-line-of-sight imaging system and reconstruction method based on a TOF sensor and a laser being applied to a server 101. It can be understood that the method can also be applied to a terminal 102.

[0069] Reference Figure 2 , Figure 2 The flowchart of the non-line-of-sight imaging system and reconstruction method based on TOF sensor and laser applied to the server provided by the embodiment of the present invention, the execution subject of the method can be any of the aforementioned computer devices (including servers or terminals). Figure 2 , the method may include the following steps:

[0070] Building a non-line-of-sight imaging scene, wherein the non-line-of-sight imaging scene is composed of a TOF sensor, a laser, an intermediate surface, and an imaging target;

[0071] Collecting a non-viewing area data set, wherein the non-viewing area data set consists of an unprocessed original depth-degraded image and a corresponding depth-clear image;

[0072] According to the non-line-of-sight data set, a three-dimensional reconstruction model based on a variable-scale encoding-decoding architecture of a deep neural network is constructed for reconstructing non-line-of-sight targets, and the preprocessed non-line-of-sight data set is used to complete the training of the three-dimensional reconstruction model;

[0073] After the training of the 3D reconstruction model is completed, the depth-degraded image to be reconstructed is input into the 3D reconstruction model for 3D reconstruction, and a depth-reconstructed image of the non-viewing area scene is output.

[0074] In some embodiments, the step of setting up a non-line-of-sight imaging scene comprises the following steps:

[0075] The target is fully illuminated by the modulated light emitted by the illumination light source, and the TOF sensor realizes depth calculation imaging through the phase difference between the illumination light and the received light; wherein, the non-line-of-sight imaging system based on the TOF sensor is mainly composed of an illumination unit, a detector, a control unit, and a processor;

[0076] The imaging step of the non-line-of-sight imaging system comprises the following steps:

[0077] Driven by the synchronization signal, the laser emits high-frequency modulated light to illuminate the target, and the light beam hits the target object and is reflected;

[0078] The detector receives the reflected light signal and forms an image on the TOF sensor;

[0079] The memory temporarily stores the captured image and depth data;

[0080] The processor processes the image and depth data, connects to the external computing device through the data interface, and transmits the data to the computer device for processing and storage.

[0081] In some embodiments, the collecting of the non-viewing area dataset comprises the following steps:

[0082] Use TOF sensor and laser to image the target and obtain the depth information of each pixel of the target;

[0083] The non-viewing area imaging scene is used to collect depth image data, and during the collection process, the interface between the intermediate surface and the imaging target is replaced with a mirror;

[0084] A rotating stage is placed under the imaging target, and the features of the imaging target at different angles can be obtained by rotating it one circle.

[0085] In some embodiments, building a 3D reconstruction model based on a variable scale encoding-decoding architecture of a deep neural network for reconstructing non-line-of-sight targets according to the non-line-of-sight dataset, and using the preprocessed non-line-of-sight dataset to complete the training of the 3D reconstruction model comprises the following steps:

[0086] On the encoder side, ResNet-50 is used as the feature extraction network, and the feature extraction network introduces the bottleneck residual module to construct the residual relationship between the input and the output; wherein, the feature extraction network is divided into 5 stages, wherein the first stage inputs the preprocessed deep degraded image into the feature extraction network to extract multi-layer features, and the next four stages contain 3, 4, 6, and 3 bottleneck residual modules respectively;

[0087] At the decoder end, the channel attention module is used to process the effective features obtained in each stage respectively; the specific process is as follows: the features extracted in the previous stage are upsampled after passing through the channel attention module, and finally added with the features extracted in the next stage as the input of the corresponding channel attention module in the next stage, and skip connections are used to achieve multi-scale effective feature fusion; at the end of the decoder, the features decoded by channel attention in each stage are upsampled, and then the features are fused to output the final deep reconstructed image.

[0088] In some embodiments, the method further comprises:

[0089] The 3D reconstruction model performs depth data preprocessing on the non-viewing area dataset before training, specifically: inputting a depth-degraded image with a size of 192×192 and outputting a depth-reconstructed image with a size of 96×96;

[0090] The MATLAB programming language is used to implement the non-line-of-sight 3D reconstruction model. The loss function is set as the mean square error. The Adam optimizer is used to backpropagate the loss function to minimize the loss function value. Finally, the depth-degraded images of the test set are sent to the trained model for 3D reconstruction, achieving high-quality 3D reconstruction in data-driven non-line-of-sight scenarios.

[0091] The implementation process of the embodiment of the present invention in a specific application scenario is described in detail below with reference to the accompanying drawings of the specification:

[0092] In view of the problems existing in the prior art, the embodiments of the present invention provide a non-line-of-sight imaging system based on a TOF sensor, which is 1 to 2 orders of magnitude lower in cost than transient imaging and range-gated imaging systems, and can achieve an imaging speed of 10 to 15 frames per second without sacrificing imaging quality. Compared with passive imaging systems, the present invention uses a lightweight artificial intelligence algorithm to reconstruct clear three-dimensional images.

[0093] The TOF camera works synchronously with the sensor and the illumination light source to achieve the effect of imaging the depth of the target. The illumination light uses a modulated laser with an operating frequency of 15MHz to 60MHz. It has the characteristics of high frequency and high power and is a high-frequency and high-power laser. In view of the working characteristics of the TOF sensor, the present invention designs a high-frequency, high-power laser with uniform light spot and adjustable size to meet the non-field of view illumination requirements.

[0094] The imaging system of the embodiment of the present invention is as follows Figure 7 As shown in the figure, the imaging system can realize three-dimensional imaging of the target within the field of view after synchronous operation, and the interference of the intermediate surface is increased for the non-field of view target, such as Figure 3 In the non-line-of-sight imaging problem, the intermediary surface has a direct impact on the imaging result. The illumination light illuminates the target through the intermediary surface. The surface optical reflection characteristics of the target cause the illumination light energy to not be fully returned. The light energy received by the TOF sensor is obtained after the illumination light is scattered twice, including the scattering of the target and the scattering of the intermediary surface. The light intensity received by a certain pixel can be expressed as: p (x,y)=f bs g bs I1. Where I1 is the illumination light intensity, f bs and g bs are the scattering distribution characteristics related to the intermediary surface and the target surface, respectively. p (x,y) is the actual intensity received by a pixel p(x,y) on the sensor.

[0095] The present invention designs a SMM neural network model with a variable scale encoding-decoding structure. The model uses an encoder-decoder neural network that combines a deep residual network with a channel attention to extract effective information from a depth-degraded image, and finally realizes the three-dimensional reconstruction of a non-field of view target through multi-scale feature fusion. The model can reconstruct the contour of the imaging target from a completely blurred depth-degraded image with fewer shadows. The non-field of view three-dimensional reconstruction method proposed in the present invention has the characteristics of good reconstruction effect, strong applicability, and low cost, and provides a new solution to the reconstruction problem of depth image degradation caused by intermediate surface scattering in non-field of view scenes.

[0096] The non-line-of-sight imaging system and reconstruction method and imaging system based on TOF sensor and laser provided by the present invention include the following steps:

[0097] Build a non-line-of-sight imaging scene, which consists of a TOF sensor, a laser, an intermediate surface, and an imaging target.

[0098] A non-viewing area dataset is collected, which consists of the unprocessed original depth-degraded images and the corresponding depth-clear images.

[0099] A 3D reconstruction model SMM with a variable-scale encoding-decoding architecture based on a deep neural network is built to reconstruct non-viewing targets, and the preprocessed depth data is used to complete the model training.

[0100] The SMM network model is trained. After the training converges, the tested depth-degraded image is input into the network for 3D reconstruction, and finally the depth-reconstructed image of the non-viewing scene is output.

[0101] The implementation process of each step is described in detail below:

[0102] 1. Build a non-line-of-sight imaging scene, which consists of a TOF sensor, an illumination source, an intermediate surface, obstacles, and an imaging target. Figure 4 This is a schematic diagram of the non-line-of-sight imaging scenario of the present invention. The simulated non-line-of-sight imaging in this scenario can achieve the effect of observing the movement of the target person. In a military confrontation scenario, for enemy reconnaissance, combat and other military activities, the enemy situation around corners or behind doors can be observed with the help of walls, etc., which can effectively assist combatants in self-protection and eliminating the enemy.

[0103] The illumination light source emits modulated light to fully illuminate the target, and the TOF sensor realizes depth calculation imaging through the phase difference between the illumination light and the received light. The non-line-of-sight imaging system based on the TOF sensor is mainly composed of an illumination unit, a detector, a control unit, and a processor. The imaging steps are as follows: (1) The laser, driven by a synchronization signal, emits high-frequency modulated light to illuminate the target, and the light beam illuminates the target object and is reflected; (2) The detector receives the reflected light signal and simultaneously forms an image on the TOF sensor. The optical lens is equipped with a bandpass filter to ensure that only light with the same wavelength as the illumination light source enters, while reducing noise and preventing the photosensitive sensor from being overexposed due to external light interference. It has superior performance in terms of imaging quality, resolution, and color reproduction, and presents high definition; (3) The memory temporarily stores the captured image and depth data; (4) The processor processes the data, connects to an external computing device (PC) through a data interface (USB), and transmits the data to the computer device for processing and storage. The variable scale coding-decoding structure SMM neural network model designed by the present invention is used for training, and the tested depth-degraded image is input into the network for three-dimensional reconstruction, and finally the depth reconstruction image of the non-line-of-sight target is realized.

[0104] Specifically, some parameters of the non-viewing area imaging system of the embodiment of the present invention are shown in Table 1, and some parameters of the optical lens are shown in Table 2:

[0105] Table 1

[0106] name parameter TOF Sensor OPT8421 Controller OPT9221

[0107] Table 2

[0108] name parameter focal length 12mm Field of view size 26.2°×22.1° N / A 0.4 Target size 1 / 3”

[0109] The illumination light source laser of the present invention provides a light source with high precision, long distance and good anti-interference ability. Its monochromaticity and directionality enable the light beam to maintain high intensity and clarity in complex environments. The laser wavelength is 850nm. Under the drive of the synchronization signal, the light beam is usually emitted in the form of a single pulse, which is conducive to reducing the influence of background noise and improving the ranging accuracy. The laser of the present invention belongs to multi-modal fiber output, supports multiple optical modes, presents a uniform circular light spot and adjustable distance, generally with an adjustment range of 5m to 30m, and can effectively irradiate the target object at a long distance. The illumination light source laser emits a laser beam, which is controlled by an external trigger signal, with a frequency of 15MHz to 60MHz, a single pulse time of 8.34 to 33.35ns, and provides 8w of power during the single pulse duration. The single pulse energy is about 30 to 300nJ. The laser of the present invention can accurately measure the propagation time of light, effectively obtain the distance and shape information of the target object, and realize high-precision three-dimensional imaging. The receiving module is used to receive the reflected light signal. For imaging targets, the intermediate surface uses common materials in nature as the reflecting surface. During non-field of view imaging, the laser is triggered by an external signal to emit laser light, and the lens receives the return light signal to the TOF sensor. The incident point of the illumination light on the intermediate surface needs to avoid the sensor's receiving field of view.

[0110] 2. Collect a non-field of view data set, which consists of an unprocessed original depth-degraded image and a corresponding deep clear image. Use a TOF sensor and a laser to image the imaging target and obtain the depth information of each pixel of the target. Since the neural network requires a large amount of data to learn the intrinsic relationship between input and output, the depth image data is collected in the non-field of view imaging scene of step 1. The resolution of the image is 240×320. In the process of collecting data, it is only necessary to change the interface between the intermediate surface and the imaging target. When the interface is a mirror, the light signal can be effectively reflected to achieve point-to-point clear imaging. Therefore, the depth image collected under the intermediate surface is regarded as the label data corresponding to the training data. When the intermediate surface is a diffuse reflection plate, the light signal becomes blurred due to the scattering effect of diffuse reflection, so the depth image collected under the intermediate surface is regarded as the training data. In order to increase the diversity of the data, that is, the imaging target needs to have different shapes, angles, and quantities, a rotating table is placed under the imaging target, and the features of the imaging target at different angles are obtained by rotating one circle. The rotating platform can obtain 850 depth images at different angles after one rotation. There are 12 imaging targets in total, and a total of 850×12=10200 depth maps are obtained. In order to simulate various interferences in real application scenarios, the obtained depth maps are rotated 90°, 180°, and 270° respectively, and then the depth maps are expanded to 40,800 depth maps, which meets the requirements of deep learning training data.

[0111] A 3D reconstruction model SMM with a variable-scale encoding-decoding architecture based on a deep neural network is built to reconstruct non-viewing targets, and the preprocessed depth data is used to complete the model training. Figure 5 It is represented as the overall structure of the SMM model. The specific implementation steps are as follows:

[0112] At the encoder end, in order to more effectively extract effective information from deep degraded images, the feature extraction network used in the present invention is ResNet-50, which introduces a bottleneck residual module composition, constructs a residual relationship between input and output, optimizes parameters more effectively, and solves the overfitting problem of the network while deepening the network depth. The feature extraction network is divided into 5 stages. In the first stage, the preprocessed deep degraded image is input into the feature extraction network to extract multi-layer features. The last four stages contain 3, 4, 6, and 3 bottleneck residual modules, respectively, and the channel dimensions of each stage are 64, 128, 256, and 512, respectively. In an embodiment of the present invention, a deep degraded image of size 192×192 is input, and the feature map shapes obtained by the last four stages of ResNet-50 are: (64, 48, 48), (128, 24, 24), (256, 12, 12), and (512, 6, 6).

[0113] At the decoder end, the channel attention module is applied to the effective features obtained in each stage. The features extracted in the previous stage are upsampled after the channel attention module, and finally added to the features extracted in the next stage as the input of the corresponding channel attention module in the next stage. The jump connection is used to achieve multi-scale effective feature fusion. At the end of the decoder, the features decoded by the channel attention in each stage are upsampled, and then the features are fused to output the final deep reconstructed image of size 96×96.

[0114] 4. Train the designed improved network. After the training converges, input the tested deep degraded image into the network for reconstruction, and finally output the deep reconstructed image of the non-field of view scene. Before training, the SMM model performs depth data preprocessing on the non-field of view imaging data set according to the method in step 3, inputs a deep degraded image of size 192×192, and outputs a deep reconstructed image of size 96×96. The method of the present invention uses the MATLAB programming language to implement the non-field of view 3D reconstruction model. The model training is run on the Windows 11 operating system and NVIDIA GeForce RTX4090, and the training time is 10 hours. The data set is randomly divided into a training set, a validation set, and a test set in a ratio of 8:1:1. The network is trained with the training set to obtain a trained model. The batch size of the training is set to 128, and a total of 150 rounds of training are performed. The loss function is set to mean square error (MSE), and the Adam optimizer is used to backpropagate the loss function to minimize the loss function value. Finally, the deep degraded image of the test set is sent to the trained model for 3D reconstruction, achieving high-quality 3D reconstruction in data-driven non-field of view scenes. Figure 6 Shown is the reconstruction result of the SMM model.

[0115] In summary, the present invention has built a complete set of three-dimensional reconstruction methods and system real-time imaging solutions for non-line-of-sight targets based on TOF sensors, which not only realizes the application of non-line-of-sight in natural scenes, but also has low cost and fast imaging speed compared to transient imaging and range-gated imaging systems. At the same time, without losing imaging quality, compared to passive imaging systems, the present invention uses lightweight artificial intelligence algorithms. Compared to two-dimensional cameras, the non-line-of-sight system imaging solution of the present invention can reconstruct clear three-dimensional depth images. At the same time, the model in the present invention uses an encoder-decoder neural network that combines a deep residual network with channel attention to extract effective information from deep degraded images, and finally realizes three-dimensional reconstruction of non-line-of-sight targets through multi-scale feature fusion. The TOF sensor non-line-of-sight system imaging solution proposed in the present invention has strong applicability and low cost, and is more practical in military confrontation scenarios such as enemy reconnaissance and scientific exploration. It has made further optimizations in the lighting module, receiving module and imaging processing.

[0116] Another aspect of the embodiments of the present invention further provides a non-line-of-sight imaging system using a TOF sensor and a laser, comprising:

[0117] The first module is used to build a non-line-of-sight imaging scene, which consists of a TOF sensor, a laser, an intermediate surface, and an imaging target;

[0118] The second module is used to collect a non-viewing area dataset, where the non-viewing area dataset consists of an unprocessed original depth-degraded image and a corresponding depth-clear image;

[0119] The third module is used to build a 3D reconstruction model based on a variable scale coding-decoding architecture of a deep neural network according to the non-line-of-sight data set for reconstructing non-line-of-sight targets, and use the preprocessed non-line-of-sight data set to complete the training of the 3D reconstruction model;

[0120] The fourth module is used to input the depth-degraded image to be reconstructed into the three-dimensional reconstruction model for three-dimensional reconstruction after completing the training of the three-dimensional reconstruction model, and output the depth-reconstructed image of the non-viewing area scene.

[0121] In some embodiments, the second module is specifically used to:

[0122] Use TOF sensor and laser to image the target and obtain the depth information of each pixel of the target;

[0123] The non-viewing area imaging scene is used to collect depth image data, and during the collection process, the interface between the intermediate surface and the imaging target is replaced with a mirror;

[0124] A rotating stage is placed under the imaging target, and the features of the imaging target at different angles can be obtained by rotating it one circle.

[0125] It can be understood that the contents of the above method embodiments are all applicable to the present system embodiments, the functions specifically implemented by the present system embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0126] The embodiment of the present invention further provides an electronic device, the electronic device includes a memory and a processor, the memory stores a computer program, and the processor implements the above-mentioned non-line-of-sight imaging system and reconstruction method based on TOF sensor and laser when executing the computer program. The electronic device can be any intelligent terminal including a tablet computer, a car computer, etc.

[0127] It can be understood that the contents of the above method embodiments are all applicable to the present device embodiments, the functions specifically implemented by the present device embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0128] See also Figure 8 , Figure 8 The hardware structure of an electronic device of another embodiment is illustrated, and the electronic device includes:

[0129] The processor 801 may be implemented by a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of the present invention;

[0130] The memory 802 can be implemented in the form of a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 802 can store an operating system and other application programs. When the technical solution provided in the embodiments of this specification is implemented by software or firmware, the relevant program code is stored in the memory 802, and the processor 801 calls and executes the non-line-of-sight imaging system and reconstruction method based on TOF sensor and laser in the embodiments of the present invention;

[0131] Input / output interface 803, used to implement information input and output;

[0132] The communication interface 804 is used to realize the communication interaction between the device and other devices. The communication can be realized through a wired manner (such as USB, network cable, etc.) or a wireless manner (such as mobile network, WIFI, Bluetooth, etc.);

[0133] A bus 805 that transmits information between the various components of the device (e.g., the processor 801, the memory 802, the input / output interface 803, and the communication interface 804);

[0134] The processor 801 , the memory 802 , the input / output interface 803 and the communication interface 804 are connected to each other in communication within the device via a bus 805 .

[0135] An embodiment of the present invention further provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the non-line-of-sight imaging system and reconstruction method based on a TOF sensor and a laser.

[0136] It can be understood that the contents of the above method embodiments are all applicable to the present storage medium embodiments, the functions specifically implemented by the present storage medium embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0137] The memory, as a non-transient computer-readable storage medium, can be used to store non-transient software programs and non-transient computer executable programs. In addition, the memory may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some embodiments, the memory may optionally include a memory remotely disposed relative to the processor, and these remote memories may be connected to the processor via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0138] It should be noted that in various specific embodiments of the present invention, when it comes to the need to perform relevant processing based on data related to user identity or characteristics such as user information, user behavior data, user historical data, and user location information, the user's permission or consent will be obtained first, and the collection, use, and processing of these data will comply with relevant laws, regulations, and standards. In addition, when the embodiment of the present invention needs to obtain the user's sensitive personal information, it will obtain the user's separate permission or consent through a pop-up window or jump to a confirmation page, and after clearly obtaining the user's separate permission or consent, it will obtain the necessary user-related data for the normal operation of the embodiment of the present invention.

[0139] The embodiments described in the embodiments of the present invention are intended to more clearly illustrate the technical solutions of the embodiments of the present invention, and do not constitute a limitation on the technical solutions provided by the embodiments of the present invention. Those skilled in the art can appreciate that with the evolution of technology and the emergence of new application scenarios, the technical solutions provided by the embodiments of the present invention are also applicable to similar technical problems.

[0140] Those skilled in the art will appreciate that the technical solutions shown in the figures do not limit the embodiments of the present invention and may include more or fewer steps than shown in the figures, or a combination of certain steps, or different steps.

[0141] The device embodiments described above are merely illustrative, and the units described as separate components may or may not be physically separated, that is, they may be located in one place or distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0142] Those skilled in the art will appreciate that all or some of the steps in the methods disclosed above, and the functional modules / units in the systems and devices may be implemented as software, firmware, hardware, or a suitable combination thereof.

[0143] The terms "first", "second", "third", "fourth", etc. (if any) in the specification of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0144] It should be understood that in the present invention, "at least one (item)" refers to one or more, and "plurality" refers to two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can represent: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the previous and next associated objects are in an "or" relationship. "At least one of the following items" or similar expressions refers to any combination of these items, including any combination of single items or plural items. For example, at least one of a, b or c can represent: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.

[0145] In the several embodiments provided by the present invention, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the above units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0146] The units described above as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0147] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0148] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a storage medium, including multiple instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the methods of various embodiments of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), disk or optical disk, etc., various media that can store programs.

[0149] The preferred embodiments of the present invention are described above with reference to the accompanying drawings, but the scope of the rights of the embodiments of the present invention is not limited thereby. Any modification, equivalent substitution and improvement made by those skilled in the art without departing from the scope and essence of the embodiments of the present invention shall be within the scope of the rights of the embodiments of the present invention.

Claims

1. A non-line-of-sight imaging system and reconstruction method based on TOF sensor and laser, characterized in that: The following steps are involved: Building a non-line-of-sight imaging scene, wherein the non-line-of-sight imaging scene is composed of a TOF sensor, a laser, an intermediate surface, and an imaging target; Collecting a non-viewing area data set, wherein the non-viewing area data set consists of an unprocessed original depth-degraded image and a corresponding depth-clear image; According to the non-line-of-sight data set, a three-dimensional reconstruction model based on a variable-scale encoding-decoding architecture of a deep neural network is constructed for reconstructing non-line-of-sight targets, and the preprocessed non-line-of-sight data set is used to complete the training of the three-dimensional reconstruction model; After the training of the 3D reconstruction model is completed, the depth-degraded image to be reconstructed is input into the 3D reconstruction model for 3D reconstruction, and a depth-reconstructed image of the non-viewing area scene is output.

2. According to claim 1, a non-line-of-sight imaging system and reconstruction method based on a TOF sensor and a laser, characterized in that: The process of constructing a non-viewing area imaging scene comprises the following steps: The target is fully illuminated by the modulated light emitted by the illumination light source, and the TOF sensor realizes depth calculation imaging through the phase difference between the illumination light and the received light; wherein, the non-line-of-sight imaging system based on the TOF sensor is mainly composed of an illumination unit, a detector, a control unit, and a processor; The imaging step of the non-line-of-sight imaging system comprises the following steps: Driven by the synchronization signal, the laser emits high-frequency modulated light to illuminate the target, and the light beam hits the target object and is reflected; The detector receives the reflected light signal and forms an image on the TOF sensor; The memory temporarily stores the captured image and depth data; The processor processes the image and depth data, connects to the external computing device through the data interface, and transmits the data to the computer device for processing and storage.

3. According to claim 1, a non-line-of-sight imaging system and reconstruction method based on a TOF sensor and a laser, characterized in that: The collecting of the non-viewing area data set comprises the following steps: Use TOF sensor and laser to image the target and obtain the depth information of each pixel of the target; The non-viewing area imaging scene is used to collect depth image data, and during the collection process, the interface between the intermediate surface and the imaging target is replaced with a mirror; A rotating stage is placed under the imaging target, and the features of the imaging target at different angles can be obtained by rotating it one circle.

4. The non-line-of-sight imaging system and reconstruction method based on TOF sensor and laser according to claim 1, characterized in that: The method of building a three-dimensional reconstruction model based on a variable-scale encoding-decoding architecture of a deep neural network for reconstructing a non-line-of-sight target according to the non-line-of-sight dataset, and using the preprocessed non-line-of-sight dataset to complete the training of the three-dimensional reconstruction model, comprises the following steps: On the encoder side, ResNet-50 is used as the feature extraction network, and the feature extraction network introduces the bottleneck residual module to construct the residual relationship between the input and the output; wherein, the feature extraction network is divided into 5 stages, wherein the first stage inputs the preprocessed deep degraded image into the feature extraction network to extract multi-layer features, and the next four stages contain 3, 4, 6, and 3 bottleneck residual modules respectively; At the decoder end, the channel attention module is used to process the effective features obtained in each stage respectively; the specific process is as follows: the features extracted in the previous stage are upsampled after passing through the channel attention module, and finally added with the features extracted in the next stage as the input of the corresponding channel attention module in the next stage, and skip connections are used to achieve multi-scale effective feature fusion; at the end of the decoder, the features decoded by channel attention in each stage are upsampled, and then the features are fused to output the final deep reconstructed image.

5. The non-line-of-sight imaging system and reconstruction method based on TOF sensor and laser according to claim 1, characterized in that: The method further comprises: The 3D reconstruction model performs depth data preprocessing on the non-viewing area dataset before training, specifically: inputting a depth-degraded image with a size of 192×192 and outputting a depth-reconstructed image with a size of 96×96; The MATLAB programming language is used to implement the non-line-of-sight 3D reconstruction model. The loss function is set as the mean square error. The Adam optimizer is used to backpropagate the loss function to minimize the loss function value. Finally, the depth-degraded images of the test set are sent to the trained model for 3D reconstruction, achieving high-quality 3D reconstruction in data-driven non-line-of-sight scenarios.

6. A non-line-of-sight imaging system using a TOF sensor and a laser, characterized in that: include: The first module is used to build a non-line-of-sight imaging scene, which consists of a TOF sensor, a laser, an intermediate surface, and an imaging target; The second module is used to collect a non-viewing area dataset, where the non-viewing area dataset consists of an unprocessed original depth-degraded image and a corresponding depth-clear image; The third module is used to build a 3D reconstruction model based on a variable scale coding-decoding architecture of a deep neural network according to the non-line-of-sight data set for reconstructing non-line-of-sight targets, and use the preprocessed non-line-of-sight data set to complete the training of the 3D reconstruction model; The fourth module is used to input the depth-degraded image to be reconstructed into the three-dimensional reconstruction model for three-dimensional reconstruction after completing the training of the three-dimensional reconstruction model, and output the depth-reconstructed image of the non-viewing area scene.

7. A non-line-of-sight imaging system using a TOF sensor and a laser according to claim 6, characterized in that: The second module is specifically used for: Use TOF sensor and laser to image the target and obtain the depth information of each pixel of the target; The non-viewing area imaging scene is used to collect depth image data, and during the collection process, the interface between the intermediate surface and the imaging target is replaced with a mirror; A rotating stage is placed under the imaging target, and the features of the imaging target at different angles can be obtained by rotating it one circle.

8. An electronic device, characterized in that: including a processor and a memory; The memory is used to store programs; The processor executes the program to implement the method according to any one of claims 1 to 5.

9. A computer-readable storage medium, characterized in that: The storage medium stores a program, and the program is executed by a processor to implement the method according to any one of claims 1 to 5.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the method according to any one of claims 1 to 5 is implemented.

Images