Depth map denoising method, device, medium and equipment

Abstract

The present application belongs to the field of computer technology, and specifically relates to a method, device, medium and equipment for denoising a depth map. Embodiments of the present invention can be applied to various scenarios such as artificial intelligence, smart transportation, and assisted driving. The method includes: dividing the depth map into one or more depth regions according to the size of each depth value in the depth map, projecting the depth region along a first direction on the depth region to obtain a projection map corresponding to the depth region, performing noise identification on the projection map to obtain a projection noise region in the projection map, obtaining a depth noise region corresponding to the projection noise region in the depth region, and denoising each depth noise region of the depth map to obtain a denoised depth map. Based on the present application, the projection noise region can be efficiently identified through the distribution of depth values ​​in the projection map, thereby achieving accurate denoising of the depth map, and improving the denoising efficiency of the depth map.

Description

Technical Field

[0001] The present application belongs to the field of computer technology, and specifically relates to a method, apparatus, medium, and device for denoising a depth map. Background Art

[0002] In real-world scenarios, depth images acquired by depth cameras often contain some noise, which significantly impacts the accuracy of the depth image and the authenticity of the corresponding 3D model. In the process of identifying the depth features of the captured object based on the depth image, noise can negatively impact the accuracy of feature recognition of the captured object.

[0003] The depth image denoising methods in related technologies have large computational complexity and low accuracy, and are difficult to meet the high-speed response requirements of outputting accurate depth image results in real time.

[0004] Therefore, how to achieve efficient and accurate noise removal in depth maps is a technical problem that needs to be solved urgently.

[0005] It should be noted that the information disclosed in the above background technology section is only used to enhance the understanding of the background of this application, and therefore may include information that does not constitute prior art known to ordinary technicians in this field. Summary of the Invention

[0006] The purpose of this application is to provide a depth map denoising method, device, medium and equipment, which at least to some extent solve the technical problem of how to achieve efficient and accurate noise removal in depth maps in the related art.

[0007] Other features and advantages of the present application will become apparent from the following detailed description, or may be learned in part by practice of the present application.

[0008] According to one aspect of an embodiment of the present application, a method for denoising a depth map is provided, the method comprising:

[0009] Dividing the depth map into one or more depth regions according to the magnitude of each depth value in the depth map, wherein each image position in each depth region has a similar depth value;

[0010] Projecting the depth region along a first direction on the depth region to obtain a projection map corresponding to the depth region, the projection map being used to represent a distribution of depth values ​​of the depth region corresponding to a second direction, where the second direction is a direction on the depth region that is different from the first direction;

[0011] performing noise identification on the projection image to obtain a projection noise area in the projection image;

[0012] A depth noise region corresponding to the projection noise region in the depth region is acquired, and a denoising process is performed on each depth noise region in the depth map to obtain a denoised depth map.

[0013] According to one aspect of an embodiment of the present application, a depth map denoising apparatus is provided, the apparatus comprising:

[0014] a depth region division module configured to divide the depth map into one or more depth regions according to the magnitude of each depth value in the depth map, wherein each image position in each depth region has a similar depth value;

[0015] a projection map acquisition module configured to project the depth region along a first direction on the depth region to obtain a projection map corresponding to the depth region, the projection map being used to represent a distribution of depth values ​​of the depth region corresponding to a second direction, where the second direction is a direction on the depth region that is different from the first direction;

[0016] a projection noise region determination module, configured to perform noise identification on the projection image to obtain a projection noise region in the projection image;

[0017] The denoising module is configured to obtain a depth noise region corresponding to the projection noise region in the depth region, and perform denoising on each depth noise region of the depth map to obtain a denoised depth map.

[0018] In some embodiments of the present application, based on the above technical solution, the projection image acquisition module includes:

[0019] a sliding window establishing unit, configured to establish a sliding window according to a first preset size;

[0020] a depth region intercepting unit configured to slide the sliding window along the first direction in the depth region according to a preset step size, and intercept the depth region using the sliding window at each sliding position to obtain a sub-region corresponding to the sliding window at each sliding position;

[0021] a projection processing unit configured to perform projection processing on the sub-regions along the first direction to obtain sub-projection images corresponding to the sub-regions; wherein the sub-projection images are used to represent the distribution of depth values ​​of the sub-regions corresponding to the second direction;

[0022] The projection image acquisition unit is configured to merge the obtained multiple sub-projection images into the projection image along the sliding direction of the sliding window.

[0023] In some embodiments of the present application, based on the above technical solution, the projection processing unit includes:

[0024] a three-dimensional conversion subunit configured to convert the depth map into a three-dimensional space point cloud using a three-dimensional coordinate system, wherein the three-dimensional coordinate system includes a first plane coordinate axis, a second plane coordinate axis, and a depth value coordinate axis;

[0025] a window point cloud acquisition subunit, configured to acquire a window point cloud corresponding to the sub-region in the three-dimensional space point cloud;

[0026] The projection processing subunit is configured to use the direction of the first plane coordinate axis as a first direction, project the window point cloud along the first direction, and obtain a sub-projection image corresponding to the sub-region.

[0027] In some embodiments of the present application, based on the above technical solution, the depth region division module includes:

[0028] A depth value column acquisition unit is configured to acquire two columns of depth values ​​in the depth map that are adjacent to each other on the first plane coordinate axis to obtain a first depth value column and a second depth value column;

[0029] a depth value group acquiring unit configured to determine depth values ​​in the first depth value column and the second depth value column that have the same position on the second plane coordinate axis as a group of depth values;

[0030] a difference sum calculation unit configured to calculate a difference between two depth values ​​in each group of depth values, and perform a sum operation on the obtained multiple differences to obtain a difference sum;

[0031] The depth region division unit is configured to, when the sum of the difference values ​​is greater than a first preset threshold, use the first depth value column and the second depth value column as a dividing line between two depth regions, and divide the first depth value column into a first depth region and divide the second depth value column into a second depth region.

[0032] In some embodiments of the present application, based on the above technical solution, the projection noise area determination module includes:

[0033] a target sub-projection image determining unit configured to determine, as a target sub-projection image, a sub-projection image corresponding to an area where depth values ​​with value jumps are concentrated in the projection image;

[0034] The projection noise area determining unit is configured to determine an area formed by a plurality of the target sub-projection images as the projection noise area.

[0035] In some embodiments of the present application, based on the above technical solution, the target sub-projection graph determining unit includes:

[0036] a depth value sequence acquisition subunit, configured to arrange the depth values ​​in the sub-projection image in descending or ascending order to obtain a depth value sequence;

[0037] a depth value extraction subunit, configured to extract a plurality of depth values ​​from the depth value sequence at intervals of a preset number;

[0038] The difference calculation subunit is configured to arrange the extracted multiple depth values ​​in order from large to small or from small to large, and calculate the difference between every two adjacent depth values;

[0039] The target sub-projection image determining sub-unit is configured to determine the sub-projection image as a target sub-projection image when the difference is greater than a second preset threshold.

[0040] In some embodiments of the present application, based on the above technical solution, the denoising module includes:

[0041] a first mean value acquisition unit configured to establish a judgment box of a second preset size on the depth noise region, intercept a region to be judged in the depth noise region through the judgment box, and calculate a mean of the region to be judged to obtain a first mean value; the region to be judged includes a center point, which is a pixel point located at a preset position in the region to be judged;

[0042] a second mean value obtaining unit configured to obtain neighboring points adjacent to the center point, and calculate the mean of the to-be-determined area corresponding to each neighboring point of the center point to obtain a second mean value;

[0043] an abnormal gradient value determining unit, configured to calculate gradient values ​​of the first mean value and each of the second mean values, and determine a gradient value greater than a preset gradient value as an abnormal gradient value;

[0044] a noise region to be processed determination unit, configured to determine the region to be determined as a noise region to be processed when the number of the abnormal gradient values ​​is greater than a preset number;

[0045] The noise removal unit is configured to re-determine the depth value of each noise area to be processed in the depth map to obtain a denoised depth map.

[0046] In some embodiments of the present application, based on the above technical solution, the noise removal unit includes:

[0047] a depth value zeroing subunit, configured to set the depth value of each to-be-processed noise area in the depth map to zero;

[0048] The interpolation processing subunit is configured to perform interpolation processing on the noise area to be processed according to the depth values ​​around the noise area to be processed, so as to obtain a denoised depth map.

[0049] In some embodiments of the present application, based on the above technical solution, the depth map denoising device further includes:

[0050] a first ratio calculation unit configured to calculate a first ratio of the number of abnormal gradient values ​​obtained in the noise region to be processed to the number of all gradient values ​​obtained in the noise region to be processed;

[0051] The first confidence determination unit is configured to determine a first confidence according to the first ratio, where the first confidence is used to indicate a reliability level of accuracy in determining the noise region to be processed.

[0052] In some embodiments of the present application, based on the above technical solution, the depth map denoising device further includes:

[0053] a target number determining unit configured to determine the number of targets in the noise region to be processed in which the first confidence level is lower than a first preset confidence level in the depth region;

[0054] a second ratio calculation unit configured to calculate a second ratio of the target number to the number of all noise regions to be processed in the depth region;

[0055] a second confidence determining unit configured to determine a second confidence based on the second ratio, wherein the second confidence is used to indicate a degree of reliability of an accurate determination of the noise region to be processed in the depth region;

[0056] The depth region determination unit requiring review is configured to mark the depth region as a depth region requiring review of accuracy of determination of the noise region to be processed when the second confidence level is lower than a second preset confidence level.

[0057] In some embodiments of the present application, based on the above technical solution, the depth map denoising device further includes:

[0058] an average value calculation unit, configured to calculate an average value of each depth value in the depth map;

[0059] The depth value removal unit is configured to remove depth values ​​in the depth map whose difference from the average value is greater than a preset difference.

[0060] In some embodiments of the present application, based on the above technical solution, the depth map is a depth map obtained by performing depth imaging of the container using a line scan camera, and the first direction is a scanning direction of the line scan camera.

[0061] According to one aspect of an embodiment of the present application, a computer-readable medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the depth map denoising method in the above technical solution is implemented.

[0062] According to one aspect of an embodiment of the present application, an electronic device is provided, comprising: a processor; and a memory for storing executable instructions of the processor; wherein the processor is configured to execute the depth map denoising method in the above technical solution by executing the executable instructions.

[0063] According to one aspect of an embodiment of the present application, a computer program product or computer program is provided, comprising computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the depth map denoising method described in the above technical solution.

[0064] In the technical solution provided in the embodiment of the present application, the depth map is divided into one or more depth regions according to the size of each depth value in the depth map, the depth region is projected along a first direction on the depth region to obtain a projection map corresponding to the depth region, noise identification is performed on the projection map to obtain a projection noise region in the projection map, a depth noise region corresponding to the projection noise region in the depth region is obtained, and each depth noise region in the depth map is denoised to obtain a denoised depth map; thus, dividing the depth map into one or more depth regions according to the size of each depth value in the depth map can prevent image distortion during depth imaging from interfering with subsequent noise identification, and can improve the accuracy of noise region identification; the distribution of depth values ​​in the projection map can be used to quickly and accurately identify the projection noise region, thereby locking the noise range in the depth map; and by denoising the depth noise region in the depth map, accurate denoising of the depth map can be achieved without performing a full-domain traversal denoising process on the depth map, thereby improving the denoising efficiency of the depth map.

[0065] It should be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0066] The accompanying drawings are incorporated into and constitute a part of the specification, illustrate embodiments consistent with the present application, and together with the specification, are used to explain the principles of the present application. Obviously, the drawings described below are only some embodiments of the present application, and those skilled in the art can derive other drawings based on these drawings without inventive effort.

[0067] Figure 1 The following schematically shows an exemplary device architecture block diagram applying the technical solution of the present application.

[0068] Figure 2 The flowchart schematically shows the steps of a depth map denoising method according to some embodiments of the present application.

[0069] Figure 3 A schematic diagram of a scenario in which a depth map of a container is divided into multiple depth areas according to the sizes of the depth values ​​in the depth map is schematically shown in an embodiment of the present application.

[0070] Figure 4 The flowchart schematically shows the steps of dividing a depth map into one or more depth regions according to the sizes of the depth values ​​in the depth map in an embodiment of the present application.

[0071] Figure 5 The flowchart schematically shows the steps before dividing the depth map into one or more depth regions according to the sizes of the depth values ​​in the depth map in an embodiment of the present application.

[0072] Figure 6 The flowchart schematically shows the steps of performing projection processing on the depth area along a first direction on the depth area to obtain a projection image corresponding to the depth area in an embodiment of the present application.

[0073] Figure 7 The schematic diagram shows a scenario in which a sub-region is projected along a first direction to obtain a sub-projection image corresponding to each sub-region, and the obtained multiple sub-projection images are merged into a projection image along the sliding direction of the sliding window.

[0074] Figure 8 The flowchart schematically shows the steps of performing projection processing on sub-regions along a first direction to obtain sub-projection images corresponding to each sub-region in an embodiment of the present application.

[0075] Figure 9 The flowchart schematically shows the steps of performing noise recognition on a projection image and obtaining a projection noise area in the projection image in an embodiment of the present application.

[0076] Figure 10 The flowchart schematically shows the steps of determining the sub-projection image corresponding to the area where the depth values ​​with value jumps in the projection image are concentrated as the target sub-projection image in an embodiment of the present application.

[0077] Figure 11 The corresponding schematic diagram of the process of denoising the deep noise area in an embodiment of the present application is schematically shown on the projection image.

[0078] Figure 12 The flowchart schematically shows the steps of performing denoising on each depth noise area of ​​a depth map to obtain a denoised depth map in an embodiment of the present application.

[0079] Figure 13 The flowchart schematically shows the steps of re-determining the depth values ​​of each noise area to be processed in the depth map to obtain the denoised depth map in an embodiment of the present application.

[0080] Figure 14 The figure schematically shows a container depth map obtained according to another embodiment of the present application.

[0081] Figure 15 Schematically shows the Figure 14 Schematic diagram of projecting a partial depth area of ​​a depth map in the embodiment shown, identifying noise, and then denoising.

[0082] Figure 16 The following schematically shows a diagram of a container depth map obtained according to another embodiment of the present application.

[0083] Figure 17 Schematically shows the Figure 16 Schematic diagram of denoising after projection of a partial depth region of a depth map according to the embodiment shown.

[0084] Figure 18 The flowchart schematically shows the steps after determining the area to be determined as a noise area to be processed when the number of abnormal gradient values ​​obtained in the area to be determined is greater than a preset number in an embodiment of the present application.

[0085] Figure 19 The flowchart schematically shows the steps after determining the first confidence level according to the first ratio in a certain embodiment of the present application.

[0086] Figure 20 A partial process diagram of a depth map denoising method according to an embodiment of the present application is schematically shown.

[0087] Figure 21 The structural block diagram of the depth map denoising device provided in an embodiment of the present application is schematically shown.

[0088] Figure 22 The block diagram schematically shows a computer system structure of an electronic device used to implement an embodiment of the present application. DETAILED DESCRIPTION

[0089] Example embodiments will now be described more fully with reference to the accompanying drawings. However, example embodiments can be implemented in many forms and should not be construed as limited to the examples set forth herein; rather, these embodiments are provided so that this application will be thorough and complete and will fully convey the concepts of the example embodiments to those skilled in the art.

[0090] In addition, described feature, structure or characteristic can be combined in one or more embodiments in any suitable manner.In the following description, many specific details are provided so as to provide a full understanding of the embodiments of the present application. However, it will be appreciated by those skilled in the art that the technical scheme of the present application can be put into practice without one or more of the specific details, or other methods, components, devices, steps etc. can be adopted. In other cases, known methods, devices, implementations or operations are not shown or described in detail to avoid blurring the various aspects of the application.

[0091] The block diagrams shown in the accompanying drawings are merely functional entities and do not necessarily correspond to physically separate entities. That is, these functional entities may be implemented in software, in one or more hardware modules or integrated circuits, or in different networks and / or processor devices and / or microcontroller devices.

[0092] The flowcharts shown in the accompanying drawings are for illustrative purposes only and do not necessarily include all contents and operations / steps, nor must they be executed in the order described. For example, some operations / steps may be decomposed, while others may be combined or partially combined. Therefore, the actual execution order may vary depending on the actual situation.

[0093] Before providing a detailed description of the technical solutions such as the depth map denoising method and the depth map denoising device provided in the embodiments of the present application, a brief introduction to the artificial intelligence technology involved in some embodiments of the present application is first given.

[0094] Artificial Intelligence (AI) refers to the theories, methods, techniques, and applied devices that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, to perceive the environment, acquire knowledge, and use that knowledge to achieve optimal results. In other words, AI is a comprehensive technology within computer science that seeks to understand the essence of intelligence and produce new intelligent machines that can respond in a manner similar to human intelligence. AI also involves studying the design principles and implementation methods of various intelligent machines, enabling them to possess the capabilities of perception, reasoning, and decision-making.

[0095] Artificial intelligence (AI) technology is a comprehensive discipline encompassing a wide range of fields, encompassing both hardware and software technologies. Foundational AI technologies generally include sensors, specialized AI chips, cloud computing, distributed storage, big data processing, operating / interactive devices, and mechatronics. AI software technologies primarily encompass computer vision, speech processing, natural language processing, and machine learning / deep learning.

[0096] Computer vision (CV) is the science of making machines "see." Specifically, it refers to using cameras and computers to replace the human eye in identifying and measuring objects, and then further processing them to produce images more suitable for human observation or transmission to instruments. As a scientific discipline, computer vision studies related theories and technologies, attempting to build artificial intelligence systems capable of extracting information from images or multidimensional data. Computer vision technologies typically include image processing, image recognition, image semantic understanding, image retrieval, optical character recognition (OCR), video processing, video semantic understanding, video content / behavior recognition, three-dimensional object reconstruction, 3D technology, virtual reality, augmented reality, simultaneous localization and mapping, and common biometric recognition technologies such as facial recognition and fingerprint recognition.

[0097] Machine learning (ML) is a multidisciplinary field that encompasses probability theory, statistics, approximation theory, convex analysis, and algorithmic complexity theory. It specifically studies how computers can simulate or implement human learning behaviors to acquire new knowledge or skills and reorganize existing knowledge structures to continuously improve their performance. Machine learning is at the core of artificial intelligence and the fundamental way to make computers intelligent. Its applications span all areas of AI. Machine learning and deep learning typically include techniques such as artificial neural networks, belief networks, reinforcement learning, transfer learning, inductive learning, and self-learning.

[0098] With the research and advancement of artificial intelligence technology, artificial intelligence technology has been studied and applied in many fields, such as common smart homes, smart wearable devices, virtual assistants, smart speakers, smart marketing, unmanned driving, autonomous driving, drones, robots, smart medical care, smart customer service, etc. It is believed that with the development of technology, artificial intelligence technology will be applied in more fields and play an increasingly important role.

[0099] The system involved in the embodiments of the present application can be a distributed system formed by connecting a client and multiple nodes (any form of computing devices in the access network, such as servers and user terminals) through network communication.

[0100] The depth map denoising method and corresponding device provided in this application are described in detail below in conjunction with specific implementation methods.

[0101] Figure 1 The following schematically shows an exemplary device architecture block diagram applying the technical solution of the present application.

[0102] like Figure 1 As shown, device architecture 100 may include a terminal device 110, a network 120, and a server 130. Terminal device 110 may include various electronic devices such as smartphones, tablet computers, laptop computers, and desktop computers. Server 130 may be a standalone physical server, a server cluster or distributed device consisting of multiple physical servers, or a cloud server providing cloud computing services. Network 120 may be a communication medium of various connection types capable of providing a communication link between terminal device 110 and server 130, such as a wired communication link or a wireless communication link.

[0103] Depending on implementation needs, the device architecture in the embodiments of the present application can include any number of terminal devices, networks, and servers. For example, server 130 can be a server group consisting of multiple server devices. In addition, the technical solutions provided in the embodiments of the present application can be applied to terminal device 110, server 130, or can be implemented by both terminal device 110 and server 130, and this application does not impose any special restrictions on this.

[0104] For example, the server 130 can execute the depth map denoising method provided in the present application, divide the depth map into one or more depth regions according to the size of each depth value in the depth map, project the depth region along a first direction on the depth region to obtain a projection map corresponding to the depth region, perform noise identification on the projection map to obtain a projection noise region in the projection map, obtain a depth noise region corresponding to the projection noise region in the depth region, and denoise each depth noise region in the depth map to obtain a denoised depth map; thereby, dividing the depth map into one or more depth regions according to the size of each depth value in the depth map can prevent image distortion during depth imaging from interfering with subsequent noise identification, and can improve the accuracy of noise region identification; the projection noise region can be quickly and accurately identified through the distribution of depth values ​​in the projection map, thereby locking the noise range in the depth map; and then, by denoising the depth noise region in the depth map, accurate denoising of the depth map can be achieved without performing denoising on the depth map by a full-domain traversal, thereby improving the denoising efficiency of the depth map.

[0105] The depth map denoising method provided in this application is described in detail below in conjunction with specific implementation methods.

[0106] Figure 2 The following is a flowchart schematically illustrating the steps of a depth map denoising method according to certain embodiments of the present application. The execution entity of the depth map denoising method can be a terminal device or a server, etc., and this application does not limit this. Terminal devices include but are not limited to mobile phones, computers, intelligent voice interaction devices, smart home appliances, vehicle-mounted terminals, aircraft, etc. The embodiments of the present invention can be applied to various scenarios, including but not limited to cloud technology, artificial intelligence, smart transportation, assisted driving, etc.

[0107] like Figure 2 As shown, the depth map denoising method may mainly include the following steps S210 to S240.

[0108] S210 . Divide the depth map into one or more depth regions according to the magnitude of each depth value in the depth map, wherein each image position in each depth region has a similar depth value.

[0109] The depth map may include multiple pixels, and the grayscale value of each pixel in the depth map can be used to represent the distance between a certain point in the scene and the camera, that is, depth information.

[0110] Therefore, the depth map is divided into one or more depth areas according to the size of each depth value in the depth map. A larger depth map can be divided into smaller depth areas. At the same time, pixels with similar depth values ​​can be divided into the same depth area, which is convenient for subsequent noise identification according to the projection area corresponding to the depth area. It can prevent the depth distortion in the image imaging process from interfering with the noise identification and can improve the accuracy of noise area identification.

[0111] It can be understood that the depth value in the smaller depth area has a smaller variation. Therefore, for the smaller depth area, the first direction (i.e., the depth difference caused by turning during the container's movement) can be reduced. Figure 3 The influence of depth distortion (horizontal direction in the image) and image distortion can be reduced, thereby improving the subsequent recognition accuracy of the noise area.

[0112] In some embodiments, the depth map is a depth map obtained by performing depth imaging of the container using a line scan camera, and the first direction is a scanning direction of the line scan camera.

[0113] Line scan cameras are depth cameras that use linear array scanning to form images. Line scan imaging involves scanning the image plane to form a one-dimensional line image or a two-dimensional image. These images are then stitched together along the direction of the object's motion to create a complete image and a depth map.

[0114] In some embodiments, an area scan camera may be used to perform depth imaging on the container to obtain a depth map. Alternatively, other depth cameras may be used to perform depth imaging on various items to obtain a depth map.

[0115] In related technologies, depth imaging can be performed using a binocular camera. However, for containers, binocular cameras cannot cover the entire container, resulting in incomplete capture and the inability to obtain complete depth features of the container. Certain embodiments of the present application use a line scan camera to perform depth imaging of the container to obtain a depth map, which can fully identify the depth features of the depth map and project the depth area along a first direction on the depth area. The first direction is the scanning direction of the line scan camera, which is generally the transportation direction of the container. The projection image is subjected to noise identification, and the projection noise area in the projection image is obtained and then denoised. The imaging laws of the line scan camera and the shape and movement laws of the container can be used to improve the imaging integrity and denoising effect of the depth map, thereby improving the imaging effect of the depth map.

[0116] Figure 3 A schematic diagram of a scenario in which a depth map of a container is divided into multiple depth areas according to the sizes of the depth values ​​in the depth map is schematically shown in an embodiment of the present application. Figure 3 It can be regarded as a depth map on the XY plane composed of X coordinates and Y coordinates, wherein the depth value of the depth map is represented by the grayscale value.

[0117] As shown in the figure below, a line scan depth image of a container side panel is input, where the depth image is acquired by a line scan 3D camera. Our proposed method can accurately output and remove the noise area of ​​the container depth image.

[0118] Containers are used for transporting goods. In real-world applications, they can be subject to deformation, damage, and other surface defects due to impacts such as squeezing and impact. In these situations, collecting a depth map of the container surface can help locate defects based on changes in depth information. However, these depth maps often contain noise, whose depth values ​​closely match the deformation. Without noise removal, defect detection accuracy will be significantly reduced.

[0119] Specifically, the sources of noise in the collected depth map may be common noise in the container depth imaging process, such as camera imaging process noise, electromagnetic interference noise, and noise caused by box painting.

[0120] The inventors of this application have conducted relevant practices on depth map denoising of containers transported into a container yard. During the specific practice, they found that the application of traditional depth map denoising methods would bring two problems: First, when the containers are transported into the yard, the images formed by the line scan camera will be distorted due to uneven road conditions, turning, etc., and the depth values ​​at the beginning and end of the container's travel direction will vary greatly, which can easily interfere with the noise recognition process; Second, when noise, deformation, holes, etc. appear in the same area at the same time, it is difficult to distinguish them.

[0121] To solve the first problem above, in order to improve the accuracy and efficiency of container depth map denoising, the depth denoising method of this application is adopted, such as Figure 3 As shown, the depth map can be divided into multiple depth regions according to the direction of travel of the container. In combination with the unevenness change pattern of the container surface, the depth map can be divided into multiple long strip depth regions perpendicular to the direction of travel of the container. Each image position in each long strip depth region has a similar depth map. Therefore, dividing the pixels with similar depth values ​​in the depth map of the container into the same depth region can facilitate the subsequent identification of noise according to the projection area corresponding to the depth region, thereby preventing the depth distortion during the image formation process from interfering with the identification of noise, and improving the accuracy of the identification of the noise region. In addition, dividing the larger depth map into smaller long strip depth regions can avoid the impact of large changes in the depth values ​​at the beginning and end of the container's travel direction on the accuracy of noise identification, and can also prevent the imaging distortion of the line scan camera and the depth distortion from affecting the accuracy of noise identification.

[0122] In some embodiments, the scanning direction of the line scan camera may be the same as the direction of travel of the container.

[0123] From this, it can be understood that for containers in the container yard, the free area of ​​the yard is small and the business is busy. If the container is imaged and denoised after the truck loaded with the container stops, it will affect the normal operation of the yard. Therefore, the container is line-scanned and imaged at the same time as it is transported into the yard. The depth imaging and subsequent denoising of the container can be completed without the truck loaded with the container stopping, which can greatly reduce the time cost of detecting damage and deformation of the container.

[0124] The line scan camera's scanning direction aligns with the container's travel direction, thus avoiding image and depth distortion perpendicular to the line scan camera's scanning direction. Furthermore, dividing the depth map into multiple, elongated depth regions perpendicular to the container's travel direction avoids image and depth distortion within these depth regions, enabling this embodiment of the present application to achieve more accurate and efficient denoising.

[0125] It should be noted that depth imaging and denoising of containers in a container yard is only one application scenario of the present application. The depth image denoising method of the present application can also be applied to other items transported or stored in other ways.

[0126] Figure 4 The flowchart of the steps of dividing the depth map into one or more depth regions according to the size of each depth value in the depth map in an embodiment of the present application is schematically shown. Figure 4 As shown, based on the above embodiment, step S210 of dividing the depth map into one or more depth regions according to the magnitude of each depth value in the depth map may further include the following steps S410 to S440.

[0127] S410. Obtaining two columns of adjacent depth values ​​on the first plane coordinate axis in the depth map to obtain a first depth value column and a second depth value column;

[0128] S420. Determine the depth values ​​in the first depth value column and the second depth value column that have the same position on the second plane coordinate axis as a group of depth values;

[0129] S440. Calculate the difference between two depth values ​​in each group of depth values, and perform a sum operation on the obtained multiple differences to obtain a total difference;

[0130] S440. When the sum of the difference values ​​is greater than a first preset threshold, use the first depth value column and the second depth value column as a boundary between two depth regions, and divide the first depth value column into a first depth region and the second depth value column into a second depth region.

[0131] In some embodiments, a plane coordinate system can be established on the depth map, the plane coordinate system including a first plane coordinate axis and a second plane coordinate axis. In a specific embodiment, the first plane coordinate axis can be perpendicular to the second plane coordinate axis. In some embodiments, the direction of the first plane coordinate axis, i.e., the first direction, can be the scanning direction of a line scan camera used to obtain the depth map, and the direction of the second plane coordinate axis, i.e., the second direction, can be perpendicular to the scanning direction of the line scan camera.

[0132] Please continue to combine Figure 3 ,pass Figure 4 In one embodiment, the height of the first depth value column is m pixels and the width is 1 pixel, and the height of the second depth value column is n pixels and the width is 1 pixel. The sliding box composed of the first depth value column and the second depth value column can traverse the container depth map from left to right with a step size of 1 pixel. The first direction (i.e., the depth value of the container) is calculated by calculating the difference between the two depth values ​​in each group of depth values. Figure 3When the gradient of the horizontal direction (i.e., the X direction) shown in the figure is greater than the first preset threshold, that is, when the gradient of the horizontal direction of the current sliding frame changes greatly, the first depth value column and the second depth value column are used as the dividing line between the two depth areas to obtain the accurately divided depth areas.

[0133] For example, starting from the X coordinate a, slide from left to right on the depth map of the container with a step size of 1 pixel. If the gradient value difference between all depth values ​​at b-1 and all depth values ​​at b is large (for example, the average difference between each group of depth values ​​exceeds 20), then b-1 and b are used as the dividing line between the first depth area and the second depth area, that is, the depth values ​​in the interval [a, b-1] are divided into the first depth area, and continue to slide from left to right on the depth map of the container from b until the depth map of the container is traversed, so as to obtain all depth areas on the depth map.

[0134] In some embodiments, as Figure 3 As shown, the container body area can be first captured from the depth map of the container, and the outer frame can be removed. The depth map of the container body area after removing the outer frame is then used as the depth map to be denoised, and the depth region division is performed. It can be understood that most container problems that require repair, such as holes and deformations, occur in the main body of the container, excluding the outer frame. Therefore, only the denoised depth information of the main body of the container, excluding the outer frame, is needed to complete the detection of container holes, deformations, and other problems, thereby improving the denoising efficiency of the depth map.

[0135] In some embodiments, step S440 may be replaced by: calculating the difference between the depth values ​​from the first depth value column and the depth values ​​from the second depth value column in each group of depth values, and summing the multiple differences obtained between the two depth value columns to obtain a difference sum. Specifically, step S440 may be: calculating a first sum of the depth values ​​from the first depth value column in each group of depth values; calculating a second sum of the depth values ​​from the second depth value column in each group of depth values; calculating the difference between the first sum and the second sum in each group of depth values; and summing the multiple differences obtained between the two depth value columns to obtain a difference sum.

[0136] For example, the height of the first depth value column is m pixels and the width is 2 pixels, and the height of the second depth value column is n pixels and the width is 2 pixels. The sliding box composed of the first depth value column and the second depth value column can traverse the container depth map from left to right with a step size of 2 pixels. In some embodiments, the height of the first depth value column is m pixels and the width is 3 pixels, and the height of the second depth value column is n pixels and the width is 3 pixels. The sliding box composed of the first depth value column and the second depth value column can traverse the container depth map from left to right with a step size of 3 pixels. The pixel width of the first depth value column is equal to the pixel width of the second depth value column. This application does not impose any special restrictions on the pixel width of the first depth value column and the pixel width of the second depth value column.

[0137] In this way, the depth map can be traversed with a sliding frame consisting of a wider first depth value column and a wider second depth value column. As a result, for a depth map whose gradient change characteristics are not obvious, the corresponding gradient changes can also be captured. Then, when the horizontal gradient change of the current sliding frame is large, the first depth value column and the second depth value column are used as the dividing line between the two depth areas to obtain accurately divided depth areas.

[0138] As described above, the first depth value column and the second depth value column are used to detect features with obvious gradient changes in the first direction in the depth map, thereby enabling accurate division of different depth areas, that is, areas composed of pixels with similar depth values ​​can be accurately divided into the same depth area, and pixels with large depth value gradient changes can be divided into different depth areas, which is conducive to further detection of noise in the depth area according to the depth interval.

[0139] Figure 5 The flowchart schematically shows the steps before dividing the depth map into one or more depth regions according to the size of each depth value in the depth map in an embodiment of the present application. Figure 5 As shown, based on the above embodiment, before dividing the depth map into one or more depth regions according to the magnitude of each depth value in the depth map in step S210, the following steps S510 and S520 may be further included.

[0140] S510. Calculate the average value of each depth value in the depth map;

[0141] S520: Remove depth values ​​in the depth map whose difference from the average value is greater than a preset difference.

[0142] Thus, the depth values ​​in the depth map whose difference from the average value is greater than the preset difference can be removed, thereby removing obvious noise. Figure 3The average of the depth values ​​in the container depth map shown is removed from all depth values ​​in the depth map that differ from the average by more than 140, 150, 160, 170, or 180.

[0143] S220. Project the depth area along a first direction on the depth area to obtain a projection map corresponding to the depth area, where the projection map is used to represent the distribution of depth values ​​of the depth area corresponding to a second direction, where the second direction is a direction on the depth area different from the first direction.

[0144] Thus, the change of the depth value with respect to the second direction can be obtained, and the projection noise area can be efficiently identified through the distribution of the depth values ​​in the projection image, thereby locking the noise range in the depth image. It can be understood that, if Figure 7 As shown, in the distribution of the depth values ​​of the projection image with respect to the second direction, the noise has obvious outlier and jump characteristics, which can facilitate the subsequent efficient and accurate identification of the noise from the projection image through relevant algorithms.

[0145] Figure 6 The flowchart schematically shows the steps of performing projection processing on the depth region along the first direction on the depth region in an embodiment of the present application to obtain the projection image corresponding to the depth region. Figure 6 As shown, based on the above embodiment, step S220 of performing projection processing on the depth area along the first direction on the depth area to obtain a projection map corresponding to the depth area may further include the following steps S610 to S640.

[0146] S610. Establish a sliding window according to a first preset size;

[0147] S620. Slide the sliding window along the first direction in the depth region according to a preset step size, and intercept the depth region using the sliding window at each sliding position to obtain a sub-region corresponding to the sliding window at each sliding position;

[0148] S630. Projecting the sub-regions along the first direction to obtain sub-projection images corresponding to the sub-regions; wherein the sub-projection images are used to represent the distribution of depth values ​​corresponding to the sub-regions with respect to the second direction;

[0149] S640. Fuse the obtained multiple sub-projection images into a projection image along the sliding direction of the sliding window.

[0150] Therefore, by cutting out sub-regions through a sliding window on the depth area, the depth area can be divided into smaller sub-regions. For the smaller sub-regions, the impact of the image distortion of the container in the second direction on noise recognition can be reduced, thereby improving the subsequent recognition accuracy of the noise area.

[0151] Furthermore, the deformed areas in a container that require inspection are generally large. Therefore, smaller sub-areas will not show significant changes in the absence of noise. Therefore, dividing the area into smaller sub-areas can eliminate interference from the deformed areas, enabling noise identification and improving noise recognition accuracy.

[0152] Figure 7 The schematic diagram shows a scenario in which a sub-region is projected along a first direction to obtain a sub-projection image corresponding to each sub-region, and the obtained multiple sub-projection images are merged into a projection image along the sliding direction of the sliding window.

[0153] Specifically, Figure 7 The left side shown is the Figure 3 The enlarged area 305 in FIG. 3 is a schematic diagram of the sliding process of the sliding window. Figure 7 As shown in the left part of , a sliding window can be established with l as the width of the sliding window and h as the height of the sliding window.

[0154] In some embodiments, the width l of the sliding window can be the same as the width of the corresponding depth region in the first direction. Thus, the sliding window can be slid in one direction to traverse the depth region. In addition, the application of image depth distortion of the depth map in the first direction to noise recognition is avoided in the division of the depth region. Therefore, there is no need to divide the depth region by sliding the window in the first direction. This can improve the implementation efficiency of the method of the present application without affecting the accuracy of noise recognition.

[0155] In a specific embodiment, the height h of the sliding window may be 1 / 200 of the height of the depth region, and the step length d may be 1 / 2 of the height h of the sliding window. In some examples, the height h of the sliding window may be 1 / 100, 1 / 150, 1 / 250, 1 / 300, etc. of the height of the depth region, and the step length d may be 1 / 3, 1 / 4, 1 / 5, etc. of the height h of the sliding window. These steps may be adjusted accordingly based on the accuracy and efficiency of denoising during implementation of the present method.

[0156] Figure 7 The right part shown is a partial area of ​​the projection image obtained by merging multiple sub-projection images along the sliding direction of the sliding window. Figure 7 As shown, line 720 is the baseline line obtained by projecting the average depth value of all points in the original depth map using this value as the reference plane. It can be seen that noise region 730 in the projected image is an area where depth values ​​with sudden changes are concentrated. Noise 740 is the noise in the depth region corresponding to noise region 730.

[0157] Figure 8The flowchart schematically shows the steps of performing projection processing on the sub-regions along the first direction to obtain the sub-projection images corresponding to the sub-regions in an embodiment of the present application. Figure 8 As shown, based on the above embodiment, step S630 of performing projection processing on the sub-regions along the first direction to obtain sub-projection images corresponding to each sub-region may further include the following steps S810 to S830.

[0158] S810. Converting the depth map into a three-dimensional space point cloud using a three-dimensional coordinate system, where the three-dimensional coordinates include a first plane coordinate axis, a second plane coordinate axis, and a depth value coordinate axis;

[0159] S820. Obtaining a window point cloud corresponding to a sub-region in the three-dimensional point cloud;

[0160] S830. Taking the direction of the first plane coordinate axis as the first direction, projecting the window point cloud along the first direction to obtain a sub-projection image corresponding to the sub-region.

[0161] For specific embodiments, please refer to Figure 7 , the depth map can be converted into a three-dimensional space point cloud using the XYZ three-dimensional coordinate system. The three-dimensional coordinates include the first plane coordinate axis X, the second plane coordinate axis Y and the depth value coordinate axis Z. Among them, the value of the depth value coordinate axis Z is Figure 7 The depth map on the left is represented by grayscale values. Then, when obtaining the window point cloud corresponding to the sub-region in the 3D point cloud, the X direction can be used as the projection direction, and the window point cloud can be projected along the X axis to obtain the sub-projection image 7101 corresponding to sub-region 3051 and the sub-projection image 7102 corresponding to sub-region 3052.

[0162] S230. Perform noise identification on the projection image to obtain a projection noise area in the projection image.

[0163] Therefore, noise identification is performed on the projection image to obtain a projection noise area in the projection image. Noise identification in the projection image can be achieved through the value distribution of depth values ​​in the projection image.

[0164] Figure 9 The flowchart of the steps of performing noise recognition on the projection image and obtaining the projection noise area in the projection image in a certain embodiment of the present application is schematically shown. Figure 9 As shown, based on the above embodiment, step S230 of performing noise recognition on the projection image to obtain the projection noise area in the projection image may further include the following steps S910 and S920.

[0165] S910. Determine the sub-projection image corresponding to the area where the depth values ​​with value jumps in the projection image are concentrated as the target sub-projection image;

[0166] S920. Determine an area consisting of multiple target sub-projection images as a projection noise area.

[0167] Please continue reading Figure 7 The area 710 on the right is a projection noise area formed by the fusion of the target sub-projection image 7101, the sub-projection image 7102, and the sub-projection image 7103 along the sliding direction of the sliding window.

[0168] It can be understood that because pixels at various image locations in the depth region have similar depth values, when noise is present in the depth region, the depth values ​​will experience significant jumps, causing the depth values ​​in the depth region to no longer have a uniform distribution. Therefore, the sub-projection image corresponding to the area where the depth values ​​with jumps in the projection image are concentrated is determined as the target sub-projection image, and the area composed of multiple target sub-projection images is determined as the projection noise area. This can accurately and efficiently identify the noisy areas in the projection image and obtain the projection noise area.

[0169] Regarding the second problem of the traditional depth map denoising method mentioned above: when noise and deformation, holes, etc. appear in the same area at the same time, it is difficult to distinguish them. The embodiment of the present application identifies the area where the depth values ​​with value jumps in the projection map are concentrated, identifies the continuous up and down jump values ​​and outliers in the projection map, and obtains the projection noise area. It can be understood that the depth values ​​of normal deformation and hole areas generally change continuously and slowly. Therefore, determining the sub-projection map corresponding to the area where the depth values ​​with value jumps in the projection map are concentrated as the target sub-projection map can distinguish the noise from the deformation and hole areas in the depth map, thereby improving the accuracy of noise recognition in the depth map.

[0170] Figure 10 The following schematically shows a flowchart of the steps of determining the sub-projection image corresponding to the area where the depth values ​​with value jumps in the projection image are concentrated as the target sub-projection image in an embodiment of the present application. Figure 10 As shown, based on the above embodiment, step S910 of determining the sub-projection image corresponding to the area where the depth values ​​with value jumps in the projection image are concentrated as the target sub-projection image may further include the following steps S1010 to S1040.

[0171] S1010. Arrange the depth values ​​in the sub-projection image in descending order or in ascending order to obtain a depth value sequence;

[0172] S1020. Extracting multiple depth values ​​from the depth value sequence at intervals of a preset number;

[0173] S1030. Arrange the extracted depth values ​​in descending order or in ascending order, and calculate the difference between each two adjacent depth values;

[0174] S1040. When the difference is greater than a second preset threshold, the sub-projection image is determined to be a target sub-projection image.

[0175] Therefore, steps S1010 to S1040 are respectively performed on the sub-projection images corresponding to the sliding window at each sliding position, thereby realizing noise detection for the sub-region intercepted by the sliding window at each sliding position in the depth region.

[0176] In some embodiments, the mean of the depth values ​​in the depth map can be first obtained, and then the difference between the depth value of each pixel in the sub-projection map and the mean is calculated. Then, the depth values ​​in the sub-projection are converted and calculated: the minimum depth value in the sub-projection map is determined to be 0, and the remaining depth values ​​are all taken based on the minimum depth value of 0, with the maximum depth value being max. All depth values ​​after conversion calculation in the sub-projection are then arranged from large to small. Assuming that there are k points in the sub-projection map with converted and calculated depth values, they are arranged in descending order into a depth value sequence: {d1, d2…, 0}. Then, the depth values ​​at positions 0.1k, 0.2k, 0.3k, 0.4k, ..., 0.9k in the depth value sequence are extracted to obtain a set D = {dk1, dk2, ..., dk9}; and based on set D, a set F = {max-dk1, dk1-dk2, ..., dk8-dk9, dk9-0} is calculated. If any value in F is greater than the second preset threshold, it indicates that the depth region corresponding to the sub-projection image may be noisy, and the sub-projection image is determined to be the target sub-projection image. In a specific embodiment, the second preset threshold can be 25, 27, 30, 32, 35, etc. The second preset threshold can be set to an appropriate value according to the depth image denoising of different objects. For example, when denoising the depth image of a container, setting the second preset threshold to 30 can achieve good noise recognition accuracy.

[0177] S240 . Obtain a depth noise region corresponding to the projection noise region in the depth region, and perform denoising on each depth noise region of the depth map to obtain a denoised depth map.

[0178] The related art uses manual methods to remove noise from depth maps, which is inefficient and has high labor costs. The embodiment of the present application achieves automatic, efficient, and accurate denoising of the depth noise area through S210 to S240 as described above.

[0179] Figure 11 The corresponding schematic diagram of the process of denoising the deep noise area in a certain embodiment of the present application is schematically shown on the projection image. Figure 11As shown, each projection noise region can be denoised to obtain a denoised projection noise region. For example, noise region 730 in the projection noise region is denoised to obtain a denoised region 1110. The denoised projection noise region is then mapped to a denoised depth noise region using the mapping relationship between the depth region and the projection map. A denoised depth map is then obtained based on the denoised depth noise region.

[0180] Specifically, denoising each projected noise region may be performed by fitting a normal depth value based on the distribution of normal points surrounding the outlier in the projected noise region to replace the outlier, thereby removing noise. Alternatively, denoising each projected noise region may be performed by inputting the projected noise region into a pre-trained machine learning model to obtain a projected noise region after noise removal.

[0181] Figure 12 The flowchart of the steps of performing denoising on each depth noise area of ​​the depth map to obtain the denoised depth map in an embodiment of the present application is schematically shown. Figure 12 As shown, based on the above embodiment, step S240 of performing denoising on each depth noise area of ​​the depth map to obtain a denoised depth map may further include the following steps S1210 to S1250.

[0182] S1210. Establish a judgment box of a second preset size on the depth noise region, intercept the area to be judged in the depth noise region through the judgment box, and calculate the mean of the area to be judged to obtain a first mean; the area to be judged includes a center point, which is a pixel located at a preset position in the area to be judged;

[0183] S1220. Obtaining neighboring points adjacent to the center point, and calculating the mean of the to-be-determined area corresponding to each neighboring point of the center point to obtain a second mean;

[0184] S1230. Calculate the gradient values ​​of the first mean and each second mean, and determine the gradient value greater than the preset gradient value as an abnormal gradient value;

[0185] S1240. When the number of abnormal gradient values ​​is greater than a preset number, the area to be determined is determined as a noise area to be processed;

[0186] S1250. Re-determine the depth value of each noise area to be processed in the depth map to obtain a denoised depth map.

[0187] The area to be determined corresponding to the neighborhood point is the area to be determined corresponding to the neighborhood point as the center point.

[0188] In this way, the precise location of the noise can be further located in the deep noise area, which is conducive to accurate noise removal.

[0189] In a specific embodiment, the upper left corner of the deep noise area can be used as the starting point of the preset position, and the area to be determined can be traversed through the deep noise area, thereby achieving accurate positioning of the noise area to be processed in the deep noise area.

[0190] For example, the second preset size can be a rectangular frame of 3 pixels * 3 pixels, that is, the judgment frame is used to intercept a 3 pixel * 3 pixel area to be judged in the depth noise area. Among them, the 3 pixel * 3 pixel area to be judged includes 1 center point pixel and 8 neighboring pixels of the center point pixel. Then, the depth value average of each pixel in the 3 pixel * 3 pixel area to be judged is calculated, and then the area to be judged and the center point are traversed from left to right and from top to bottom in the depth noise area to obtain multiple 3 pixel * 3 pixel areas to be judged. The pixel average of all 3 pixel * 3 pixel areas to be judged is calculated, and then the gradient value of each area to be judged in the eight neighboring directions of the center point is calculated. If there are more than 4 gradient values ​​greater than 10, that is, the gradient value greater than 10 is determined to be an abnormal gradient value. When the number of abnormal gradient values ​​is greater than 4, the 3 pixel * 3 pixel area to be judged is considered to be a noise area to be processed.

[0191] Therefore, it is possible to achieve efficient and accurate positioning of the noise area to be processed with a relatively small amount of calculation.

[0192] Figure 13 The flowchart schematically shows the steps of re-determining the depth values ​​of each noise region to be processed in the depth map to obtain the denoised depth map in an embodiment of the present application. Figure 13 As shown, based on the above embodiment, step S1250 of re-determining the depth value of each noise area to be processed in the depth map to obtain a denoised depth map may further include the following steps S1310 and S1320.

[0193] S1310. Setting the depth value of each noise area to be processed in the depth map to zero;

[0194] S1320. Re-determine the depth value of each depth value in the noise area to be processed by an interpolation algorithm according to the depth values ​​around the noise area to be processed, and obtain a denoised depth map.

[0195] Specifically, the interpolation algorithm may include a bicubic interpolation algorithm, a bilinear interpolation algorithm, or a nearest neighbor interpolation algorithm, etc., and this application does not impose any special restrictions on this. In this way, each depth value in the noise area to be processed can be re-determined as a normal depth value, thereby achieving accurate denoising of the depth map.

[0196] Figure 14 The figure schematically shows a container depth map obtained according to another embodiment of the present application. Figure 14 The depth map shown has image distortion in the horizontal direction, which causes corresponding interference to traditional noise judgment and removal methods. Figure 14 The noise 1410 , the noise 1420 , and the noise 1430 are noises to be identified.

[0197] Figure 15 Schematically shows the Figure 14 The schematic diagram of the embodiment shown is a diagram of projecting a partial depth area of ​​the depth map and identifying and denoising the noise. Figure 15 As shown, the region 1510 where the depth values ​​with value jumps or outliers are concentrated in the projection image is identified as noise, and then the noise is removed. The region 1520 is the denoised region.

[0198] Figure 16 The following schematically shows a diagram of a container depth map obtained according to another embodiment of the present application. Figure 16 The depth map shown has image distortion in the horizontal direction, which causes corresponding interference to traditional noise judgment and removal methods. Figure 16 The noise 1610 and the noise 1620 in FIG. 1 are noises to be identified.

[0199] Figure 17 Schematically shows the Figure 16 Schematic diagram of denoising after projection of a partial depth area of ​​the depth map of the embodiment shown. Figure 17 As shown, the region 1710 where the depth values ​​with value jumps or outliers are concentrated in the projection image is identified as noise, and then the noise is removed. The region 1720 is the denoised region.

[0200] Figure 18 The flowchart schematically shows the steps after determining the area to be judged as a noise area to be processed when the number of abnormal gradient values ​​obtained in the area to be judged is greater than a preset number in an embodiment of the present application. Figure 18 As shown, based on the above embodiment, when the number of abnormal gradient values ​​obtained in the area to be determined is greater than a preset number, step S1240 determines that the area to be determined is a noise area to be processed, and the following steps S1810 and S1820 may be further included.

[0201] S1810. Calculate a first ratio of the number of abnormal gradient values ​​obtained in the noise region to be processed to the number of all gradient values ​​obtained in the noise region to be processed;

[0202] S1820. Determine a first confidence level based on the first ratio, where the first confidence level is used to indicate the reliability of the accuracy of the determination of the noise region to be processed.

[0203] The number of all gradient values ​​obtained in the noise region to be processed is the total number of gradient values ​​in each neighborhood direction of the center point of the noise region to be processed, and the number of abnormal gradient values ​​obtained in the noise region to be processed is the number of abnormal gradient values ​​in the total number.

[0204] For example, the first confidence level may be: C = (a / b) * 100%, where a is the number of abnormal gradient values ​​obtained in the noise region to be processed, and b is all gradient values ​​obtained in the noise region to be processed. When the center point of the noise region to be processed has 8 neighbors, the number b is correspondingly 8.

[0205] Thus, a quantitative calculation of the reliability of the accurate determination of the noise region to be processed is achieved, which can facilitate the monitoring of the accuracy of the depth map denoising process.

[0206] Figure 19 The flowchart of the steps after determining the first confidence level according to the first ratio in an embodiment of the present application is schematically shown. Figure 19 As shown, based on the above embodiment, after determining the first confidence level according to the first ratio in step S1820, the following steps S1910 to S1940 may be further included.

[0207] S1910. Determine the target number of noise regions to be processed in the depth region whose first confidence level is lower than a first preset confidence level;

[0208] S1920. Calculate a second ratio of the number of targets to the number of all noise regions to be processed in the depth region;

[0209] S1930. Determine a second confidence level based on the second ratio, where the second confidence level is used to indicate the reliability of the accuracy of the determination of the noise region to be processed in the depth region;

[0210] S1940. When the second confidence level is lower than a second preset confidence level, mark the depth region as a depth region requiring review of accuracy of determination of the noise region to be processed.

[0211] The first preset reliability can be 75%, 80%, 85%, 90%, etc., and this application does not impose any special limitation on this.

[0212] In some embodiments, the number of targets of the noise areas to be processed in the depth map whose first confidence level is lower than the first preset confidence level can also be determined; a second ratio of the number of targets to the number of all noise areas to be processed in the depth map is calculated; a second confidence level is determined based on the second ratio, and the second confidence level is used to indicate the degree of reliability of the accurate determination of the noise areas to be processed in the depth map; when the second confidence level is lower than the second preset confidence level, the depth map is marked as a depth map for which the accuracy of the determination of the noise areas to be processed needs to be reviewed.

[0213] Therefore, when the second confidence level is lower than the second preset confidence level, the depth area is marked as a depth area that needs to review the accuracy of the determination of the noise area to be processed, so as to facilitate manual review of the depth area or depth map with low confidence (for example, confidence level less than 80%) in actual production to ensure that the image denoising is accurate.

[0214] Figure 20 This schematic diagram illustrates a partial process of a depth map denoising method according to an embodiment of the present application. Thus, a depth map is acquired, the depth map is segmented into depth regions, noise is identified in the projections obtained by projecting the depth regions, and the identified noise is removed to obtain a denoised depth map, thereby achieving efficient and accurate denoising of the depth map.

[0215] It should be noted that although the steps of the method of the present application are described in a specific order in the drawings, this does not require or imply that the steps must be performed in this specific order, or that all steps must be performed to achieve the desired results. Additionally or alternatively, some steps may be omitted, multiple steps may be combined into one step, and / or one step may be decomposed into multiple steps.

[0216] The following describes an apparatus embodiment of the present application, which can be used to execute the depth map denoising method in the above-mentioned embodiment of the present application. Figure 21 The structure block diagram of the depth map denoising device provided in the embodiment of the present application is schematically shown. Figure 21 As shown, the depth map denoising apparatus 2100 includes:

[0217] The depth region division module 2110 is configured to divide the depth map into one or more depth regions according to the magnitude of each depth value in the depth map, wherein each image position in each depth region has a similar depth value;

[0218] a projection map acquisition module 2120 configured to project the depth region along a first direction in the depth region to obtain a projection map corresponding to the depth region, where the projection map is used to represent a distribution of depth values ​​in the depth region corresponding to a second direction, where the second direction is a direction in the depth region different from the first direction;

[0219] The projection noise region determining module 2130 is configured to perform noise recognition on the projection image to obtain the projection noise region in the projection image;

[0220] The denoising module 2140 is configured to obtain a depth noise region corresponding to the projection noise region in the depth region, and perform denoising on each depth noise region of the depth map to obtain a denoised depth map.

[0221] In some embodiments of the present application, based on the above embodiments, the projection image acquisition module includes:

[0222] a sliding window establishing unit, configured to establish a sliding window according to a first preset size;

[0223] a depth region intercepting unit configured to slide the sliding window along a first direction in the depth region according to a preset step size, and intercept the depth region using the sliding window at each sliding position to obtain a sub-region corresponding to the sliding window at each sliding position;

[0224] a projection processing unit configured to perform projection processing on the sub-regions along the first direction to obtain sub-projection images corresponding to the sub-regions; wherein the sub-projection images are used to represent the distribution of depth values ​​of the sub-regions corresponding to the second direction;

[0225] The projection image acquisition unit is configured to merge the obtained multiple sub-projection images into a projection image along the sliding direction of the sliding window.

[0226] In some embodiments of the present application, based on the above embodiments, the projection processing unit includes:

[0227] a three-dimensional conversion subunit configured to convert the depth map into a three-dimensional space point cloud using a three-dimensional coordinate system, wherein the three-dimensional coordinate system includes a first plane coordinate axis, a second plane coordinate axis, and a depth value coordinate axis;

[0228] a window point cloud acquisition subunit, configured to acquire a window point cloud corresponding to a subregion in the three-dimensional space point cloud;

[0229] The projection processing subunit is configured to use the direction of the first plane coordinate axis as the first direction, project the window point cloud along the first direction, and obtain a sub-projection image corresponding to the sub-region.

[0230] In some embodiments of the present application, based on the above embodiments, the depth region division module includes:

[0231] A depth value column acquisition unit is configured to acquire two columns of depth values ​​in the depth map that are adjacent to each other on the first plane coordinate axis to obtain a first depth value column and a second depth value column;

[0232] a depth value group acquiring unit configured to determine depth values ​​in the first depth value column and the second depth value column that have the same position on the second plane coordinate axis as a group of depth values;

[0233] a difference sum calculation unit configured to calculate a difference between two depth values ​​in each group of depth values, and perform a sum operation on the obtained multiple differences to obtain a difference sum;

[0234] The depth region division unit is configured to, when the sum of the difference values ​​is greater than a first preset threshold, use the first depth value column and the second depth value column as a dividing line between two depth regions, and divide the first depth value column into the first depth region and the second depth value column into the second depth region.

[0235] In some embodiments of the present application, based on the above embodiments, the projection noise area determination module includes:

[0236] a target sub-projection image determining unit configured to determine, as a target sub-projection image, a sub-projection image corresponding to an area where depth values ​​with value jumps are concentrated in the projection image;

[0237] The projection noise region determining unit is configured to determine a region formed by a plurality of target sub-projection images as a projection noise region.

[0238] In some embodiments of the present application, based on the above embodiments, the target sub-projection graph determining unit includes:

[0239] The depth value sequence acquisition subunit is configured to arrange the depth values ​​in the sub-projection image in descending or ascending order to obtain a depth value sequence;

[0240] The depth value extraction subunit is configured to extract a plurality of depth values ​​from the depth value sequence at intervals of a preset number;

[0241] The difference calculation subunit is configured to arrange the extracted multiple depth values ​​in order from large to small or from small to large, and calculate the difference between every two adjacent depth values;

[0242] The target sub-projection image determining sub-unit is configured to determine the sub-projection image as the target sub-projection image when the difference value is greater than a second preset threshold.

[0243] In some embodiments of the present application, based on the above embodiments, the denoising module includes:

[0244] a first mean value acquisition unit configured to establish a judgment box of a second preset size on the depth noise region, intercept a region to be judged in the depth noise region through the judgment box, and calculate a mean of the region to be judged to obtain a first mean value; the region to be judged includes a center point, which is a pixel point located at a preset position in the region to be judged;

[0245] A second mean value obtaining unit is configured to obtain neighboring points adjacent to the center point, and calculate the mean of the to-be-determined area corresponding to each neighboring point of the center point to obtain a second mean value;

[0246] an abnormal gradient value determination unit, configured to calculate the gradient values ​​of the first mean value and each second mean value, and determine a gradient value greater than a preset gradient value as an abnormal gradient value;

[0247] a noise region to be processed determination unit, configured to determine the region to be determined as a noise region to be processed when the number of abnormal gradient values ​​is greater than a preset number;

[0248] The noise removal unit is configured to re-determine the depth value of each noise area to be processed in the depth map to obtain a denoised depth map.

[0249] In some embodiments of the present application, based on the above embodiments, the noise removal unit includes:

[0250] a depth value zeroing subunit, configured to set the depth value of each noise area to be processed in the depth map to zero;

[0251] The interpolation processing subunit is configured to perform interpolation processing on the noise area to be processed according to the depth values ​​around the noise area to be processed, so as to obtain a denoised depth map.

[0252] In some embodiments of the present application, based on the above embodiments, the depth map denoising apparatus further includes:

[0253] a first ratio calculation unit configured to calculate a first ratio of the number of abnormal gradient values ​​obtained in the noise region to be processed to the number of all gradient values ​​obtained in the noise region to be processed;

[0254] The first confidence determination unit is configured to determine a first confidence according to the first ratio, where the first confidence is used to indicate a reliability level of accuracy in determining the noise region to be processed.

[0255] In some embodiments of the present application, based on the above embodiments, the depth map denoising apparatus further includes:

[0256] a target number determining unit configured to determine the number of targets in the noise region to be processed in the depth region whose first confidence level is lower than a first preset confidence level;

[0257] a second ratio calculation unit configured to calculate a second ratio of the number of targets to the number of all noise regions to be processed in the depth region;

[0258] a second confidence determination unit configured to determine a second confidence based on the second ratio, the second confidence being used to indicate a degree of reliability in determining whether the noise region to be processed in the depth region is accurately determined;

[0259] The depth region determination unit requiring review is configured to mark the depth region as a depth region requiring review of the accuracy of determination of the noise region to be processed when the second confidence level is lower than a second preset confidence level.

[0260] In some embodiments of the present application, based on the above embodiments, the depth map denoising apparatus further includes:

[0261] an average value calculation unit, configured to calculate an average value of each depth value in the depth map;

[0262] The depth value removal unit is configured to remove depth values ​​in the depth map whose difference from an average value is greater than a preset difference.

[0263] In some embodiments of the present application, based on the above embodiments, the depth map is a depth map obtained by performing depth imaging of the container using a line scan camera, and the first direction is the scanning direction of the line scan camera.

[0264] The specific details of the depth map denoising device provided in each embodiment of the present application have been described in detail in the corresponding related method embodiments and will not be repeated here.

[0265] Figure 22 The block diagram schematically shows a computer system structure of an electronic device used to implement an embodiment of the present application.

[0266] It should be noted that Figure 22 The computer system 2200 of the electronic device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0267] like Figure 22 As shown, the computer system 2200 includes a central processing unit 2201 (CPU), which can perform various appropriate actions and processes according to the program stored in the read-only memory 2202 (ROM) or the program loaded from the storage part 2208 to the random access memory 2203 (RAM). Various programs and data required for system operation are also stored in the random access memory 2203. The central processing unit 2201, the read-only memory 2202 and the random access memory 2203 are connected to each other via a bus 2204. An input / output interface 2205 (i.e., an I / O interface) is also connected to the bus 2204.

[0268] The following components are connected to the input / output interface 2205: an input section 2206 including a keyboard, a mouse, and the like; an output section 2207 including devices such as a cathode ray tube (CRT), a liquid crystal display (LCD), and a speaker; a storage section 2208 including a hard disk; and a communication section 2209 including a network interface card such as a local area network card or a modem. The communication section 2209 performs communication processing via a network such as the Internet. A drive 2210 is also connected to the input / output interface 2205 as needed. Removable media 2211, such as a magnetic disk, an optical disk, a magneto-optical disk, or a semiconductor memory, is installed in the drive 2210 as needed, so that computer programs read therefrom can be installed into the storage section 2208 as needed.

[0269] In particular, according to an embodiment of the present application, the processes described in the various method flow charts can be implemented as computer software programs. For example, an embodiment of the present application includes a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for executing the methods shown in the flow charts. In such an embodiment, the computer program can be downloaded and installed from a network via the communication section 2209 and / or installed from a removable medium 2211. When the computer program is executed by the central processing unit 2201, the various functions defined in the system of the present application are performed.

[0270] It should be noted that the computer-readable medium shown in the embodiments of the present application may be a computer-readable signal medium or a computer-readable storage medium or any combination of the two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or component, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM), a flash memory, an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present application, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, device or device. In the present application, a computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, which carries a computer-readable program code. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium that can transmit, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device. Program code embodied on a computer-readable medium may be transmitted using any suitable medium, including but not limited to wireless, wired, or any suitable combination thereof.

[0271] The flowcharts and block diagrams in the accompanying drawings illustrate the possible implementation architecture, functions and operations of the systems, methods and computer program products according to various embodiments of the present application. In this regard, each box in the flowchart or block diagram can represent a module, program segment, or a part of code, and the above-mentioned module, program segment, or a part of code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in an order different from that marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flowchart, and the combination of the boxes in the block diagram or flowchart, can be implemented with a dedicated hardware-based system that performs the specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0272] It should be noted that, although several modules or units of the device for action execution are mentioned in the above detailed description, this division is not mandatory. In fact, according to the embodiment of the application, the features and functions of two or more modules or units described above can be concretized in one module or unit. On the contrary, the features and functions of one module or unit described above can be further divided into multiple modules or units to be concretized.

[0273] Through the description of the above embodiments, it is easy for those skilled in the art to understand that the example embodiments described herein can be implemented by software or by combining software with necessary hardware. Therefore, the technical solution according to the embodiments of the present application can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash drive, a mobile hard disk, etc.) or on a network, and includes several instructions to enable a computing device (which can be a personal computer, a server, a touch terminal, or a network device, etc.) to execute the method according to the embodiments of the present application.

[0274] Those skilled in the art will readily appreciate other embodiments of the present invention after considering the specification and practicing the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of the present invention that follow the general principles of this application and include common knowledge or customary techniques in the art that are not disclosed herein.

[0275] It should be understood that the present application is not limited to the exact structures described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present application is limited only by the appended claims.

Claims

1. A depth map denoising method, characterized in that: The method comprises: Dividing the depth map into one or more depth regions according to the magnitude of each depth value in the depth map, wherein each image position in each depth region has a similar depth value; Projecting the depth region along a first direction on the depth region to obtain a projection map corresponding to the depth region, the projection map being used to represent a distribution of depth values ​​of the depth region corresponding to a second direction, where the second direction is a direction on the depth region that is different from the first direction; Performing noise identification on the projection image to obtain a projection noise region in the projection image; the projection noise region includes a region corresponding to at least one of a jump depth value and an outlier depth value in the projection image; Acquire a depth noise region corresponding to the projection noise region in the depth region, and perform denoising on each depth noise region of the depth map to obtain a denoised depth map; Among them, the depth area is projected along a first direction on the depth area to obtain a projection map corresponding to the depth area, including: establishing a sliding window according to a first preset size; sliding the sliding window along the first direction in the depth area according to a preset step size, and intercepting the depth area through the sliding window at each sliding position to obtain a sub-area corresponding to the sliding window at each sliding position; converting the depth map into a three-dimensional space point cloud using a three-dimensional coordinate system, the three-dimensional coordinates including a first plane coordinate axis, a second plane coordinate axis and a depth value coordinate axis; obtaining a window point cloud corresponding to the sub-area in the three-dimensional space point cloud; taking the direction of the first plane coordinate axis as the first direction, and projecting the window point cloud along the first direction to obtain a sub-projection map corresponding to the sub-area, wherein the sub-projection map is used to represent the value distribution of the depth value of the sub-area corresponding to the second direction; and merging the obtained multiple sub-projection maps into the projection map along the sliding direction of the sliding window.

2. The method according to claim 1, characterized in that Dividing the depth map into one or more depth regions according to the magnitude of each depth value in the depth map includes: Obtaining two columns of adjacent depth values ​​on a first plane coordinate axis in the depth map to obtain a first depth value column and a second depth value column; Determine depth values ​​in the first depth value column and the second depth value column that have the same position on the second plane coordinate axis as a group of depth values; Calculating the difference between two depth values ​​in each group of depth values, and summing the obtained multiple differences to obtain a total difference value; When the sum of the difference values ​​is greater than a first preset threshold, the first depth value column and the second depth value column are used as a boundary between two depth regions, and the first depth value column is divided into a first depth region, and the second depth value column is divided into a second depth region.

3. The method according to claim 1, characterized in that Performing noise identification on the projection image to obtain a projection noise area in the projection image includes: Determine the sub-projection image corresponding to the area where the depth values ​​with value jumps in the projection image are concentrated as the target sub-projection image; An area formed by a plurality of the target sub-projection images is determined as the projection noise area.

4. The method according to claim 3, characterized in that Determining a sub-projection image corresponding to an area where depth values ​​with value jumps are concentrated in the projection image as a target sub-projection image includes: Arrange the depth values ​​in the sub-projection image in descending order or in ascending order to obtain a depth value sequence; Extracting a plurality of depth values ​​from the depth value sequence at predetermined intervals; Arrange the extracted depth values ​​in descending order or in ascending order, and calculate the difference between each two adjacent depth values; When the difference is greater than a second preset threshold, the sub-projection image is determined to be a target sub-projection image.

5. The method according to claim 4, characterized in that Denoising each depth noise region of the depth map to obtain a denoised depth map, including: Establishing a judgment box of a second preset size on the depth noise area, intercepting an area to be judged in the depth noise area through the judgment box, and calculating a mean of the area to be judged to obtain a first mean; the area to be judged includes a center point, and the center point is a pixel point located at a preset position in the area to be judged; Obtaining neighboring points adjacent to the center point, and calculating the mean of the to-be-determined area corresponding to each neighboring point of the center point to obtain a second mean; Calculating gradient values ​​of the first mean value and each of the second mean values, and determining a gradient value greater than a preset gradient value as an abnormal gradient value; When the number of abnormal gradient values ​​is greater than a preset number, the area to be determined is determined as a noise area to be processed; The depth value of each noise area to be processed in the depth map is re-determined to obtain a denoised depth map.

6. The method according to claim 5, characterized in that Re-determining the depth value of each noise area to be processed in the depth map to obtain a denoised depth map, including: Setting the depth value of each noise area to be processed in the depth map to zero; According to the depth values ​​around the noise area to be processed, the depth values ​​of each depth value in the noise area to be processed are re-determined by an interpolation algorithm to obtain a denoised depth map.

7. The method according to claim 5, characterized in that When the number of abnormal gradient values ​​obtained in the to-be-determined area is greater than a preset number, after determining the to-be-determined area as a noise area to be processed, the method further includes: Calculating a first ratio of the number of abnormal gradient values ​​obtained in the noise region to be processed to the number of all gradient values ​​obtained in the noise region to be processed; A first confidence level is determined according to the first ratio, where the first confidence level is used to indicate the reliability of the accuracy of the determination of the noise region to be processed.

8. The method according to claim 7, characterized in that After determining a first confidence level according to the first ratio, the method further includes: determining a target number of noise regions to be processed in the depth region whose first confidence level is lower than a first preset confidence level; calculating a second ratio of the number of targets to the number of all noise regions to be processed in the depth region; determining a second confidence level according to the second ratio, wherein the second confidence level is used to indicate a degree of reliability in determining whether the noise region to be processed in the depth region is accurately determined; When the second confidence level is lower than a second preset confidence level, the depth region is marked as a depth region for which the accuracy of determination of the noise region to be processed needs to be reviewed.

9. The method according to any one of claims 1 to 8, characterized in that Before dividing the depth map into one or more depth regions according to the magnitude of each depth value in the depth map, the method further includes: Calculating an average of depth values ​​in the depth map; Depth values ​​in the depth map whose difference from the average value is greater than a preset difference are removed.

10. The method according to any one of claims 1 to 8, characterized in that: The depth map is a depth map obtained by performing depth imaging of the container using a line scan camera, and the first direction is a scanning direction of the line scan camera.

11. A depth map denoising device, characterized in that: The device comprises: a depth region division module configured to divide the depth map into one or more depth regions according to the magnitude of each depth value in the depth map, wherein each image position in each depth region has a similar depth value; a projection map acquisition module configured to project the depth region along a first direction on the depth region to obtain a projection map corresponding to the depth region, the projection map being used to represent a distribution of depth values ​​of the depth region corresponding to a second direction, where the second direction is a direction on the depth region that is different from the first direction; a projection noise region determination module configured to perform noise identification on the projection image to obtain a projection noise region in the projection image; the projection noise region includes a region corresponding to at least one of a jump depth value and an outlier depth value in the projection image; a denoising module configured to obtain a depth noise region in the depth region corresponding to the projection noise region, and perform denoising on each depth noise region in the depth map to obtain a denoised depth map; Among them, the depth area is projected along a first direction on the depth area to obtain a projection map corresponding to the depth area, including: establishing a sliding window according to a first preset size; sliding the sliding window along the first direction in the depth area according to a preset step size, and intercepting the depth area through the sliding window at each sliding position to obtain a sub-area corresponding to the sliding window at each sliding position; converting the depth map into a three-dimensional space point cloud using a three-dimensional coordinate system, the three-dimensional coordinates including a first plane coordinate axis, a second plane coordinate axis and a depth value coordinate axis; obtaining a window point cloud corresponding to the sub-area in the three-dimensional space point cloud; taking the direction of the first plane coordinate axis as the first direction, and projecting the window point cloud along the first direction to obtain a sub-projection map corresponding to the sub-area, wherein the sub-projection map is used to represent the value distribution of the depth value of the sub-area corresponding to the second direction; and merging the obtained multiple sub-projection maps into the projection map along the sliding direction of the sliding window.

12. A computer-readable medium, characterized in that A computer program is stored thereon, and when the computer program is executed by a processor, the method according to any one of claims 1 to 10 is implemented.

13. An electronic device, characterized in that: include: processor; as well as a memory for storing executable instructions of the processor; The processor is configured to perform the method according to any one of claims 1 to 10 by executing the executable instructions.

14. A computer program product, characterized in that The invention comprises a computer program carried on a computer-readable storage medium, wherein when the computer program is executed by a processor, the method according to any one of claims 1 to 10 is implemented.

Images