Image processing methods, apparatus, devices and storage media

Abstract

This application discloses an image processing method, apparatus, device, and storage medium, applicable to fields such as cloud computing, computer vision, and blockchain. The method includes: determining a first feature image of a first initial image and a second feature image of a second initial image; determining at least one sub-image from the first and second feature images based on a preset pixel window; determining at least one image pair based on the region of each sub-image in the corresponding feature image; fusing the first and second sub-images in each image pair to obtain a fused image; determining the image registration state of the corresponding image pair based on each fused image; and determining the image registration state of the first and second initial images based on the image registration state of each image pair, wherein the image registration state is either registered or unregistered. Using this application embodiment, the image registration state can be determined efficiently and accurately, exhibiting high applicability.

Description

Technical Field

[0001] This application relates to the field of computer vision, and more particularly to an image processing method, apparatus, and storage medium. Background Technology

[0002] With the development of artificial intelligence technology, image registration technology has emerged. Image registration is the process of registering and overlaying two or more images acquired at different times, by different sensors (imaging devices), or under different conditions (weather, illumination, camera position and angle, etc.). It has been widely used in remote sensing data analysis, computer vision, image processing and other fields.

[0003] Existing methods for determining whether images are registered often rely on visual inspection, which has low accuracy and significantly reduces efficiency when dealing with a large number of image pairs, resulting in poor applicability. Summary of the Invention

[0004] This application provides an image processing method, apparatus, device, and storage medium that can efficiently and accurately determine the image registration state and has high applicability.

[0005] On one hand, embodiments of this application provide an image processing method, the method comprising:

[0006] A first feature image of a first initial image and a second feature image of a second initial image are determined, wherein the first initial image and the second initial image correspond to the same scene, and the first feature image and the second feature image are any one of a depth image, a grayscale image, or a color image;

[0007] At least one sub-image is determined from the first feature image and the second feature image based on a preset pixel window. At least one image pair is determined based on the region of each sub-image in the corresponding feature image. The first sub-image and the second sub-image in each image pair belong to different feature images and have the same region in the corresponding feature image.

[0008] The first and second sub-images in each of the above image pairs are fused to obtain a fused image, and the image registration state of the first and second sub-images in the corresponding image pair is determined based on each of the above fused images;

[0009] Based on the image registration status of each of the above image pairs, the image registration status of the first initial image and the second initial image is determined, and the image registration status is either a registered state or an unregistered state.

[0010] On the other hand, embodiments of this application provide an image processing apparatus, which includes:

[0011] The image determination module is used to determine a first feature image of a first initial image and a second feature image of a second initial image, wherein the first initial image and the second initial image correspond to the same scene, and the first feature image and the second feature image are any one of a depth image, a grayscale image, or a color image;

[0012] The image segmentation module is used to determine at least one sub-image from the first feature image and the second feature image respectively based on a preset pixel window, and to determine at least one image pair based on the region of each sub-image in the corresponding feature image, wherein the first sub-image and the second sub-image in each image pair belong to different feature images and have the same region in the corresponding feature image.

[0013] The image fusion module is used to fuse the first sub-image and the second sub-image in each of the above image pairs to obtain a fused image, and to determine the image registration state of the first sub-image and the second sub-image in the corresponding image pair based on each of the above fused images;

[0014] The state determination module is used to determine the image registration state of the first initial image and the second initial image based on the image registration state of each of the above image pairs, wherein the image registration state is either a registered state or an unregistered state.

[0015] On the other hand, embodiments of this application provide an electronic device, including a processor and a memory, which are interconnected;

[0016] The memory is used to store computer programs;

[0017] The processor is configured to execute the image processing method provided in the embodiments of this application when the computer program is invoked.

[0018] On the other hand, embodiments of this application provide a computer-readable storage medium storing a computer program that is executed by a processor to implement the image processing method provided in embodiments of this application.

[0019] On the other hand, embodiments of this application provide a computer program product, which includes a computer program that, when executed by a processor, implements the image processing method provided in embodiments of this application.

[0020] In this embodiment, by defining the first feature image of the first initial image and the second feature image of the second initial image as multiple image pairs, the information complexity of the sub-images in each image pair can be reduced. This allows for accurate and rapid determination of the image registration state of each image pair, further improving the accuracy and efficiency of determining the image registration state of the first and second initial images. Furthermore, the first and second feature images can be at least one of depth images, grayscale images, or color images, allowing for flexible determination of the image registration state of the first and second initial images based on either the depth or grayscale image, thus offering greater flexibility. Attached Figure Description

[0021] To more clearly illustrate the technical solutions in the embodiments of this application, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0022] Figure 1 This is a schematic flowchart of the image processing method provided in the embodiments of this application;

[0023] Figure 2 This is a scene diagram illustrating the determination of feature images provided in an embodiment of this application;

[0024] Figure 3 This is a schematic diagram of a scenario for determining key areas provided in an embodiment of this application;

[0025] Figure 4 This is a schematic diagram of a scene for determining image pairs provided in an embodiment of this application;

[0026] Figure 5 This is a schematic diagram of a scenario for determining the image registration state provided in an embodiment of this application;

[0027] Figure 6 This is a schematic diagram of a neural network structure provided in an embodiment of this application;

[0028] Figure 7 This is another schematic diagram of the neural network structure provided in the embodiments of this application;

[0029] Figure 8 This is another scenario diagram illustrating the determination of image registration status provided in the embodiments of this application;

[0030] Figure 9 This is a schematic diagram of the structure of the image processing apparatus provided in the embodiments of this application;

[0031] Figure 10This is a schematic diagram of the structure of the electronic device provided in the embodiments of this application. Detailed Implementation

[0032] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0033] The image processing method in this application embodiment can be used to determine the image registration state. For example, in the field of industrial quality inspection, it can determine the image registration state of two initial images of the same industrial component (such as machine parts) acquired at different times, by different sensors (imaging devices) or under different conditions (weather, illumination, camera position and angle, etc.), so as to determine whether the industrial component has defects based on the determined image registration state.

[0034] For example, in the fields of intelligent transportation or mapping, the image registration status of two initial road images of the same traffic area at different times can be determined based on the Intelligent Traffic System (ITS) or Intelligent Vehicle Infrastructure Cooperative Systems (IVICS). Based on the determined image registration status, it can be determined whether the road scene has changed, and then the map information or navigation information corresponding to the road scene can be updated.

[0035] Intelligent Transportation Systems (ITS), also known as Intelligent Transportation Systems, effectively integrate advanced science and technology (information technology, computer technology, data communication technology, sensor technology, electronic control technology, automatic control theory, operations research, artificial intelligence, etc.) into transportation, service control, and vehicle manufacturing. This strengthens the connection between vehicles, roads, and users, thereby forming a comprehensive transportation system that ensures safety, improves efficiency, enhances the environment, and saves energy.

[0036] Intelligent vehicle-road cooperative systems, or vehicle-road cooperative systems for short, represent a development direction for Intelligent Transportation Systems (ITS). These systems utilize advanced wireless communication and next-generation internet technologies to implement comprehensive, real-time dynamic information exchange between vehicles and infrastructure. Based on the collection and fusion of dynamic traffic information across all times and spaces, they enable proactive vehicle safety control and collaborative road management, fully realizing effective coordination between people, vehicles, and roads. This ensures traffic safety, improves traffic efficiency, and ultimately forms a safe, efficient, and environmentally friendly road traffic system.

[0037] The image processing method provided in this application embodiment can be executed by any terminal device or server. The server can be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server or server cluster providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms. The terminal device can be a smartphone, tablet, laptop, desktop computer, smart speaker, smartwatch, or in-vehicle terminal, but is not limited to these.

[0038] For example, when the image processing method provided in this application embodiment is executed by an in-vehicle terminal, the in-vehicle terminal can determine the image registration status of two initial road images of the same traffic area at different times, and determine whether the road scene has changed based on the determined image registration status, and re-plan the navigation if the road scene has changed.

[0039] See Figure 1 , Figure 1 This is a schematic flowchart of the image processing method provided in an embodiment of this application. Figure 1 As shown, the image processing method provided in this application embodiment may specifically include the following steps:

[0040] Step S11: Determine the first feature image of the first initial image and the second feature image of the second initial image.

[0041] In some feasible implementations, the first initial image and the second initial image are two images corresponding to the same scene, such as two images corresponding to the same work area or traffic area. Alternatively, the first initial image and the second initial image can also be determined to correspond to the same scene when they include the same target object. For example, the first initial image and the second initial image can be two images of the target object acquired at different times, through different sensors (imaging devices), and under different conditions (weather, illumination, camera position and angle, etc.).

[0042] The first initial image and the second initial image can also be two images that have already undergone image registration based on existing technology and need to be further judged in terms of image registration status. The specific choice can be determined based on the actual application scenario requirements and is not limited here.

[0043] Furthermore, a first feature image of the first initial image and a second feature image of the second initial image can be determined.

[0044] In this embodiment, the first initial image and the second initial image can be single-channel images, such as infrared images, depth images, or grayscale images, or images including at least two channels, such as color images including R, G, and B color channels or color images including G and B color channels, etc. The specific image can be determined based on the actual application scenario requirements, and no restrictions are imposed here.

[0045] The first feature image can be any one of a depth image, a grayscale image, or a color image. That is, if the first initial image is a depth image, a grayscale image, or a color image, the first initial image itself can be used as the first feature image; or if the first initial image is not a depth image, the first initial image can be converted into a depth image to obtain the first feature image; or if the first initial image is not a grayscale image, the first initial image can be converted into a grayscale image to obtain the first feature image; or if the first initial image is a single-channel image, the first initial image can be converted into a multi-channel color image to obtain the first feature image.

[0046] Similarly, the second feature image can be any one of a depth image, a grayscale image, or a color image. That is to say, if the second initial image is a depth image, a grayscale image, or a color image, the second initial image itself can be used as the second feature image; or if the second initial image is not a depth image, it can be converted into a depth image to obtain the second feature image; or if the second initial image is not a grayscale image, it can be converted into a grayscale image to obtain the second feature image; or if the second initial image is a single-channel image, it can be converted into a multi-channel color image to obtain the second feature image.

[0047] For example, see Figure 2 , Figure 2 This is a scene diagram illustrating the determination of feature images provided in an embodiment of this application. For example... Figure 2 As shown, the first initial image and the second initial image include the same object, that is, they correspond to the same scene. At this point, the grayscale image of the first initial image can be determined, and this grayscale image is defined as the first feature image of the first initial image. Simultaneously, the depth image of the second initial image can be determined, and this depth image is defined as the second feature image of the second initial image.

[0048] The specific implementation of determining the first feature image of the first initial image and the second feature image of the second initial image can be based on image processing technology, which will not be described here.

[0049] Step S12: Determine at least one sub-image from the first feature image and the second feature image based on a preset pixel window, and determine at least one image pair based on the region of each sub-image in the corresponding feature image.

[0050] In some feasible implementations, if the image sizes of the first initial image and the second initial image are small, and if the pixel sizes (pixel height and pixel width) of the first feature image and the second feature image are the same, then at least one sub-image can be determined from the first feature image and the second feature image respectively based on a preset pixel window and a preset sliding rule.

[0051] If the pixel dimensions (pixel height and pixel width) of the first feature image and the second feature image are inconsistent, the first feature image and the second feature image can be scaled to the same pixel height and pixel width first, and then at least one sub-image can be determined from the first feature image and the second feature image respectively based on the preset pixel window and the preset sliding rule.

[0052] The pixel height and pixel width of the preset pixel window can be determined based on the actual application scenario requirements, such as being able to... The sliding window is not restricted here.

[0053] The preset sliding rules are used to represent the movement direction and step size of the sliding window. The specific rules can be determined based on the actual application scenario requirements and are not restricted here.

[0054] The preset pixel windows corresponding to the first feature image and the second feature image are the same, as are their starting positions, and the preset sliding rules corresponding to the first feature image and the second feature image are the same.

[0055] Based on this, a sub-feature image can be determined from each identical region in the first feature image and the second feature image. That is, after determining at least one sub-image from the first feature image and a sub-image from the second feature image, for any sub-image in the first feature image, there exists a target sub-image in the sub-image of the second feature image, and the region of the target sub-image in the second feature image is the same as the region of the sub-image in the first feature image.

[0056] Based on this, at least one image pair can be determined based on the regions of each sub-image corresponding to the first feature image and the second feature image in the corresponding feature image, wherein the first sub-image and the second sub-image in each image pair belong to different feature images and have the same region in the corresponding feature image.

[0057] In each image pair, the first sub-image belongs to the first feature image, and the second sub-image belongs to the second feature image.

[0058] Optionally, at least one first key region in the first feature image may also be determined, each first key region being used to characterize local features of the first feature image. For example, a region containing an object or a region including an object in the first feature image may be determined as a first key region, or a region containing local features of an object may be determined as a first key region. The specific determination can be based on the actual application scenario requirements and is not limited here.

[0059] For example, if the first initial image is an original image of a container, then a region in the first feature image that starts from the left edge of the container area, has a width of 1 / 20 of the container area, and a height equal to the height of the container area can be defined as a first key region, and a region in the first feature image that starts from the right edge of the container area, has a width of 1 / 20 of the container area, and a height equal to the height of the container area can be defined as a first key region.

[0060] The determination of the first key region in the first feature image can be based on the value represented by each pixel in the first feature image. If the first feature image is a grayscale image, the regions where different objects are located in the first feature image can be determined based on the differences between the grayscale values ​​of each pixel, and then the region where each object is located or the region where the local features of each object are located can be determined as the first key region.

[0061] For example, if the first feature image is a depth image, the regions where different objects are located in the first feature image can be determined based on the differences in the depth values ​​of each pixel. For instance, pixels with depth value differences less than a depth threshold can be identified as the regions where the same object is located, and then the regions where each object is located or the regions where the local features of each object are located can be identified as the first key regions.

[0062] Furthermore, after determining at least one first key region from the first feature image, a region in the second feature image that is identical to each first key region can be defined as a second key region. For example, if a certain first key region is the top-left pixel of the first feature image with a size of [missing information], then [missing information]. In the region where the top-left pixel size is [value], the second feature image will have a [value] pixel size of [value]. The area was identified as a second key area.

[0063] See Figure 3 , Figure 3 This is a schematic diagram of a scenario for determining key areas provided in an embodiment of this application. For example... Figure 3 As shown, the first feature image and the second feature image contain the same object, but the object presents different poses in the different feature images. In this case, the identical regions in the first feature image and the second feature image can be respectively identified as the first key region and the second key region.

[0064] If a certain first key region is the area where the container is located, and the image registration status of the first initial image and the second initial image is unregistered, the second key region determined based on the first key region may be the area where the container is located or may not be the area where the container is located.

[0065] Furthermore, if the pixel dimensions (pixel height and pixel width) of each first key region and each second key region are the same, then at least one sub-image can be determined from each first key region and each second key region based on a preset pixel window and a preset sliding rule.

[0066] If the pixel sizes of the first key regions and the second key regions are inconsistent, the first key regions and the second key regions can be scaled to the same pixel height and pixel width, and then at least one sub-image can be determined from each of the first key regions and the second key regions based on the preset pixel window and the preset sliding rules.

[0067] When determining at least one sub-image from each first key region and each second key region, at least one key region combination can be determined first, that is, the first key region and the second key region with the same region are determined as a key region combination, and then multiple sub-images are determined from the first key region and the second key region in each key region combination based on a preset pixel window and a preset sliding rule.

[0068] In each key region combination, the preset pixel windows corresponding to the first and second key regions and the starting positions of the preset pixel windows are the same, and the preset sliding rules corresponding to the first and second key regions within the key region combination are the same.

[0069] Among them, the preset pixel windows corresponding to any two key regions can be the same or different, and the corresponding preset sliding rules can be the same or different. The specific rules can be determined based on the actual application scenario requirements, and no restrictions are imposed here.

[0070] Based on this, for each key region combination, after determining multiple sub-images from the first key region and the second key region in the key region combination, for any sub-image in the first key region, there exists a target sub-image in the sub-image in the second key region, and the area of ​​the target sub-image in the second key region is the same as the area of ​​the sub-image in the first key region.

[0071] Based on this, for each key region combination, at least one image pair can be determined based on the regions of each sub-image corresponding to the first and second key regions in the key region combination within the corresponding key region combination. The first and second sub-images in each image pair belong to different key regions and have the same region within the corresponding key regions.

[0072] See Figure 4 , Figure 4 This is a schematic diagram of a scene for determining image pairs provided in an embodiment of this application. For example... Figure 4 As shown, based on Figure 3 The scene shown can identify a key region combination. Based on a preset pixel window, three sub-images can be determined from the first key region of this key region combination, and three more sub-images can be determined from the second key region of this key region combination. Then, based on the area of ​​each sub-image within its corresponding key region, a sub-image can be determined... Figure 4 The three image pairs shown.

[0073] Step S13: Fuse the first sub-image and the second sub-image in each image pair to obtain a fused image, and determine the image registration state of the first sub-image and the second sub-image in the corresponding image pair based on each fused image.

[0074] In some feasible implementations, when determining the fused image corresponding to each image pair, for the first sub-image and the second sub-image in each image pair, a fused image can be constructed based on the channel values ​​of each pixel of the first sub-image and the second sub-image, wherein each pixel of the fused image includes at least one channel.

[0075] In a depth image, each pixel's channel value is a depth value, while in a grayscale image, each pixel's channel value is a grayscale value. That is, each pixel in both grayscale and depth images has a single channel. In a color image, each pixel corresponds to multiple channel values, and each channel value is the color value of one color channel.

[0076] The number of channels for each pixel in the fused image can be determined based on the actual application scenario requirements, and is not limited here.

[0077] The channel value of each channel of each pixel in the fused image is determined by at least one of the channel values ​​of the first sub-image corresponding to that pixel or the channel values ​​of the second sub-image corresponding to that pixel.

[0078] Specifically, when each pixel of the fused image has a single channel, if the first sub-image and the second sub-image are either depth images or grayscale images, the channel value of the fused image at each pixel can be the channel value of the first sub-image at that pixel, or the channel value of the second sub-image at that pixel, or it can be determined based on the channel values ​​of the first sub-image and the second sub-image at that pixel, such as by adding the channel values ​​or taking the average, etc., without any restrictions.

[0079] As an example, the first sub-image and the second sub-image are depth images. The fusion depth value of the fused image at the corresponding pixel can be obtained based on the depth value of the first sub-image and the second sub-image at the corresponding pixel (such as adding the depth values ​​or taking the average value). Then, the fused image can be obtained based on the fusion depth value of each pixel.

[0080] If the first sub-image is either a depth image or a grayscale image, and the second sub-image is a color image, then the channel value of the fused image at each pixel can be determined by the channel value of the first sub-image at that pixel and the first channel value of the second sub-image at that pixel. This can be achieved by adding the two channel values ​​or taking their average, etc., without restriction. The first channel value of the second sub-image at any pixel can be the color value of any color channel of the second sub-image at that pixel, or it can be the sum, average, or extreme value of the color values ​​of all color channels, without restriction.

[0081] If both the first sub-image and the second sub-image are color images, then the channel value of the fused image at each pixel can be determined by the second channel value of the first sub-image at that pixel and the first channel value of the second sub-image at that pixel. The second channel value of the first sub-image at any pixel can be the color value of any color channel of the first sub-image at that pixel, or it can be the sum, average, or extreme value of the color values ​​of all color channels; no restriction is placed here.

[0082] Optionally, if each pixel of the fused image has multiple channels, and if the first sub-image and the second sub-image are either depth images or grayscale images, then the channel value of each pixel in the fused image can be determined based on the channel value of the first sub-image at that pixel and / or the channel value of the second sub-image at that pixel.

[0083] As an example, if each pixel of the fused image has two channels, then the channel values ​​of the fused image at each pixel can correspond to the channel values ​​of the first sub-image and the second sub-image at that pixel, respectively, or the channel values ​​of the fused image at each pixel can be either the channel values ​​of the first sub-image or the second sub-image at that pixel. For example, if the first and second sub-images are depth images, the depth values ​​of the first and second sub-images at corresponding pixels can be determined as the channel values ​​of one channel of the fused image at that pixel, and then the fused image can be obtained based on the channel values ​​of each channel of each pixel. If the depth value of the first sub-image at a certain pixel is 1 and the depth value of the second sub-image at that pixel is 2, then the channel values ​​of each channel of the fused image at that pixel are 1 and 2, respectively.

[0084] As an example, if each pixel of the fused image has three channels, then the channel values ​​of two channels at each pixel can correspond to the channel values ​​of the first sub-image and the second sub-image at that pixel, respectively. The channel value of the third channel can be any one of the channel values ​​of the first sub-image and the second sub-image at that pixel, or it can be determined based on the channel values ​​of the first sub-image and the second sub-image at that pixel, such as by adding the channel values ​​or taking the average, etc., without restriction. For example, the channel values ​​of each pixel of the fused image can be represented as (channel value A, channel value B), where channel value A and channel value B are the channel values ​​of the first sub-image and the second sub-image at that pixel, respectively. As another example, if the first sub-image is a grayscale image and the second sub-image is a depth image, the grayscale value of a pixel in the first sub-image can be determined as the channel values ​​of the two channels of the fused image at that pixel, and the depth value of the second sub-image at that pixel can be determined as the channel value of one channel of the fused image at that pixel. For example, the channel values ​​of each pixel of the fused image can be represented as (grayscale value, grayscale value, depth value).

[0085] If the first sub-image is either a depth image or a grayscale image, and the second sub-image is a color image, then the channel value of the fused image at each pixel can be determined based on the channel value of the first sub-image at that pixel and / or at least one channel value of the second sub-image at that pixel.

[0086] As an example, if each pixel of the fused image has two channels, then each channel value of the fused image at each pixel can correspond to the channel value of the first sub-image at that pixel and the first channel value of the second sub-image at that pixel, respectively, or each channel value of the fused image at each pixel can be either the channel value of the first sub-image at that pixel or the first channel value of the second sub-image at that pixel.

[0087] As an example, if each pixel of the fused image has three channels, then the channel value of one channel of the fused image at each pixel can be the channel value of the first sub-image at that pixel, and the channel values ​​of the other two channels can be the channel values ​​of any two color channels of the second sub-image at that pixel, or both can be the first channel value of the second sub-image at that pixel.

[0088] As an example, if each pixel of the fused image has three channels, then the channel values ​​of two channels of the fused image at each pixel are the channel values ​​of the first sub-image at that pixel, and the channel value of the other channel is the first channel value of the second sub-image at that pixel.

[0089] Optionally, when the number of channels per pixel in the fused image is multi-channel, if both the first sub-image and the second sub-image are color images, then the channel value of each pixel in the fused image can be determined by the channel values ​​of the first sub-image and the second sub-image in the corresponding channels, such as adding the two channel values ​​or taking the average, etc., without limitation. For example, if both the first sub-image and the second sub-image are R, G, and B three-channel color images, then the fused image is also an R, G, and B three-channel color image, and the channel values ​​of the R, G, and B channels of the fused image at each pixel are determined by the channel values ​​of the first sub-image and the second sub-image in the R, G, and B channels, respectively.

[0090] In some feasible implementations, after determining the fused image corresponding to each image pair, feature processing can be performed on each fused image to obtain fusion features. Then, based on the fusion features, a prediction result representing the image registration state of the first and second sub-images in the image pair corresponding to each fused image is obtained. Based on the prediction result, it is determined whether the first and second sub-images in the image pair corresponding to each fused image are in a registered or unregistered state. For example... Figure 5 As shown, Figure 5 This is a schematic diagram of a scenario for determining the image registration state according to an embodiment of this application. The image registration state of the sub-images in each image pair can be determined by determining the fused image corresponding to each image pair.

[0091] The process of performing feature processing on each fused image to obtain fused features can be based on neural networks, such as recurrent neural networks or residual networks (e.g., ResNet18). The specific method can be determined based on the actual application scenario requirements and is not limited here.

[0092] For example, such as Figure 6 As shown, Figure 6 This is a schematic diagram of the neural network structure provided in an embodiment of this application. For each fused image, the fused image can be downsampled based on a downsampling convolutional layer, and the downsampling convolutional features are input into a residual network. The output features of the residual network are determined as the fused features corresponding to the fused image. The residual network includes multiple residual blocks, and the network layer settings of each residual block can be determined based on the actual application scenario requirements, and are not limited here. Further processing of the fused features based on pooling layers and fully connected layers yields the image registration states of the first and second sub-images in the image pair corresponding to the fused image.

[0093] For example, such as Figure 7 As shown, Figure 7This is another schematic diagram of the neural network structure provided in this application embodiment. For each fused image, the fused image can be downsampled based on a downsampling convolutional layer, and the downsampling convolutional features are then input into a first residual network. The output features of the first residual network are used as the inputs of the second and third residual networks. The output features of the second residual network are processed based on pooling layers and fully connected layers to obtain a first initial fused feature, and the output features of the third residual network are processed based on fully connected layers to obtain a second initial fused feature. The first and second initial fused features are further fused, such as by adding or averaging the corresponding feature values, to finally obtain the fused features corresponding to the fused image. The fused features are then processed based on fully connected layers to obtain the image registration state of the first and second sub-images in the image pair corresponding to the fused image.

[0094] The specific network layer settings in the first residual network, the second residual network, and the third residual network can be determined based on the actual application scenario requirements, and are not restricted here.

[0095] In this system, the dimension of the output feature of any residual block is less than or equal to the dimension of the input feature of the previous residual block. The third residual network effectively avoids the situation where the initial fused features are not prominent due to the small dimension, further improving the stability of the image registration state.

[0096] In some feasible implementations, the image registration state of the first and second sub-images in the corresponding image pair based on each fused image can be achieved through a state detection model.

[0097] The state detection model was trained in the following way:

[0098] Multiple sample feature image pairs are determined. Each sample feature image pair includes a first sample feature image of a first initial sample image and a second sample feature image of a second initial sample image. The first initial sample image and the second initial sample image correspond to the same scene, such as including the same object. The first sample feature image and the second sample feature image are any one of a depth image, a grayscale image, or a color image.

[0099] Furthermore, the first and second sample feature images in each sample feature image pair are fused to obtain a sample fused image. The specific implementation can be based on... Figure 1 The implementation method of fusing the first sub-image and the second sub-image shown in step S13 will not be described again here.

[0100] After obtaining the sample fusion image corresponding to each sample feature image pair, each sample fusion image can be input into the initial model to obtain the prediction result corresponding to each sample fusion image. Based on the prediction result, the prediction image registration state of the first sample feature image and the second sample feature image in the sample feature image pair corresponding to each sample fusion image can be determined.

[0101] Furthermore, the training loss value is determined based on the actual image registration state and the predicted image registration state of the first and second sample feature images in each sample feature image pair. The initial model is then trained based on the training loss value and each sample feature image pair until the training loss value meets the training termination condition. The model at the end of training is then determined as the state detection model.

[0102] The training loss value mentioned above can be determined by the cross-entropy loss function or other loss functions. The specific value can be determined based on the actual application scenario requirements, and there are no restrictions here.

[0103] The training termination condition mentioned above can be that the training loss value reaches convergence, or that the training loss value no longer changes after a preset number of consecutive training sessions; there are no restrictions on this.

[0104] The network structure of the initial model described above can be: Figure 6 or Figure 7 The network structure shown can also be other network structures, and no restrictions are placed here.

[0105] In some feasible implementations, each sample feature image pair can be determined based on at least one set of sample initial images with a registered state (hereinafter referred to as the third sample initial image and the fourth sample initial image for convenience). For each set of sample initial images, the third sample feature image of the third sample initial image and the fourth sample feature image of the fourth sample initial image in the set of sample initial images can be determined, and the third sample feature image and the fourth sample feature image are determined as a sample feature image pair, and the actual image registration state of the sample feature image pair is a registered state. Further, the image content of at least one of the third sample feature image or the fourth sample feature image is translated to generate multiple sets of sample feature image pairs with an actual image registration state of unregistered.

[0106] Optionally, for each set of initial images where the image registration state is already registered, it can also be based on... Figure 1In step S12, multiple image pairs corresponding to the initial images of the sample group are determined, and each image pair is defined as a sample feature image pair, with the actual image registration state of the sample feature pair being registered. Furthermore, any sample feature image from the multiple image pairs can be shifted to generate multiple sets of sample feature image pairs with an actual image registration state of unregistered.

[0107] Step S14: Based on the image registration status of each image pair, determine the image registration status of the first initial image and the second initial image.

[0108] In some feasible implementations, after determining the image registration status of each image pair, the image registration status of the first initial image and the second initial image can be determined based on the image registration status of each image pair.

[0109] Specifically, if the number of image pairs in the image registration state reaches a certain proportion of all image pairs, then the first initial image and the second initial image can be determined to be in the registered state; otherwise, the first initial image and the second initial image can be determined to be in the unregistered state.

[0110] Optionally, at least one key region combination may also be determined, each key region combination including a first key region and a second key region corresponding to the first key region. Further, a first number of image pairs corresponding to each key region combination and a second number of image pairs in the image pairs corresponding to the key region combination whose image registration state is unregistered may be determined.

[0111] For each key region combination, the image registration state of the first and second key regions within that key region combination can be determined based on the first and second quantities corresponding to that key region combination. Therefore, the image registration state of the first and second initial images can be determined based on the image registration state of each key region combination.

[0112] In some feasible implementations, when determining the image registration status of each key region combination, a first proportion of the second quantity corresponding to the key region combination to the corresponding first quantity can be determined, that is, the second quantity of image pairs in the key region combination whose image registration status is unregistered is determined to be the first proportion of the first quantity of image pairs corresponding to the key region combination.

[0113] When the first proportion is greater than or equal to the first threshold, it indicates that a certain number of image pairs in the image pairs corresponding to the key region combination are in an unregistered state. Therefore, it can be determined that the image registration state of the first key region and the second key region in the key region combination is unregistered.

[0114] When the first proportion is less than the first threshold, it means that most of the image pairs in the image pairs corresponding to the key region combination are in the image registration state, and thus the image registration state of the first key region and the second key region in the key region combination is in the image registration state.

[0115] Alternatively, when the first proportion is less than the first threshold, the registration confidence of the key region combination can be further determined based on the first proportion and the second quantity corresponding to the key region combination. When the registration confidence is greater than the confidence threshold, the image registration status of the first and second key regions in the key region combination can be determined as a registered state. When the registration confidence is not greater than the confidence threshold, the image registration status of the first and second key regions in the key region combination can be re-determined based on a preset method, such as manual verification.

[0116] When determining the registration confidence level of this key region combination, it can be calculated based on RST. R is a preset value, such as 100, S is the first proportion, and T is the second quantity.

[0117] In some feasible implementations, when determining the image registration status of the first initial image and the second initial image based on the image registration status of each key region combination, if the image registration status of each key region combination is a registered state, then the image registration status of the first initial image and the second initial image can be determined to be a registered state. If the image registration status of at least one key region combination is an unregistered state, then the image registration status of the first initial image and the second initial image can be determined to be an unregistered state.

[0118] like Figure 8 As shown, Figure 8 This is another scenario diagram illustrating the determination of image registration status provided in an embodiment of this application. For example... Figure 8 It can be determined Figure 3 The contours of the same object in the first and second key regions shown differ, therefore it can be determined that the image registration state of the first and second key regions is unregistered, and the first and second feature images only correspond to each other. Figure 3 In the case of a key region combination, it can be determined Figure 2 The first and second initial images in the image are in an unregistered state.

[0119] The training process of the state detection model in this embodiment can be implemented based on machine learning in artificial intelligence. Artificial intelligence is the theory, method, technology, and application system that uses digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to obtain optimal results. In other words, artificial intelligence is a comprehensive technology in computer science that attempts to understand the essence of intelligence and produce a new kind of intelligent machine that can react in a way similar to human intelligence. Artificial intelligence studies the design principles and implementation methods of various intelligent machines, enabling them to have perception, reasoning, and decision-making functions. Machine learning specifically studies how computers can simulate or implement human learning behavior to acquire new knowledge or skills, reorganize existing knowledge structures to continuously improve their performance, and thus implement the training process of the state detection model in this embodiment.

[0120] The image processing involved in the embodiments of this application can be implemented based on computer vision technology. Computer vision is a science that studies how to make machines "see". More specifically, it refers to machine vision, which uses cameras and computers to replace human eyes to identify and measure targets, and then performs further graphic processing, such as determining depth maps or grayscale images.

[0121] The sample feature image pairs in this application embodiment can be stored in a database, cloud storage, or blockchain, depending on the actual application scenario requirements, and are not limited here.

[0122] In short, a database can be viewed as an electronic filing cabinet—a place to store electronic files. In this application, it can be used to store pairs of sample feature images. Blockchain is a new application model of computer technologies such as distributed data storage, peer-to-peer transmission, consensus mechanisms, and encryption algorithms. Essentially, a blockchain is a decentralized database, a chain of data blocks linked together using cryptographic methods. In this application, each data block in the blockchain can store pairs of sample feature images. Cloud storage is a new concept that extends and develops from cloud computing. It refers to the collaborative storage of various types of storage devices (also called storage nodes) across a network through application software or application interfaces, using cluster applications, grid technology, and distributed storage file systems to jointly store pairs of sample feature images.

[0123] The data processing and computation processes involved in the embodiments of this application can be performed based on cloud computing and other methods. Cloud computing is a product of the development and integration of traditional computer and network technologies such as grid computing, distributed computing, parallel computing, utility computing, network storage technologies, virtualization, and load balancing. Based on cloud computing, the efficiency of data processing and computation in the embodiments of this application can be improved.

[0124] See Figure 9 , Figure 9 This is a schematic diagram of the structure of the image processing apparatus provided in the embodiments of this application. The image processing apparatus provided in the embodiments of this application includes:

[0125] The image determination module 91 is used to determine a first feature image of a first initial image and a second feature image of a second initial image, wherein the first initial image and the second initial image correspond to the same scene, and the first feature image and the second feature image are any one of a depth image, a grayscale image or a color image;

[0126] The image segmentation module 92 is used to determine at least one sub-image from the first feature image and the second feature image respectively based on a preset pixel window, and to determine at least one image pair based on the region of each sub-image in the corresponding feature image, wherein the first sub-image and the second sub-image in each image pair belong to different feature images and have the same region in the corresponding feature image.

[0127] The image fusion module 93 is used to fuse the first sub-image and the second sub-image in each of the above image pairs to obtain a fused image, and to determine the image registration state of the first sub-image and the second sub-image in the corresponding image pair based on each of the above fused images;

[0128] The state determination module 94 is used to determine the image registration state of the first initial image and the second initial image based on the image registration state of each of the above image pairs, wherein the image registration state is either a registered state or an unregistered state.

[0129] In some feasible implementations, for each of the first and second sub-images in the above image pair, the image fusion module 93 is configured to:

[0130] A fused image is constructed based on the channel values ​​of each pixel in the first sub-image and the second sub-image. Each pixel in the fused image includes at least one channel, and the channel value of each channel is determined by at least one of the channel values ​​of the first sub-image corresponding to the pixel or the channel values ​​of the second sub-image corresponding to the pixel.

[0131] In this image, each pixel's channel value is a depth value; in a grayscale image, each pixel's channel value is a grayscale value; and in a color image, each pixel corresponds to multiple channel values, with each channel value being the color value of a color channel.

[0132] In some feasible implementations, the image segmentation module 92 described above is used for:

[0133] At least one first key region in the first feature image is determined, and a region in the second feature image that is the same as each of the first key regions is determined as a second key region, and each of the first key regions is used to characterize the local features of the first feature image.

[0134] At least one sub-image is determined from each of the aforementioned first key regions and each of the aforementioned second key regions based on a preset pixel window and a preset sliding rule.

[0135] In some feasible implementations, the aforementioned state determination module 94 is used for:

[0136] Determine at least one combination of key regions, each of the above key region combinations including a first key region and a second key region corresponding to the first key region;

[0137] Determine the first number of image pairs corresponding to each of the above key region combinations, and the second number of image pairs in the corresponding image pairs whose image registration status is unregistered;

[0138] For each of the above key region combinations, the image registration status of the first key region and the second key region in the key region combination is determined based on the first number and the second number corresponding to the key region combination.

[0139] Based on the image registration status of each of the above-mentioned key region combinations, the image registration status of the first initial image and the second initial image is determined.

[0140] In some feasible implementations, for each of the above-mentioned key region combinations, the state determination module 94 is used to:

[0141] Determine the first proportion of the second quantity corresponding to the first quantity corresponding to the key area combination;

[0142] In response to the first proportion corresponding to the key region combination being less than a first threshold, the registration confidence of the key region combination is determined based on the first proportion and the second quantity corresponding to the key region combination, and the image registration status of the first key region and the second key region in the key region combination is determined based on the above registration confidence.

[0143] In response to the first proportion corresponding to the key region combination being greater than or equal to the first threshold mentioned above, the image registration status of the first key region and the second key region in the key region combination is determined to be unregistered.

[0144] In some feasible implementations, the aforementioned state determination module 94 is used for:

[0145] In response to the image registration state of at least one of the above-mentioned key regions being unregistered, the image registration state of the first initial image and the second initial image is determined to be unregistered.

[0146] In response to the fact that the image registration status of each of the above-mentioned key region combinations is a registered state, the image registration status of the first initial image and the second initial image is determined to be a registered state.

[0147] In some feasible implementations, the image registration state of the first and second sub-images in the corresponding image pair determined based on each of the fused images is achieved through a state detection model. This state detection model is trained using a training device, which is used to:

[0148] Multiple sample feature image pairs are determined, each of the sample feature image pairs including a first sample feature image of a first sample initial image and a second sample feature image of a second sample initial image, wherein the first sample initial image and the second sample initial image correspond to the same scene, and the first sample feature image and the second sample feature image are any one of a depth image, a grayscale image or a color image;

[0149] The first and second sample feature images in each of the above sample feature image pairs are fused to obtain a sample fusion image. Each of the above sample fusion images is input into the initial model to obtain the prediction result corresponding to each of the above sample fusion images. Based on the above prediction result, the prediction image registration state of the first and second sample feature images in each of the above sample fusion images is determined.

[0150] Based on the actual image registration state and predicted image registration state of the first and second sample feature images in each of the above sample feature image pairs, a training loss value is determined. The above initial model is trained based on the above training loss value and each of the above sample feature image pairs until the above training loss value meets the training termination condition, and the model at the end of training is determined as the above state detection model.

[0151] In specific implementation, the above-mentioned device can perform the above-described actions through its built-in functional modules. Figure 1 The implementation methods provided for each step are detailed in the above-mentioned implementation methods and will not be repeated here.

[0152] See Figure 10 , Figure 10 This is a schematic diagram of the structure of the electronic device provided in an embodiment of this application. For example... Figure 10 As shown, the electronic device 1000 in this embodiment may include: a processor 1001, a network interface 1004, and a memory 1005. Furthermore, the electronic device 1000 may also include: an object interface 1003, and at least one communication bus 1002. The communication bus 1002 is used to implement communication between these components. The object interface 1003 may include a display screen and a keyboard; optionally, the object interface 1003 may also include a standard wired interface or a wireless interface. The network interface 1004 may optionally include a standard wired interface or a wireless interface (such as a Wi-Fi interface). The memory 1004 may be a high-speed RAM or non-volatile memory (NVM), such as at least one disk storage device. The memory 1005 may optionally be at least one storage device located remotely from the processor 1001. Figure 10 As shown, the memory 1005, which is a computer-readable storage medium, may include an operating system, a network communication module, an object interface module, and a device control application.

[0153] exist Figure 10 In the illustrated electronic device 1000, the network interface 1004 provides network communication functionality; the object interface 1003 is primarily used to provide an input interface for the user; and the processor 1001 can be used to call the device control application stored in the memory 1005 to achieve:

[0154] A first feature image of a first initial image and a second feature image of a second initial image are determined, wherein the first initial image and the second initial image correspond to the same scene, and the first feature image and the second feature image are any one of a depth image, a grayscale image, or a color image;

[0155] At least one sub-image is determined from the first feature image and the second feature image based on a preset pixel window. At least one image pair is determined based on the region of each sub-image in the corresponding feature image. The first sub-image and the second sub-image in each image pair belong to different feature images and have the same region in the corresponding feature image.

[0156] The first and second sub-images in each of the above image pairs are fused to obtain a fused image, and the image registration state of the first and second sub-images in the corresponding image pair is determined based on each of the above fused images;

[0157] Based on the image registration status of each of the above image pairs, the image registration status of the first initial image and the second initial image is determined, and the image registration status is either a registered state or an unregistered state.

[0158] In some feasible implementations, for each of the first and second sub-images in the above image pair, the processor 1001 is used to:

[0159] A fused image is constructed based on the channel values ​​of each pixel in the first sub-image and the second sub-image. Each pixel in the fused image includes at least one channel, and the channel value of each channel is determined by at least one of the channel values ​​of the first sub-image corresponding to the pixel or the channel values ​​of the second sub-image corresponding to the pixel.

[0160] In this image, each pixel's channel value is a depth value; in a grayscale image, each pixel's channel value is a grayscale value; and in a color image, each pixel corresponds to multiple channel values, with each channel value being the color value of a color channel.

[0161] In some feasible implementations, the processor 1001 described above is used for:

[0162] At least one first key region in the first feature image is determined, and a region in the second feature image that is the same as each of the first key regions is determined as a second key region, and each of the first key regions is used to characterize the local features of the first feature image.

[0163] At least one sub-image is determined from each of the aforementioned first key regions and each of the aforementioned second key regions based on a preset pixel window and a preset sliding rule.

[0164] In some feasible implementations, the processor 1001 described above is used for:

[0165] Determine at least one combination of key regions, each of the above key region combinations including a first key region and a second key region corresponding to the first key region;

[0166] Determine the first number of image pairs corresponding to each of the above key region combinations, and the second number of image pairs in the corresponding image pairs whose image registration status is unregistered;

[0167] For each of the above key region combinations, the image registration status of the first key region and the second key region in the key region combination is determined based on the first number and the second number corresponding to the key region combination.

[0168] Based on the image registration status of each of the above-mentioned key region combinations, the image registration status of the first initial image and the second initial image is determined.

[0169] In some feasible implementations, for each of the above-described key region combinations, the processor 1001 is used to:

[0170] Determine the first proportion of the second quantity corresponding to the first quantity corresponding to the key area combination;

[0171] In response to the first proportion corresponding to the key region combination being less than a first threshold, the registration confidence of the key region combination is determined based on the first proportion and the second quantity corresponding to the key region combination, and the image registration status of the first key region and the second key region in the key region combination is determined based on the above registration confidence.

[0172] In response to the first proportion corresponding to the key region combination being greater than or equal to the first threshold mentioned above, the image registration status of the first key region and the second key region in the key region combination is determined to be unregistered.

[0173] In some feasible implementations, the processor 1001 described above is used for:

[0174] In response to the image registration state of at least one of the above-mentioned key regions being unregistered, the image registration state of the first initial image and the second initial image is determined to be unregistered.

[0175] In response to the fact that the image registration status of each of the above-mentioned key region combinations is a registered state, the image registration status of the first initial image and the second initial image is determined to be a registered state.

[0176] In some feasible implementations, the image registration state of determining the first and second sub-images in the corresponding image pair based on each of the aforementioned fused images is achieved through a state detection model, which is trained by the processor 1001 in the following manner:

[0177] Multiple sample feature image pairs are determined, each of the sample feature image pairs including a first sample feature image of a first sample initial image and a second sample feature image of a second sample initial image, wherein the first sample initial image and the second sample initial image correspond to the same scene, and the first sample feature image and the second sample feature image are any one of a depth image, a grayscale image or a color image;

[0178] The first and second sample feature images in each of the above sample feature image pairs are fused to obtain a sample fusion image. Each of the above sample fusion images is input into the initial model to obtain the prediction result corresponding to each of the above sample fusion images. Based on the above prediction result, the prediction image registration state of the first and second sample feature images in each of the above sample fusion images is determined.

[0179] Based on the actual image registration state and predicted image registration state of the first and second sample feature images in each of the above sample feature image pairs, a training loss value is determined. The above initial model is trained based on the above training loss value and each of the above sample feature image pairs until the above training loss value meets the training termination condition, and the model at the end of training is determined as the above state detection model.

[0180] It should be understood that in some feasible implementations, the processor 1001 described above may be a central processing unit (CPU), or it may be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or any conventional processor. The memory may include read-only memory and random access memory, and provides instructions and data to the processor. A portion of the memory may also include non-volatile random access memory. For example, the memory may also store device type information.

[0181] In specific implementation, the aforementioned electronic device 1000 can perform the above-described actions through its built-in functional modules. Figure 1 The implementation methods provided for each step are detailed in the above-mentioned implementation methods and will not be repeated here.

[0182] This application also provides a computer-readable storage medium storing a computer program that is executed by a processor to implement... Figure 1 The methods provided in each step are detailed in the implementation methods provided in the above steps, and will not be repeated here.

[0183] The aforementioned computer-readable storage medium can be an internal storage unit of the processing apparatus or electronic device provided in any of the foregoing embodiments, such as a hard disk or memory of the electronic device. The computer-readable storage medium can also be an external storage device of the electronic device, such as a plug-in hard disk, smart media card (SMC), secure digital (SD) card, flash card, etc., equipped on the electronic device. The aforementioned computer-readable storage medium can also include magnetic disks, optical disks, read-only memory (ROM), or random access memory (RAM), etc. Furthermore, the computer-readable storage medium can include both internal storage units and external storage devices of the electronic device. The computer-readable storage medium is used to store the computer program and other programs and data required by the electronic device. The computer-readable storage medium can also be used to temporarily store data that has been output or will be output.

[0184] This application provides a computer program product, which includes a computer program that is executed by a processor. Figure 1 The methods provided for each step in the process.

[0185] The terms "first," "second," etc., used in the claims, description, and drawings of this application are used to distinguish different objects, not to describe a specific order. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or electronic device that includes a series of steps or units is not limited to the listed steps or units, but may optionally include steps or units not listed, or may optionally include other steps or units inherent to these processes, methods, products, or electronic devices. References to "embodiment" herein mean that a particular feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of this application. The presentation of this phrase in various places in the specification does not necessarily refer to the same embodiment, nor is it a separate or alternative embodiment mutually exclusive with other embodiments. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described herein can be combined with other embodiments. The term "and / or" as used in this application's description and appended claims refers to any combination of one or more of the associated listed items and all possible combinations, and includes such combinations.

[0186] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Those skilled in the art can implement the described functions using different methods for each specific application, but such implementations should not be considered beyond the scope of this application.

[0187] The above-disclosed embodiments are merely preferred embodiments of this application and should not be construed as limiting the scope of this application. Therefore, any equivalent variations made in accordance with the claims of this application shall still fall within the scope of this application.

Claims

1. An image processing method, characterized by, The method includes: A first feature image of a first initial image and a second feature image of a second initial image are determined. The first initial image and the second initial image correspond to the same scene. The first feature image and the second feature image are any one of a depth image, a grayscale image, or a color image. The first initial image and the second initial image are two images of a target object acquired at different times, through different sensors, and / or under different conditions. At least one first key region in the first feature image is determined, and a region in the second feature image that is the same as each of the first key regions is determined as a second key region, and each first key region is used to characterize the local features of the first feature image. Based on a preset pixel window and a preset sliding rule, at least one sub-image is determined from each of the first key regions and each of the second key regions. At least one image pair is determined based on the region of each sub-image in the corresponding feature image. The first sub-image and the second sub-image in each image pair belong to different feature images and have the same region in the corresponding feature image. The first and second sub-images in each image pair are fused to obtain a fused image. Feature processing is performed on each fused image to obtain fused features. Based on the fused features of each fused image, the image registration state of the first and second sub-images in the corresponding image pair is determined. Based on the image registration status of each image pair, the image registration status of the first initial image and the second initial image is determined, wherein the image registration status is either a registered state or an unregistered state; The process of performing feature processing on each fused image to obtain fused features includes: Each of the fused images is downsampled based on a downsampling convolutional layer; The downsampled convolutional features are input into the first residual network; The output features of the first residual network are used as the inputs of the second and third residual networks. The output features of the second residual network are processed based on the pooling layer and the fully connected layer to obtain the first initial fusion feature. The output features of the third residual network are processed based on the fully connected layer to obtain the second initial fusion feature. The first initial fusion feature and the second initial fusion feature are fused to obtain the fusion feature corresponding to the fusion image; wherein the dimension of the output feature of any residual block is less than or equal to the dimension of the input feature of the previous residual block.

2. The method of claim 1, wherein, For each image pair, the first sub-image and the second sub-image are fused to obtain a fused image, including: A fused image is constructed based on the channel values ​​of each pixel in the first sub-image and the second sub-image; In this image, each pixel's channel value is a depth value; in a grayscale image, each pixel's channel value is a grayscale value; and in a color image, each pixel corresponds to multiple channel values, with each channel value being the color value of a color channel.

3. The method according to claim 1, characterized in that, Determining the image registration status of the first initial image and the second initial image based on the image registration status of each image pair includes: Determine at least one combination of key regions, each combination of key regions including a first key region and a second key region corresponding to the first key region; Determine a first number of image pairs corresponding to each key region combination, and a second number of image pairs in the corresponding image pairs whose image registration status is unregistered. For each key region combination, the image registration status of the first key region and the second key region in the key region combination is determined based on the first number and the second number corresponding to the key region combination. Based on the image registration status of each of the key region combinations, the image registration status of the first initial image and the second initial image is determined.

4. The method according to claim 3, characterized in that, For each key region combination, determining the image registration status of the first and second key regions in the key region combination based on the first and second quantities corresponding to that key region combination includes: Determine the first proportion of the second quantity corresponding to the first quantity corresponding to the key area combination; In response to the first proportion corresponding to the key region combination being less than a first threshold, the registration confidence of the key region combination is determined based on the first proportion and the second quantity corresponding to the key region combination, and the image registration status of the first key region and the second key region in the key region combination is determined based on the registration confidence. In response to the first proportion corresponding to the key region combination being greater than or equal to the first threshold, the image registration status of the first key region and the second key region in the key region combination is determined to be unregistered.

5. The method according to claim 3, characterized in that, The process of determining the image registration states of the first initial image and the second initial image based on the image registration states of each of the key region combinations includes: In response to the image registration state of at least one of the key region combinations being unregistered, the image registration states of the first initial image and the second initial image are determined to be unregistered. In response to the fact that the image registration status of each of the key region combinations is a registered state, the image registration status of the first initial image and the second initial image is determined to be a registered state.

6. The method according to claim 1, characterized in that, The image registration state of determining the first and second sub-images in the corresponding image pair based on each fused image is achieved through a state detection model, which is trained in the following manner: Multiple sample feature image pairs are determined, each sample feature image pair including a first sample feature image of a first sample initial image and a second sample feature image of a second sample initial image, the first sample initial image and the second sample initial image correspond to the same scene, and the first sample feature image and the second sample feature image are any one of depth image, grayscale image or color image; The first sample feature image and the second sample feature image in each sample feature image pair are fused to obtain a sample fusion image. Each sample fusion image is input into the initial model to obtain the prediction result corresponding to each sample fusion image. Based on the prediction result, the prediction image registration state of the first sample feature image and the second sample feature image in the sample feature image pair corresponding to each sample fusion image is determined. Based on the actual image registration state and predicted image registration state of the first and second sample feature images in each sample feature image pair, a training loss value is determined. The initial model is trained based on the training loss value and each sample feature image pair until the training loss value meets the training termination condition, at which point training stops, and the model at the end of training is determined as the state detection model.

7. An image processing apparatus, characterized in that, The device includes: The image determination module is used to determine a first feature image of a first initial image and a second feature image of a second initial image, wherein the first initial image and the second initial image correspond to the same scene, and the first feature image and the second feature image are any one of a depth image, a grayscale image, or a color image; wherein the first initial image and the second initial image are two images of a target object acquired at different times, through different sensors, and / or under different conditions. An image segmentation module is used to determine at least one first key region in the first feature image, and to determine a region in the second feature image that is the same as each of the first key regions as a second key region. Each first key region is used to characterize the local features of the first feature image. Based on a preset pixel window and a preset sliding rule, at least one sub-image is determined from each of the first key regions and each of the second key regions, and at least one image pair is determined based on the region of each sub-image in the corresponding feature image. The first sub-image and the second sub-image in each image pair belong to different feature images and have the same region in the corresponding feature image. The image fusion module is used to fuse the first sub-image and the second sub-image in each image pair to obtain a fused image, perform feature processing on each fused image to obtain fusion features, and determine the image registration state of the first sub-image and the second sub-image in the corresponding image pair based on the fusion features of each fused image; A state determination module is used to determine the image registration state of the first initial image and the second initial image based on the image registration state of each image pair, wherein the image registration state is either a registered state or an unregistered state. Specifically, the image fusion module is used to: downsample each fused image based on a downsampling convolutional layer; input the downsampled convolutional features into a first residual network; use the output features of the first residual network as inputs to a second and third residual network, process the output features of the second residual network based on a pooling layer and a fully connected layer to obtain a first initial fusion feature, process the output features of the third residual network based on a fully connected layer to obtain a second initial fusion feature; fuse the first initial fusion feature and the second initial fusion feature to obtain the fusion feature corresponding to the fused image; wherein the dimension of the output feature of any residual block is less than or equal to the dimension of the input feature of the previous residual block.

8. The image processing apparatus according to claim 7, characterized in that, For each sub-image and second sub-image in the image pair, the image fusion module is configured to: A fused image is constructed based on the channel values ​​of each pixel in the first sub-image and the second sub-image; In this image, each pixel's channel value is a depth value; in a grayscale image, each pixel's channel value is a grayscale value; and in a color image, each pixel corresponds to multiple channel values, with each channel value being the color value of a color channel.

9. The image processing apparatus according to claim 7, characterized in that, The state determination module is used for: Determine at least one combination of key regions, each combination of key regions including a first key region and a second key region corresponding to the first key region; Determine a first number of image pairs corresponding to each key region combination, and a second number of image pairs in the corresponding image pairs whose image registration status is unregistered. For each key region combination, the image registration status of the first key region and the second key region in the key region combination is determined based on the first number and the second number corresponding to the key region combination. Based on the image registration status of each of the key region combinations, the image registration status of the first initial image and the second initial image is determined.

10. The image processing apparatus according to claim 9, characterized in that, For each of the aforementioned key region combinations, the state determination module is configured to: Determine the first proportion of the second quantity corresponding to the first quantity corresponding to the key area combination; In response to the first proportion corresponding to the key region combination being less than a first threshold, the registration confidence of the key region combination is determined based on the first proportion and the second quantity corresponding to the key region combination, and the image registration status of the first key region and the second key region in the key region combination is determined based on the registration confidence. In response to the first proportion corresponding to the key region combination being greater than or equal to the first threshold, the image registration status of the first key region and the second key region in the key region combination is determined to be unregistered.

11. The image processing apparatus according to claim 9, characterized in that, The state determination module is used for: In response to the image registration state of at least one of the key region combinations being unregistered, the image registration states of the first initial image and the second initial image are determined to be unregistered. In response to the fact that the image registration status of each of the key region combinations is a registered state, the image registration status of the first initial image and the second initial image is determined to be a registered state.

12. The image processing apparatus according to claim 7, characterized in that, The determination of the image registration state of the first and second sub-images in the corresponding image pair based on each fused image is achieved through a state detection model, which is trained using a training device. The training device is used to: Multiple sample feature image pairs are determined, each sample feature image pair including a first sample feature image of a first sample initial image and a second sample feature image of a second sample initial image, the first sample initial image and the second sample initial image correspond to the same scene, and the first sample feature image and the second sample feature image are any one of depth image, grayscale image or color image; The first sample feature image and the second sample feature image in each sample feature image pair are fused to obtain a sample fusion image. Each sample fusion image is input into the initial model to obtain the prediction result corresponding to each sample fusion image. Based on the prediction result, the prediction image registration state of the first sample feature image and the second sample feature image in the sample feature image pair corresponding to each sample fusion image is determined. Based on the actual image registration state and predicted image registration state of the first and second sample feature images in each sample feature image pair, a training loss value is determined. The initial model is trained based on the training loss value and each sample feature image pair until the training loss value meets the training termination condition, at which point training stops, and the model at the end of training is determined as the state detection model.

13. An electronic device, characterized in that, It includes a processor and a memory, which are interconnected; The memory is used to store computer programs; The processor is configured to perform the method as described in any one of claims 1 to 6 when the computer program is invoked.

14. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that is executed by a processor to implement the method of any one of claims 1 to 6.

Images