Image processing method, device and readable storage medium

Abstract

The present application provides an image processing method, device, and readable storage medium. The method includes: acquiring a first image and a second image of the same scene; extracting feature points from the first image and the second image, respectively, to obtain a first feature point set for the first image and a second feature point set for the second image; obtaining semantic segmentation information for the first image, the semantic segmentation information including pixel point identification information corresponding to multiple types of targets in the first image; filtering the first feature point set based on the semantic segmentation information to obtain a third feature point set, the third feature point set including feature points corresponding to non-moving targets in the first image; and performing feature point matching on the third feature point set and the second feature point set to obtain a matched feature point pair. This method can improve the accuracy of image feature point matching and reduce the latency and power consumption of feature point matching.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to an image processing method, device, and readable storage medium. Background Art

[0002] Feature matching technology, a hot topic in computer vision research, directly extracts and matches feature points from images and serves as the underlying input for many computer vision processing tasks. It can be applied to scenarios such as image stitching, 3D reconstruction, simultaneous localization and mapping, object detection and tracking, and positioning and navigation.

[0003] Feature matching is a fundamental and critical technology, and in many application scenarios, extremely high requirements are placed on its accuracy. In multi-processing visual tasks, matching errors will gradually accumulate, severely restricting the effective implementation of the final visual task. For example, in the task of solving camera motion parameters based on feature point matching results to achieve Structure From Motion (SFM), incorrect matching will lead to incorrect estimation of the camera pose. If the underlying technology makes incorrect estimates, the 3D reconstruction results of the SFM task will deviate significantly from the actual situation. Therefore, improving the accuracy of feature matching has very important practical application value. Summary of the Invention

[0004] The present application provides an image processing method, device, and readable storage medium, which can improve the accuracy of image feature point matching and reduce the delay and power consumption of feature point matching.

[0005] In a first aspect, an image processing method is provided, comprising:

[0006] Acquire a first image and a second image of the same scene;

[0007] Extracting feature points from the first image and the second image respectively to obtain a first feature point set of the first image and a second feature point set of the second image;

[0008] Acquire semantic segmentation information of the first image, where the semantic segmentation information includes pixel identification information corresponding to multiple types of objects in the first image;

[0009] Filtering the first feature point set according to the semantic segmentation information to obtain a third feature point set, where the third feature point set includes feature points corresponding to non-moving objects in the first image;

[0010] Feature point matching is performed on the third feature point set and the second feature point set to obtain matching feature point pairs.

[0011] In the above scheme, the processing device filters the non-motion type feature points in the image, performs feature point matching on the filtered non-motion type feature points, and obtains feature point matching pixel pairs of the two images, which can improve the accuracy of feature point matching in dynamic scenes and reduce processing delay and power consumption overhead.

[0012] Optionally, obtaining semantic segmentation information of the first image includes:

[0013] The first image is input into a semantic segmentation intelligent model to obtain the semantic segmentation information output by the semantic segmentation intelligent model.

[0014] In the above solution, the processing device can apply artificial intelligence technology to use a semantic segmentation intelligent model to infer semantic segmentation information to improve the accuracy of semantic segmentation.

[0015] Optionally, the filtering of the first feature point set to obtain a third feature point set according to the semantic segmentation information includes:

[0016] Dividing the first image into a plurality of regions according to the semantic segmentation information, the plurality of regions including motion type target regions and non-motion type target regions;

[0017] A third feature point set belonging to the non-motion type target area is obtained by screening from the first feature point set.

[0018] In the above solution, the first image is divided into multiple regions based on semantic segmentation information, and feature points belonging to non-motion type target regions are screened from the first feature point set to obtain a third feature point set.

[0019] Optionally, the non-motion type target area includes a repetitive texture target area and a static target area, wherein the repetitive texture target area is a pixel area corresponding to a non-motion type target with a repetitive texture in the scene, and the static target area is a pixel area other than the motion type target area and the repetitive texture target area;

[0020] Performing feature point matching on the third feature point set and the second feature point set to obtain matching feature point pairs includes:

[0021] The feature points belonging to the repeated texture target area and the feature points belonging to the stationary target area in the third feature point set are matched with the feature points in the second feature point set using different matching methods to obtain the matching pixel pairs.

[0022] In the above solution, the processing device subdivides the feature points of the non-motion target area into a repetitive texture target area and a static target area. This enables the processing device to use different matching methods for different feature point types to improve the efficiency of feature point matching.

[0023] Optionally, performing feature point matching on the third feature point set and the second feature point set to obtain matching feature point pairs includes:

[0024] The feature points of the third feature point set belonging to the repeated texture target area are matched with the feature points in the second feature point set in a cross matching manner to obtain matching feature point pairs.

[0025] In the above solution, for the feature points belonging to the repeated texture target area, a cross matching method is used to perform feature point matching, which can improve the matching accuracy of the feature points in the repeated texture target area.

[0026] Optionally, performing feature point matching on the third feature point set and the second feature point set to obtain matching feature point pairs includes:

[0027] The feature points of the third feature point set belonging to the stationary target area are matched with the feature points in the second feature point set using a nearest neighbor matching method based on similarity calculation to obtain matching feature point pairs.

[0028] In the above solution, for the feature points belonging to the stationary target area, the nearest neighbor matching method based on similarity calculation is used to perform feature point matching, which can reduce the time consumption of matching while ensuring the accuracy of feature point matching.

[0029] Optionally, the third feature point set belonging to the non-motion type target area is obtained by screening from the first feature point set, including:

[0030] According to the target area to which the feature points in the first feature point set belong, setting the matching weights of the feature points in the first feature point set, wherein the matching weights of the feature points in the first feature point set belonging to the motion type target area are the first matching weights, the matching weights of the feature points in the first feature point set belonging to the repetitive texture target area are the second matching weights, and the matching weights of the feature points in the first feature point set belonging to the stationary target area are the third matching weights;

[0031] A third feature point set is determined according to the matching weights of the feature points in the first feature point set, where the third feature point set is a set of feature points in the first feature point set whose matching weight is the second matching weight or the third matching weight.

[0032] In the above scheme, feature points belonging to the motion type target area with a matching weight of the first matching weight are screened out from the first feature point set, and feature point matching is performed based on feature points belonging to the non-motion type target area, which can effectively improve the matching accuracy of global image feature points.

[0033] Optionally, the first image and the second image are two images obtained by continuously shooting the same scene.

[0034] In a second aspect, an image processing apparatus includes:

[0035] an acquisition unit, configured to acquire a first image and a second image of the same scene;

[0036] a processing unit, configured to extract feature points from the first image and the second image respectively, to obtain a first feature point set of the first image and a second feature point set of the second image;

[0037] The processing unit is further configured to obtain semantic segmentation information of the first image, the semantic segmentation information including pixel identification information corresponding to multiple types of objects in the first image;

[0038] The processing unit is further configured to filter the first feature point set according to the semantic segmentation information to obtain a third feature point set, where the third feature point set includes feature points corresponding to non-moving objects in the first image;

[0039] The processing unit is further configured to perform feature point matching on the third feature point set and the second feature point set to obtain matching feature point pairs.

[0040] Optionally, the processing unit is specifically used to input the first image into a semantic segmentation intelligent model to obtain the semantic segmentation information output by the semantic segmentation intelligent model.

[0041] Optionally, the processing unit is specifically configured to:

[0042] Dividing the first image into a plurality of regions according to the semantic segmentation information, the plurality of regions including motion type target regions and non-motion type target regions;

[0043] A third feature point set belonging to the non-motion type target area is obtained by screening from the first feature point set.

[0044] Optionally, the non-motion type target area includes a repetitive texture target area and a static target area, wherein the repetitive texture target area is a pixel area corresponding to a non-motion type target with a repetitive texture in the scene, and the static target area is a pixel area other than the motion type target area and the repetitive texture target area;

[0045] The processing unit is specifically configured to match the feature points in the third feature point set belonging to the repeated texture target area and the feature points in the stationary target area with the feature points in the second feature point set using different matching methods to obtain the matched feature point pairs.

[0046] Optionally, the processing unit is specifically configured to perform feature point matching on the feature points of the third feature point set belonging to the repeated texture target area with the feature points in the second feature point set using a cross matching method to obtain matching feature point pairs.

[0047] Optionally, the processing unit is specifically configured to perform feature point matching on the feature points of the third feature point set belonging to the stationary target area with the feature points in the second feature point set using a nearest neighbor matching method based on similarity calculation to obtain matching feature point pairs.

[0048] Optionally, the processing unit is specifically configured to:

[0049] According to the target area to which the feature points in the first feature point set belong, setting the matching weights of the feature points in the first feature point set, wherein the matching weights of the feature points in the first feature point set belonging to the motion type target area are the first matching weights, the matching weights of the feature points in the first feature point set belonging to the repetitive texture target area are the second matching weights, and the matching weights of the feature points in the first feature point set belonging to the stationary target area are the third matching weights;

[0050] A third feature point set is determined according to the matching weights of the feature points in the first feature point set. The third feature point set is a set of feature points in the first feature point set whose matching weight is the second matching weight or the third matching weight.

[0051] Optionally, the first image and the second image are two images obtained by continuously shooting the same scene.

[0052] In a third aspect, a processing device is provided, comprising: a logic circuit and a communication interface, wherein the communication interface is used to obtain data to be processed and / or output processed data, and the logic circuit is used to obtain processed data from the data to be processed, so that the processing device executes the method in the first aspect and any possible implementation of the first aspect.

[0053] In a feasible design, the communication interface includes an input interface and an output interface.

[0054] Optionally, the processing device provided in the third aspect is a chip.

[0055] In a fourth aspect, a processing device is provided, comprising a processor and a memory. The processor is configured to read instructions stored in the memory and receive and transmit signals via a transceiver to execute the method of the first aspect and any possible implementation of the first aspect.

[0056] In a fifth aspect, a computer program product is provided, which includes: a computer program (also referred to as code, or instructions), which, when executed, enables a computer to execute the method in the above-mentioned first aspect and any possible implementation of the first aspect.

[0057] In a sixth aspect, a computer-readable medium is provided, which stores a computer program (also referred to as code, or instructions) which, when run on a computer, enables the computer to execute the method in the above-mentioned first aspect and any possible implementation of the first aspect.

[0058] The image processing method provided in the present application screens non-motion feature points in an image, performs feature point matching on the screened non-motion feature points, and obtains feature point matching pixel pairs of the two images. This can improve the accuracy of feature point matching in dynamic scenes and reduce processing delay and power consumption overhead. BRIEF DESCRIPTION OF THE DRAWINGS

[0059] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0060] Figure 1 This is a schematic diagram of an application scenario applicable to the image processing method provided in this application;

[0061] Figure 2 is a schematic flow chart of the image processing method provided by this application;

[0062] Figure 3 is a schematic diagram of the first image provided in this application;

[0063] Figure 4 It is a structural diagram of the image processing device provided by this application;

[0064] Figure 5 It is a structural diagram of the processing equipment provided by this application;

[0065] Figure 6 This is another structural diagram of the processing equipment provided by this application.

[0066] The above drawings illustrate specific embodiments of the present application, which will be described in more detail below. These drawings and the textual description are not intended to limit the scope of the present application in any way, but rather to illustrate the concepts of the present application to those skilled in the art by reference to specific embodiments. DETAILED DESCRIPTION

[0067] Exemplary embodiments will be described in detail herein, with examples illustrated in the accompanying drawings. In the following description, when referring to the drawings, identical numerals in different figures represent identical or similar elements, unless otherwise indicated. The embodiments described in the following exemplary embodiments are not intended to represent all embodiments consistent with the present application. Rather, they are merely examples of apparatus and methods consistent with certain aspects of the present application, as detailed in the appended claims.

[0068] Figure 1 This is a schematic diagram of an application scenario suitable for the image processing method provided by this application. Figure 1 As shown, Figure 1 As shown, the application scenario may include at least one image acquisition device, such as image acquisition device 101. For example, the image acquisition device may be a camera, a video recorder, etc. The image acquisition device may transmit the acquired image to the image processing device 102, and the image processing device 102 may perform image processing based on the acquired image. It should be noted that Figure 1 This is only an example of an application scenario, and in a specific implementation, the image acquisition device 101 and the image processing device 102 may be configured in different devices or in the same device, which is not limited in this application.

[0069] The image processing method provided by the present application will be described below with reference to the accompanying drawings. The embodiments described below may be combined with each other, and the same or similar concepts or processes may not be described in detail in some embodiments.

[0070] In order to improve the accuracy of feature point matching, the present application proposes to filter non-motion type feature points in the image, perform feature point matching on the filtered non-motion type feature points, and obtain feature point matching pixel pairs of the two images, which can improve the accuracy of feature point matching in dynamic scenes and reduce processing delay and power consumption overhead.

[0071] Figure 2 This is a schematic flow chart of the image processing method provided by this application. The method can be executed by a processing device, such as the processing device can be Figure 1 The image processing device 102 in the application scenario shown. The processing device can be a processing device, or the processing device is configured in a processing device, such as a chip. However, the present application is not limited thereto. The method includes but is not limited to the following steps:

[0072] S201: A processing device obtains a first image and a second image of the same scene.

[0073] The processing device can obtain the first image and the second image of the same scene from the image acquisition device. Alternatively, the processing device has an image acquisition function, and the processing device acquires the first image and the second image of the same scene.

[0074] Optionally, the first image and the second image are two images obtained by continuously photographing the same scene, or the time interval between the acquisition time of the first image and the acquisition time of the second image is less than or equal to a preset time length.

[0075] The processing device obtains matching pixel pairs in two images based on two images taken continuously or two images taken within a preset time period through the image processing method provided by the present application. The matching pixel pairs in the two images obtained by the processing device can be used to calculate the motion parameters of the image acquisition device (such as a camera, video recorder, etc.), thereby realizing motion structure recovery, etc. Alternatively, the matching pixel pairs in the two images obtained by the processing device can be used to realize High Dynamic Range Imaging (HDRI). It should be understood that the application scenarios of the present application may include but are not limited to this.

[0076] Specifically, the processing device acquires the first image and the second image and obtains grayscale data of pixels in the first image and the second image respectively.

[0077] S202: The processing device extracts feature points from the first image and the second image respectively to obtain a first feature point set of the first image and a second feature point set of the second image.

[0078] It should be noted that this application does not limit the order in which the processing device executes S202 and S203. The processing device may execute S202 first and then S203, or may execute S203 first and then S202, or may execute S202 and S203 simultaneously.

[0079] The processing device may first perform feature point detection, searching for stable pixels in the first and second images that do not change with movement, rotation, or illumination changes of the image acquisition device, namely, feature points, and obtain position information of the feature points in the images. The set of feature points in the first image is referred to as the first feature point set, and the feature points in the second image is referred to as the second feature point set. The processing device may employ, but is not limited to, an algorithm such as the Scale-Invariant Feature Transform (SIFT) algorithm or the SURF algorithm to extract feature points from the first and second images, respectively.

[0080] After obtaining the feature point sets from the first and second images, the processing device can obtain a feature descriptor for each feature point. A feature descriptor, also known as a feature vector, describes the feature point using the position information of the feature point and its surrounding pixels. This feature descriptor can eliminate changes in image scale and orientation caused by changes in viewing angle.

[0081] S203: The processing device obtains semantic segmentation information of the first image, where the semantic segmentation information includes pixel identification information corresponding to multiple types of objects in the first image.

[0082] The processing device pre-stores pixel identification information corresponding to various targets, such as the sky, a mountain, a river, a person, a cow, a dog, and a cat. The pixel identification information for different targets is different. The processing device can identify targets in the first image, assign an target type to each pixel, and identify the pixel identification information corresponding to each target at the pixel position corresponding to the target to obtain semantic segmentation information.

[0083] For example, the first image is Figure 3 The image shown includes a mountain 301, a car 302, and a tree 303. The processing device identifies the target in the first image, and the obtained semantic segmentation information includes pixel identification information corresponding to the mountain 301, the car 302, and the tree 303.

[0084] In an example, the identification information of the pixel point corresponding to mountain 301 may be 1, and the processing device in the semantic segmentation information identifies that the position corresponding to the pixel point belonging to mountain 301 in the first image is marked as 1; the identification information of the pixel point corresponding to car 302 may be 2, and the processing device in the semantic segmentation information identifies that the position corresponding to the pixel point belonging to car 302 in the first image is marked as 2; the identification information of the pixel point corresponding to tree 303 may be 3, and the processing device in the semantic segmentation information identifies that the position corresponding to the pixel point belonging to tree 303 in the first image is marked as 3.

[0085] In another example, the semantic segmentation information includes a mask image corresponding to each of the multiple targets, each mask image corresponds to one target, and the mask value of the pixel corresponding to the target in each mask image is 1, and the mask value of other pixels is 0. Figure 3 In the example shown, the semantic segmentation information includes a mask image of mountain 301. In this mask image, the mask value of the pixels of mountain 301 is 1, and the mask values ​​of other pixels are 0. The semantic segmentation information also includes a mask image of car 301. In this mask image of car 301, the mask value of the pixels of car 302 is 1, and the mask values ​​of other pixels are 0. The semantic segmentation information also includes a mask image of tree 303. In this mask image of tree 303, the mask value of the pixels of tree 303 is 1, and the mask values ​​of other pixels are 0.

[0086] It should be understood that the above are only two examples of semantic segmentation information. This application does not limit the specific form of the semantic segmentation information. The semantic segmentation information can identify the pixel points corresponding to different targets in the image, so that in the steps described below, feature pixel points can be screened for the pixel points corresponding to different targets based on the semantic segmentation information.

[0087] Optionally, the processing device inputs the first image into a semantic segmentation intelligent model to obtain semantic segmentation information output by the semantic segmentation intelligent model.

[0088] The processing device can utilize artificial intelligence technology and adopt an artificial intelligence model, namely the semantic segmentation intelligent model, to obtain semantic segmentation information based on the first image reasoning.

[0089] For example, in order to reduce the delay of reasoning semantic segmentation information, the semantic segmentation intelligent model can be implemented using a full convolutional network 32x upsampling (FCN-32s) model, but the present application is not limited to this.

[0090] S204: The processing device filters the first feature point set to obtain a third feature point set based on the semantic segmentation information. The third feature point set includes feature points corresponding to non-moving objects in the first image.

[0091] The processing device can determine the pixel area of ​​the motion type target and the pixel area of ​​the non-motion type target in the first image based on the semantic segmentation information. The motion type target is a target in a motion state in the image, which can be called the image foreground. For example, the motion type target is a target whose position in the scene may change over time. For example, the motion type target includes motion type targets such as people, animals, and cars. The non-motion type target is a target that is not in a motion state in the image. For example, the non-motion type target is a target whose position in the scene does not change over time. For example, the sky, mountains, rivers, trees, etc.

[0092] Optionally, the processing device can classify and mark the pixel points of the target in the first image as motion type target areas, repeated texture target areas and static target areas based on the semantic segmentation information. That is, the processing device can subdivide the non-motion type target into repeated texture target areas and static target areas, wherein the repeated texture target area is the pixel area corresponding to the non-motion type target with repeated texture in the scene, that is, the pixel area where the appearance texture in the image repeats or contains multiple similar textures. For example, the repeated texture target area can include pixel areas corresponding to targets such as large vegetation, leaves, building exteriors, river surfaces, and ground with repeated textures (such as large areas of brickwork). The static target area is the pixel area other than the motion type target area and the repeated texture area, and the static target area can also be called the background pixel area. However, the present application is not limited to this.

[0093] After the processing device determines the motion type area and the non-motion type area in the first image, the processing device filters the feature points belonging to the non-motion type area in the first feature point set to obtain a set of feature points belonging to the non-motion type area in the first feature point set, that is, the third feature point set.

[0094] According to the above-mentioned scheme proposed in this application, based on the semantic segmentation information of the image, the non-motion type area in the image can be obtained, thereby screening out the feature points in the non-motion type area for feature point matching between the two images. Screening out the feature points corresponding to the motion type targets can improve the accuracy of image feature point matching and the efficiency of feature point matching.

[0095] In an optional embodiment, the processing device may be configured with a motion type inference model, and after the first image is input into the motion type inference model, the motion type inference information in the first image obtained by the motion type inference model based on the first image is obtained. The processing device can determine the motion type target area and the non-motion type target area based on the motion type inference information. For example, the motion type inference information includes the pixel coordinate information of the motion type target area in the first image, and / or the pixel coordinate information of the non-motion type target area. Optionally, the pixel coordinate information included in the non-motion type target area can specifically distinguish the pixel coordinate information included in the repeated texture target area and the pixel coordinate information included in the static target area.

[0096] The processing device obtains the motion type inference information in the first image obtained by inferring the motion type inference model, and can filter feature points belonging to the non-motion type target area from the first feature point set based on the inference information to obtain a third feature point set.

[0097] In a specific implementation, the processing device can assign different matching weights to each pixel point in different target areas. When the processing device performs the matching operation of the feature points in the next step, it determines whether the target area to which the feature point belongs can be determined based on the matching weight, and determines whether to perform the matching operation on the feature point. For example, the matching weight of the feature points in the first feature point set belonging to the motion type target area is 0, and the matching weight of the feature points in the first feature point set belonging to the non-motion type target area is 1. Or if the non-motion target area includes a repetitive texture target area and a stationary target area, the matching weight of the feature points in the first feature point set belonging to the motion type target area is 0, the matching weight of the feature points in the first feature point set belonging to the repetitive texture target area is 0.5, and the matching weight of the feature points in the first feature point set belonging to the stationary target area is 1. Then the feature points with matching weights of 0.5 and 1 in the first feature point set are the third feature point set. When performing feature point matching, the processing device determines based on the matching weights that the matching operation is not performed on the feature points with matching weights of 0, and the feature point matching operation is performed on the feature points with matching weights of 0.5 and 1.

[0098] S205: The processing device performs feature point matching on the third feature point set and the second feature point set to obtain matching feature point pairs.

[0099] The processing device can calculate the similarity of the two feature points based on the feature descriptors of the two feature points. The similarity can be reflected by the distance metric between the feature descriptors of the two feature points. The smaller the distance metric, the smaller the difference between the feature points and the higher the similarity. For example, if the numerical value in the feature descriptor is a floating-point type numerical value, the Euclidean distance between the feature descriptors of the two feature points can be calculated to obtain the distance metric between the feature points. Alternatively, if the numerical value in the feature descriptor is a binary numerical value, the Hamming distance between the feature descriptors of the two feature points can be calculated to obtain the distance metric between the feature points. However, the present application is not limited to this. The processing device can calculate the similarity between the two feature points through a matching function. The matching function can be a nonlinear function or an artificial intelligence model. For example, the processing device can use the matching function to obtain the similarity of the two feature points, or the matching function can be a function that calculates the difference between the two feature points. The smaller the difference, the higher the matching degree.

[0100] The processing device can use a nearest neighbor matching method based on similarity calculation or a cross matching method to match the feature points in the third feature point set and the feature points in the second feature point set to obtain matching pixel pairs, each matching pixel pair including a feature point in the third feature point set and a feature point in the second feature point set.

[0101] The nearest neighbor matching method based on similarity calculation can be that the processing device searches and matches each feature point in the third feature point set within the corresponding semantic area of ​​the second feature point set according to the semantic segmentation information, and the feature point with the highest similarity is the matching feature point of the feature point in the third feature point set in the second feature point set.

[0102] A cross-matching method can involve the processing device first using feature points from the third feature point set as feature points to be matched, searching for matching points of the feature points in the second feature point set, and obtaining matching points for each feature point in the third feature point set, thereby obtaining a first set of candidate matching pixel pairs. The processing device then uses feature points from the second feature point set as feature points to be matched, searches for matching points of the feature points in the third feature point set, and obtains matching points for each feature point in the second feature point set, thereby obtaining a second set of candidate matching pixel pairs. The processing device then obtains the intersection of the first and second candidate matching pixel pairs, thereby obtaining a set of feature point matching pixel pairs for the first and second images. Specifically, matching pixel pairs in the feature point matching pixel pair set are matched when matching with feature points from the third feature point set as feature points to be matched, and are also matched when matching with feature points from the second feature point set as feature points to be matched. These matching pixel pairs belong to both the first and second candidate matching pixel pair sets. The cross-matching method can improve the accuracy of feature point matching.

[0103] Optionally, the non-motion type target area is specifically subdivided into a repetitive texture target area and a static target area, and the processing device can use a nearest neighbor matching method based on similarity calculation to determine matching points in the second feature point set for the feature points in the third feature point set belonging to the static target area. A cross matching method is used to determine matching points in the second feature point set for the feature points in the third feature point set belonging to the repetitive texture target area. For example, the processing device can determine the feature point matching method based on the matching weights of the feature points in the third feature point set, and for feature points with a matching weight of 1, a nearest neighbor matching method based on similarity calculation is used to determine matching pixel pairs. For feature points with a matching weight of 0.5, a cross matching method is used to determine matching pixel pairs. However, the present application is not limited to this.

[0104] The processing device uses different methods to perform feature point matching based on the type of feature point. For example, for feature points in target areas with repeated textures that are similar but lack uniqueness, a cross-matching method is used to improve matching accuracy. For feature points in stationary target areas that are sufficiently unique, a nearest neighbor matching method based on similarity calculation is used to reduce matching time. Targeted selection of matching methods can improve both feature point matching accuracy and efficiency.

[0105] It should be noted that the above description uses the nearest neighbor matching method and the cross matching method based on similarity calculation as examples to illustrate the processing method of feature point matching, but the present application is not limited to this. In specific implementations, other feature point matching methods can also be used to perform feature point matching on non-motion type feature points obtained based on semantic segmentation information.

[0106] According to the image processing method provided in the present application, by screening non-motion type feature points in the image, feature point matching is performed on the screened non-motion type feature points to obtain feature point matching pixel pairs of the two images, which can improve the accuracy of feature point matching in dynamic scenes and reduce processing delay and power consumption overhead.

[0107] The image processing method provided by the present application is described above in conjunction with the accompanying drawings. The processing device and processing equipment provided by the present application are described below.

[0108] Figure 4 4 is a schematic diagram of the structure of the image processing device provided by this application. The processing device can be implemented as a processing device itself or configured in a processing device through software, hardware, or a combination of both. The processing device includes an acquisition unit 401 and a processing unit 402.

[0109] An acquisition unit 401 is configured to acquire a first image and a second image of the same scene;

[0110] A processing unit 402 is configured to extract feature points from the first image and the second image respectively to obtain a first feature point set of the first image and a second feature point set of the second image;

[0111] The processing unit 402 is further configured to obtain semantic segmentation information of the first image, where the semantic segmentation information includes pixel identification information corresponding to multiple types of objects in the first image;

[0112] The processing unit 402 is further configured to filter the first feature point set according to the semantic segmentation information to obtain a third feature point set, where the third feature point set includes feature points corresponding to non-moving objects in the first image.

[0113] The processing unit 402 is further configured to perform feature point matching on the third feature point set and the second feature point set to obtain matching feature point pairs.

[0114] Optionally, the processing unit 402 is specifically configured to input the first image into a semantic segmentation intelligent model to obtain the semantic segmentation information output by the semantic segmentation intelligent model.

[0115] Optionally, the processing unit 402 is specifically configured to:

[0116] Dividing the first image into a plurality of regions according to the semantic segmentation information, the plurality of regions including motion type target regions and non-motion type target regions;

[0117] A third feature point set belonging to the non-motion type target area is obtained by screening from the first feature point set.

[0118] Optionally, the non-motion type target area includes a repetitive texture target area and a static target area, wherein the repetitive texture target area is a pixel area corresponding to a non-motion type target with a repetitive texture in the scene, and the static target area is a pixel area other than the motion type target area and the repetitive texture target area;

[0119] The processing unit 402 is specifically configured to match the feature points in the third feature point set belonging to the repeated texture target area and the feature points in the stationary target area with the feature points in the second feature point set using different matching methods to obtain the matched feature point pairs.

[0120] Optionally, the processing unit 402 is specifically configured to perform feature point matching on the feature points of the third feature point set belonging to the repeated texture target area with the feature points in the second feature point set using a cross matching method to obtain matching feature point pairs.

[0121] Optionally, the processing unit 402 is specifically configured to perform feature point matching on the feature points of the third feature point set belonging to the stationary target area with the feature points in the second feature point set using a nearest neighbor matching method based on similarity calculation to obtain matching feature point pairs.

[0122] Optionally, the processing unit 402 is specifically configured to:

[0123] According to the target area to which the feature points in the first feature point set belong, setting the matching weights of the feature points in the first feature point set, wherein the matching weights of the feature points in the first feature point set belonging to the motion type target area are the first matching weights, the matching weights of the feature points in the first feature point set belonging to the repetitive texture target area are the second matching weights, and the matching weights of the feature points in the first feature point set belonging to the stationary target area are the third matching weights;

[0124] A third feature point set is determined according to the matching weights of the feature points in the first feature point set. The third feature point set is a set of feature points in the first feature point set whose matching weight is the second matching weight or the third matching weight.

[0125] Optionally, the first image and the second image are two images obtained by continuously shooting the same scene.

[0126] The present application also provides a processing device. The processing device includes: a logic circuit and a communication interface, wherein the communication interface is used to obtain data to be processed and / or output processed data, and the logic circuit is used to process the data to be processed to obtain processed data, so that the processing device executes the image processing method of the above method embodiment. In one feasible design, the communication interface includes an input interface and an output interface.

[0127] Optionally, the processing device provided in the embodiment of the present application may be a chip.

[0128] Optionally, the processing device may be a central processing unit (CPU), other general-purpose processors, digital signal processors (DSP), or application-specific integrated circuits (ASICs). A general-purpose processor may be a microprocessor or any conventional processor. The steps of the method disclosed in this application may be directly executed by a hardware processor or by a combination of hardware and software modules in the processor.

[0129] Figure 5 Schematic diagram of the structure of the processing device 500 provided in the embodiment of the present application. The processing device 500 can be applied to Figure 1 In the application scenario shown, the functions of the processing device in the above-mentioned method embodiment are performed. As shown in the figure, the processing device 500 includes a transceiver 501, a processor 502, and a memory 503. The processor 502, the transceiver 501, and the memory 503 can communicate with each other through an internal connection path to transmit image processing data and / or control signals. The memory 503 is used to store a computer program, and the processor 502 is used to execute the computer program in the memory to control the transceiver 501 to obtain image data from the image acquisition device. In specific implementation, the memory can also be integrated into the processor or independent of the processor.

[0130] Figure 6: is a structural diagram of a processing device 600 provided in an embodiment of the present application. The processing device 600 performs the functions of the processing device in the above-mentioned method embodiment. As shown in the figure, the processing device 600 includes an image acquisition device 601, a processor 602 and a memory 603. Among them, the processor 602, the image acquisition device 601 and the memory 603 can communicate with each other through an internal connection path to transmit image processing data and / or control signals. The memory 603 is used to store computer programs, and the processor 602 is used to execute the computer program in the memory to control the image acquisition device 601 to obtain image data. Exemplarily, the image acquisition device 601 is a camera. In specific implementation, the memory can also be integrated into the processor or independent of the processor.

[0131] This embodiment further provides a readable storage medium, in which execution instructions are stored. When at least one processor of an electronic device executes the execution instruction, when the computer execution instruction is executed by the processor, the image processing method in the above embodiment is implemented.

[0132] This embodiment further provides a computer program product, including execution instructions stored in a readable storage medium. At least one processor of an electronic device can read the execution instructions from the readable storage medium, and the at least one processor can execute the execution instructions to cause the electronic device to implement the image processing methods provided in the various embodiments described above.

[0133] In the several embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are merely schematic. For example, the division of the modules is merely a logical function division. In actual implementation, there may be other division methods, such as multiple modules or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or modules, which can be electrical, mechanical or other forms.

[0134] The modules described as separate components may or may not be physically separate, and the components shown as modules may or may not be physical modules, that is, they may be located in one place or distributed across multiple network modules. Some or all of the modules may be selected to achieve the purpose of the present embodiment according to actual needs.

[0135] In addition, the functional modules in the various embodiments of the present application may be integrated into a single processing module, or each module may exist physically separately, or two or more modules may be integrated into a single module. The aforementioned integrated modules may be implemented in the form of hardware or in the form of hardware plus software functional modules.

[0136] The above-mentioned integrated module implemented in the form of a software function module can be stored in a computer-readable storage medium. The above-mentioned software function module is stored in a storage medium and includes a number of instructions for causing a computer device (which can be a personal computer, server, or network device, etc.) or a processor (English: processor) to execute some steps of the method of each embodiment of the present application. The aforementioned storage medium includes: a USB flash drive, a mobile hard disk, a read-only memory (English: Read-Only Memory, abbreviated: ROM), a random access memory (English: Random Access Memory, abbreviated: RAM), a magnetic disk or an optical disk, and other media that can store program code.

[0137] Those skilled in the art will readily appreciate other embodiments of the present application after considering the specification and practicing the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of the present application that follow the general principles of the present application and include common knowledge or customary techniques in the art not disclosed herein. The description and examples are to be considered as exemplary only, and the true scope and spirit of the present application are indicated by the following claims.

[0138] It should be understood that the present application is not limited to the exact structure described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present application is limited only by the appended claims.

Claims

1. An image processing method, characterized in that: include: Acquire a first image and a second image of the same scene; Extracting feature points from the first image and the second image respectively to obtain a first feature point set of the first image and a second feature point set of the second image; Acquire semantic segmentation information of the first image, where the semantic segmentation information includes pixel identification information corresponding to multiple types of objects in the first image; Filtering the first feature point set to obtain a third feature point set according to the semantic segmentation information, where the third feature point set includes feature points corresponding to non-moving objects in the first image; performing feature point matching on the third feature point set and the second feature point set to obtain matching feature point pairs; The step of filtering the first feature point set according to the semantic segmentation information to obtain a third feature point set includes: Dividing the first image into a plurality of regions according to the semantic segmentation information, the plurality of regions including motion type target regions and non-motion type target regions; Filtering the first feature point set to obtain a third feature point set belonging to the non-motion type target area; The non-motion type target area includes a repetitive texture target area and a static target area, wherein the repetitive texture target area is a pixel area corresponding to a non-motion type target with a repetitive texture in the scene, and the static target area is a pixel area other than the motion type target area and the repetitive texture target area; The performing feature point matching on the third feature point set and the second feature point set to obtain matching feature point pairs includes: The feature points belonging to the repeated texture target area and the feature points belonging to the stationary target area in the third feature point set are matched with the feature points in the second feature point set using different matching methods to obtain the matched feature point pairs.

2. The method according to claim 1, characterized in that The obtaining of semantic segmentation information of the first image includes: The first image is input into a semantic segmentation intelligent model to obtain the semantic segmentation information output by the semantic segmentation intelligent model.

3. The method according to claim 1, characterized in that The performing feature point matching on the third feature point set and the second feature point set to obtain matching feature point pairs includes: The feature points of the third feature point set belonging to the repeated texture target area are matched with the feature points in the second feature point set in a cross matching manner to obtain matching feature point pairs.

4. The method according to claim 1, wherein The performing feature point matching on the third feature point set and the second feature point set to obtain matching feature point pairs includes: For the feature points of the third feature point set belonging to the stationary target area, feature point matching is performed with the feature points in the second feature point set using a nearest neighbor matching method based on similarity calculation to obtain matching feature point pairs.

5. The method according to any one of claims 1 to 4, characterized in that The step of filtering the first feature point set to obtain a third feature point set belonging to the non-motion type target area includes: Setting matching weights of feature points in the first feature point set according to the target areas to which the feature points in the first feature point set belong, wherein the matching weights of feature points in the first feature point set belonging to the motion type target area are the first matching weights, the matching weights of feature points in the first feature point set belonging to the repetitive texture target area are the second matching weights, and the matching weights of feature points in the first feature point set belonging to the stationary target area are the third matching weights; A third feature point set is determined according to the matching weights of the feature points in the first feature point set, where the third feature point set is a set of feature points in the first feature point set whose matching weight is the second matching weight or the third matching weight.

6. The method according to any one of claims 1 to 4, characterized in that The first image and the second image are two images obtained by continuously shooting the same scene.

7. An image processing device, characterized in that: include: an acquisition unit, configured to acquire a first image and a second image of the same scene; a processing unit, configured to extract feature points from the first image and the second image respectively, to obtain a first feature point set of the first image and a second feature point set of the second image; The processing unit is further configured to obtain semantic segmentation information of the first image, the semantic segmentation information including pixel identification information corresponding to multiple types of objects in the first image; The processing unit is further configured to filter the first feature point set according to the semantic segmentation information to obtain a third feature point set, wherein the third feature point set includes feature points corresponding to non-moving objects in the first image; The processing unit is further configured to perform feature point matching on the third feature point set and the second feature point set to obtain matching feature point pairs; The processing unit is specifically configured to: Dividing the first image into a plurality of regions according to the semantic segmentation information, the plurality of regions including motion type target regions and non-motion type target regions; Filtering the first feature point set to obtain a third feature point set belonging to the non-motion type target area; The non-motion type target area includes a repetitive texture target area and a static target area, wherein the repetitive texture target area is a pixel area corresponding to a non-motion type target with a repetitive texture in the scene, and the static target area is a pixel area other than the motion type target area and the repetitive texture target area; The processing unit is specifically configured to match the feature points in the third feature point set belonging to the repeated texture target area and the feature points in the stationary target area with the feature points in the second feature point set using different matching methods to obtain the matched feature point pairs.

8. The device according to claim 7, characterized in that The processing unit is specifically used to input the first image into the semantic segmentation intelligent model to obtain the semantic segmentation information output by the semantic segmentation intelligent model.

9. The device according to claim 7, characterized in that The processing unit is specifically configured to perform feature point matching on the feature points of the third feature point set belonging to the repeated texture target area with the feature points in the second feature point set by using a cross matching method to obtain matching feature point pairs.

10. The device according to claim 7, characterized in that The processing unit is specifically configured to perform feature point matching on feature points in the third feature point set belonging to the stationary target area with feature points in the second feature point set using a nearest neighbor matching method based on similarity calculation to obtain matching feature point pairs.

11. The device according to any one of claims 7 to 10, characterized in that The processing unit is specifically configured to: Setting matching weights of feature points in the first feature point set according to the target areas to which the feature points in the first feature point set belong, wherein the matching weights of feature points in the first feature point set belonging to the motion type target area are the first matching weights, the matching weights of feature points in the first feature point set belonging to the repetitive texture target area are the second matching weights, and the matching weights of feature points in the first feature point set belonging to the stationary target area are the third matching weights; A third feature point set is determined according to the matching weights of the feature points in the first feature point set, where the third feature point set is a set of feature points in the first feature point set whose matching weights are the second matching weight and the third matching weight.

12. The device according to any one of claims 7 to 10, characterized in that The first image and the second image are two images obtained by continuously shooting the same scene.

13. A processing device, characterized in that include: at least one processor and memory; The memory stores computer-executable instructions; The at least one processor executes the computer-executable instructions stored in the memory, so that the processing device performs the method according to any one of claims 1 to 6.

14. A chip, characterized in that: including at least one logic circuit and an input and output interface; The logic circuit is used to control the input and output interface and execute the method according to any one of claims 1 to 6.

15. A computer-readable storage medium, characterized in that The computer-readable storage medium stores computer-executable instructions, which are used to implement the method according to any one of claims 1 to 6 when executed by a processor.

16. A computer program product, characterized in that The method comprises instructions, which, when executed on a computer, cause the computer to execute the method according to any one of claims 1 to 6.

Images