Image processing method, electronic equipment and computer storage medium

Abstract

The invention discloses an image processing method, electronic equipment and a computer storage medium, and the method comprises the steps: obtaining a first image which is a two-dimensional image; a second image and a third image are determined according to the first image, the second image is a two-dimensional image matched with the first image in at least one two-dimensional image, and the third image is a three-dimensional image registered with the second image; determining a matched second pixel point from the second image according to the first pixel point in the first image; and determining a registered third pixel point from the third image according to the second pixel point. The method can improve the accuracy of control point extraction.

Description

Technical Field

[0001] This application relates to the field of image processing technology, and more particularly to an image processing method, electronic device, and computer storage medium. Background Technology

[0002] With the rapid development of video surveillance technology, vertical industries such as manufacturing, healthcare, and mining have deployed a large number of monocular cameras to meet daily monitoring needs. In target scenarios such as steel mills and smart industrial parks, over 80% of the area has already achieved video coverage. Therefore, fully utilizing existing camera resources to locate pedestrians, vehicles, and other moving objects is of great significance for improving the security management level of enterprises and industrial parks.

[0003] In existing technologies, when performing high-precision localization of objects such as pedestrians and vehicles based on images from a monocular camera, visual positioning control points (hereinafter referred to as control points) need to be selected based on the features of the video image and the 3D point cloud image. However, in some scenarios (such as those with weak textures or difficulty in separating the foreground and background), the accuracy of registration between the video image and the 3D point cloud is low, and the mismatch rate is high, affecting the accuracy of control point extraction and thus the accuracy of visual localization. Therefore, improving the accuracy of control point extraction is an urgent problem to be solved. Summary of the Invention

[0004] This application provides an image processing method, electronic device, and computer storage medium that can improve the accuracy of control point extraction.

[0005] Firstly, an image processing method is provided. This method can be executed by an electronic device, by a module applied in the electronic device (e.g., a processor, chip, or chip system), or by a logic node, logic module, or software capable of implementing all or part of the functions of the electronic device. The method includes:

[0006] Acquire a first image, which is a two-dimensional image; determine a second image and a third image based on the first image, where the second image is a two-dimensional image that matches the first image from at least one two-dimensional image, and the third image is a three-dimensional image registered with the second image; determine a matching second pixel from the second image based on a first pixel in the first image; and determine a registered third pixel from the third image based on the second pixel.

[0007] In image registration applications, one approach is to directly perform cross-modal matching between the visual features of a 2D image (e.g., the first image) and the pixels (or point cloud data) of a 3D image (e.g., the third image). However, this cross-modal matching method often results in a high mismatch rate. This is because the first and third images may differ significantly due to factors such as dimension, viewpoint, illumination, and scale variations. Directly matching these significantly different feature points or pixels increases the difficulty of matching and the mismatch rate. To address this, this application proposes an image processing method that performs same-dimensional matching (also known as same-dimensional registration) between a pixel (e.g., the first pixel) in a 2D image (e.g., the first image) and its corresponding pixel (e.g., the second pixel) in another 2D image (e.g., the second image). Since both images are two-dimensional, matching is relatively easy and accurate. Furthermore, because the second and third images are already registered, each pixel (or visual feature) in the second image has a corresponding spatial location in the third image. This allows the registration of the third pixel to be determined from the second pixel in the third image. Also, since the first and second images are matched (or registered), and the second and third images are registered, registration between the first and third images can be indirectly achieved. This method of indirect registration using an intermediate image (such as the second image) can more accurately determine the correspondence between a two-dimensional image (such as the first image) and a three-dimensional image (such as the third image), which is beneficial for the accuracy of subsequent control point extraction.

[0008] In one optional implementation, the first pixel is a visual feature point corresponding to a pixel in the first image, and the second pixel is a visual feature point corresponding to a pixel in the second image.

[0009] In some scenarios, extracting visual feature points from images can quickly achieve matching between images (e.g., the first image and the second image).

[0010] In one possible implementation, the aforementioned visual feature points are at least one of scale-invariant feature transform (SIFT) feature points, acceleration-stabilized feature transform (SURF) feature points, or oriented rotational feature points (ORB) feature points.

[0011] In different application scenarios, different feature processing algorithms (such as SURF, SIFT or ORB algorithms) can be used to extract visual features from the pixels of an image to ensure the accuracy of image matching using different visual feature points in different application scenarios.

[0012] In one alternative implementation, determining the registered third pixel from the third image based on the second pixel includes: determining the third pixel based on the second pixel and a coordinate transformation model.

[0013] In some scenarios, registering (or matching) the second and third pixels using a coordinate transformation model offers several advantages: 1) The coordinate transformation model aligns the second and third pixels within the same coordinate system, enabling data fusion. This cross-dimensional image information integration provides more comprehensive and accurate data support for subsequent control point extraction. 2) The coordinate transformation model accurately registers the second and third pixels, reducing registration errors caused by inconsistencies in coordinate systems, thus improving the accuracy and precision of subsequent control point extraction. 3) Since the coordinate transformation model is a technique based on mathematical algorithms and computational optimization, it enables efficient data processing and transformation. Therefore, registering the second and third pixels using this model reduces computational load and data processing time, improving the overall efficiency of image processing.

[0014] In one possible implementation, the first acquisition module and the second acquisition module satisfy the following requirements during the image acquisition process: synchronization and position fixation, wherein the first acquisition module is used to acquire the second image, the second acquisition module is used to acquire the third image, and the coordinate transformation model is related to synchronization and position fixation.

[0015] In some scenarios, synchronicity refers to the second and third images being acquired simultaneously (or almost simultaneously), meaning the two images capture the scene at the same or similar times. Fixed position refers to the second and third images being acquired from the same angle (viewpoint) or position, meaning they capture the scene from the same viewpoint. Because the two images are acquired at the same time and angle, their coordinate transformation model is fixed and does not need to be adjusted with changes in time and angle, thus improving the efficiency and accuracy of registration (or alignment) between the two images. Specifically, the advantages of this synchronous, same-angle image acquisition method are as follows: 1) Since the second and third images are acquired synchronously at the same angle, each pixel in the second and third images can be accurately matched; this accurate correspondence makes the acquired measurement results more accurate, which is beneficial for improving the accuracy of subsequent control point extraction. 2) Since the second and third images are acquired from the same angle, they can provide information about different dimensions of the same scene (such as texture information or depth information), thereby complementing each other and improving the situation of insufficient information in a single view caused by occlusion and other problems; 3) Compared with asynchronous or different angle acquisition methods, synchronous acquisition of images at the same angle does not require complex scene matching or angle transformation during preprocessing, which helps to reduce the amount of computation and improve the overall efficiency of image processing.

[0016] In an alternative implementation, the method further includes: determining the coordinates of the first control point based on the third pixel.

[0017] In one optional implementation, determining the coordinates of the first control point based on the third pixel includes: clustering the third pixel using a clustering algorithm to determine the first control point, wherein the first control point includes the first pixel; and determining the coordinates of the first control point based on the coordinates of the third pixel.

[0018] In some embodiments, users can automatically select appropriate control points (e.g., the first control point) through clustering algorithms according to the needs of visual positioning, without the need for manual selection, which is efficient and accurate.

[0019] In one alternative implementation, the second image is a two-dimensional image that matches the first image from at least one two-dimensional image, including: the second image is the two-dimensional image with the highest similarity to the first image from at least one two-dimensional image.

[0020] Since the second image is registered with the third image, determining the second image with the highest similarity to the first image is beneficial to improving the accuracy of matching each pixel in the first image with the corresponding pixel in the third image.

[0021] In one alternative implementation, acquiring the first image includes: acquiring the first image from the video.

[0022] In a second aspect, an image processing apparatus is provided, the apparatus comprising an acquisition module and a processing module;

[0023] The acquisition module is used to acquire the first image, which is a two-dimensional image;

[0024] The processing module is configured to determine a second image and a third image based on a first image, wherein the second image is a two-dimensional image that matches the first image from at least one two-dimensional image, and the third image is a three-dimensional image registered with the second image; it is also configured to determine a matching second pixel point from the second image based on a first pixel point in the first image; and it is also configured to determine a registered third pixel point from the third image based on the second pixel point.

[0025] In one optional implementation, the first pixel is a visual feature point corresponding to a pixel in the first image, and the second pixel is a visual feature point corresponding to a pixel in the second image.

[0026] In one alternative implementation, the visual feature points are one of the following: scale-invariant feature transform (SIFT) feature points, acceleration-stabilized feature transform (SURF) feature points, or oriented rotational feature points (ORB).

[0027] In one alternative implementation, the processing module is specifically used to determine the third pixel based on the second pixel and the coordinate transformation model.

[0028] In one optional implementation, the above-mentioned device further includes a first acquisition module and a second acquisition module, wherein the first acquisition module and the second acquisition module satisfy the following requirements during the image acquisition process: synchronization and position fixation, wherein the first acquisition module is used to acquire a second image, the second acquisition module is used to acquire a third image, and the coordinate transformation model is related to synchronization and position fixation.

[0029] In one alternative implementation, the processing module is further configured to: determine the coordinates of the first control point based on the third pixel.

[0030] In one alternative implementation, the processing module is further configured to: cluster the third pixel using a clustering algorithm to determine the first control point, the first control point including the first pixel; and determine the coordinates of the first control point based on the coordinates of the third pixel.

[0031] In one alternative implementation, the second image is a two-dimensional image that matches the first image from at least one two-dimensional image, including: the second image is the two-dimensional image with the highest similarity to the first image from at least one two-dimensional image.

[0032] In one alternative implementation, the acquisition module is specifically used to: acquire a first image from the video.

[0033] Thirdly, an electronic device is provided, comprising a processor and a memory, the memory for storing a computer program, and the processor for calling and running the computer program from the memory, causing the electronic device to perform any of the methods of the first aspect.

[0034] Fourthly, a computer-readable storage medium is provided that stores a computer program, which, when executed by a processor, causes the processor to perform any of the methods in the first aspect.

[0035] Fifthly, a computer program product is provided, comprising: computer program code, which, when executed by an electronic device, causes the electronic device to perform any of the methods in the first aspect.

[0036] In a sixth aspect, a chip system is provided, the chip system including a memory and a processor, the processor being configured to execute a computer program stored in the memory to implement the method of any of the first aspects.

[0037] The beneficial effects of the technical solutions in the second, third, fourth, fifth and sixth aspects of this application are the same as those in the first aspect, and will not be repeated here. Attached Figure Description

[0038] Figure 1 This is a schematic diagram of the structure of an electronic device 100 provided in an embodiment of this application;

[0039] Figure 2 A schematic diagram of the software structure of an electronic device 100 provided in an embodiment of this application;

[0040] Figure 3 A schematic diagram of a software architecture for an electronic device applicable to this application, provided as an embodiment of this application;

[0041] Figure 4 A schematic diagram of an application system architecture applicable to this application is provided for an embodiment of this application;

[0042] Figures 5A-5B The flowcharts of two image processing methods are shown;

[0043] Figure 6 A schematic flowchart of an image processing method 600 provided in an embodiment of this application;

[0044] Figure 7 This is a schematic diagram of an application scenario provided by an embodiment of this application;

[0045] Figure 8 A schematic diagram illustrating an image editing application scenario provided in an embodiment of this application;

[0046] Figure 9 A schematic diagram illustrating yet another image editing application scenario provided in this application embodiment;

[0047] Figure 10 A schematic diagram illustrating another image editing application scenario provided in this application embodiment;

[0048] Figure 11 A schematic diagram illustrating another image editing application scenario provided in this application embodiment;

[0049] Figure 12 A schematic diagram illustrating yet another image editing application scenario provided in this application embodiment;

[0050] Figure 13 A schematic diagram illustrating yet another image editing application scenario provided in this application embodiment;

[0051] Figure 14 A schematic diagram illustrating another image editing application scenario provided in this application embodiment;

[0052] Figure 15 A schematic diagram illustrating another image editing application scenario provided in this application embodiment;

[0053] Figure 16 A schematic diagram illustrating yet another image editing application scenario provided in this application embodiment;

[0054] Figure 17 A schematic diagram illustrating another image editing application scenario provided in this application embodiment;

[0055] Figure 18 This is a schematic diagram of the structure of an image processing apparatus 1800 provided in an embodiment of this application;

[0056] Figure 19 This is a schematic diagram of the structure of an electronic device 1900 provided in an embodiment of this application. Detailed Implementation

[0057] To facilitate a clear description of the technical solutions in the embodiments of this application, the terms "first" and "second" are used in the embodiments of this application to distinguish identical or similar items with essentially the same function and purpose. For example, "first terminal device" and "second terminal device" are merely used to distinguish different terminal devices and do not limit their order. Those skilled in the art will understand that the terms "first" and "second" do not limit the quantity or execution order, and that "first" and "second" do not necessarily imply that they are different.

[0058] It should be noted that, in this application, the terms "exemplary" or "for example" are used to indicate that something is being described as an example, illustration, or illustration. Any embodiment or design described as "exemplary" or "for example" in this application should not be construed as being more preferred or advantageous than other embodiments or design solutions. Specifically, the use of terms such as "exemplary" or "for example" is intended to present the relevant concepts in a concrete manner.

[0059] In this application, "at least one" means one or more, and "more than one" means two or more. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can mean: A alone, A and B simultaneously, or B alone, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one of a, b, or c can mean: a, b, c, ab, ac, bc, or abc, where a, b, and c can be single or multiple.

[0060] To facilitate understanding of this application, some of the technical terms involved in this application are explained below.

[0061] (1) Scale-invariant feature transform (SIFT) algorithm

[0062] SIFT is a machine vision algorithm used to detect and describe local feature points in images. It finds extreme points in spatial scales and extracts their position, scale, and rotation invariants. SIFT can preserve local feature points even when images are scaled or rotated. Furthermore, precise image registration can be achieved by matching SIFT feature points in different images.

[0063] (2) Oriented FAST and rotated BRIEF (ORB) algorithm

[0064] The ORB algorithm is a feature point extraction algorithm widely used in computer vision and image processing. It can be used to quickly create feature vectors from key points in an image, and these feature vectors can be used to identify objects in the image. The ORB algorithm combines the oriented features from accelerated segmenttest (oFAST) algorithm and the rotated binary robust independent elementary features (rBRIEF) algorithm to form the oriented corner detection algorithm (oriented FAST) and the rotated descriptor (rotated BRIEF) algorithm. In the ORB algorithm, the oFAST algorithm is first used to extract feature points in the image, and an orientation is defined for each feature point to achieve rotation invariance. Then, the rBRIEF algorithm is used to describe each feature point, generating a binary string as the descriptor of that feature point.

[0065] (3) Speed-up robust features (SURF)

[0066] The SURF algorithm is a robust local feature point detection and description algorithm; the SURF algorithm consists of three main parts: local feature point extraction, feature point description, and feature point matching.

[0067] (4) Image control points

[0068] Image control points, also known as visual positioning control points, visual control points, or control points, are reference points selected on an image to establish geometric transformation functions. For example, they are reference points selected on the image to establish geometric transformation functions in geometric transformations such as image geometric correction, projection transformation, and image registration.

[0069] (5) 3D point cloud image

[0070] A 3D point cloud image (or point cloud image or point cloud image), also known as 3D point cloud data, refers to a collection of points in a 3D coordinate system. Each point can be represented by a 3D coordinate, which can carry information about the point's attributes, such as color, reflectivity, and intensity. Typically, point cloud images (i.e., point cloud data) can be acquired using devices such as laser scanners (also known as LiDAR), cameras, and 3D scanners for applications such as 3D modeling, scene reconstruction, virtual reality, and augmented reality. It's important to note that point cloud images can visually represent the shape, surface, and texture of objects in space. Point cloud image processing methods include, but are not limited to, filtering, registration, segmentation, reconstruction, recognition, and classification.

[0071] (6) K-Nearest Neighbors (KNN) Algorithm

[0072] The KNN algorithm is a basic classification and regression method that classifies data by measuring the distance between different feature values. The three basic elements of K-nearest neighbors are: the choice of the k value, the distance metric, and the classification decision rule.

[0073] (7) Coordinate Transformation Model

[0074] Coordinate transformation models are typically used to establish the relationship between source coordinates and target coordinates. Different coordinate transformation models are used to achieve the transformation between source coordinates and target coordinates for different types of coordinates.

[0075] (8) Line segment detector (LSD)

[0076] The LSD algorithm is a line segment detection algorithm that accurately detects the position and direction of line segments in an image, exhibiting high precision and robustness. This LSD algorithm is based on the detection and analysis of image edges, for example, extracting line segments using the direction and intensity information of edge pixels. Its main idea is to construct a global branch and a simulated partial branch graph on the edge pixels, and extract line segments based on this graph. It should be noted that the LSD algorithm mainly includes the following steps: edge detection, edge pixel sorting, global branching, simulated partial branching, and non-maximum suppression. Edge detection refers to extracting the edges of the image using an edge detection algorithm; edge pixel sorting refers to sorting edge pixels in descending order of intensity; global branching refers to constructing a global branch graph on the edge pixels, starting from the edge pixel with the highest intensity, calculating the direction difference between its adjacent edge pixels, and constructing a global branch graph based on the direction difference; simulated partial branching refers to simulating a partial branch graph based on the global branch graph, starting from the strongest global branch, calculating a simulated partial branch graph based on the direction and intensity information of the edge pixels; non-maximum suppression: performing non-maximum suppression based on the direction and intensity information of the edge pixels to obtain the final line segment.

[0077] The following section, using practical application scenarios, outlines the technical problems that this application aims to solve.

[0078] In some scenarios, when registering point cloud images with images (or optical images), the LSD algorithm can be used to extract line features from the texture image, and the vanishing point detection algorithm can be used to extract image vanishing points. For the extraction of line features in laser point clouds, an adaptive segmentation method is used to extract line features, and the extracted line features are clustered based on a Gaussian mixture model clustering algorithm. The image line features and point cloud line features extracted in the above steps are then registered. Since the accuracy of this method depends on the number of line features in the scene, the mismatch rate is high in scenes with weak texture and high repetition, which in turn affects the accuracy of control point extraction.

[0079] In some scenarios, the registration of point cloud images with images (or optical images) can be achieved based on semantic segmentation. For example, a depth image is first generated using the elevation information of the point cloud image. In the depth image, the gray value of each pixel represents the distance of a point in the scene from the camera, reflecting the geometry and height of the ground surface. Then, a deep learning algorithm is used to segment and extract building surfaces from the depth image of the laser point cloud and the optical image, constructing the minimum bounding rectangle of the building outlines in the two types of images respectively. Finally, based on the aspect ratio constraint of the minimum bounding rectangle, the center point of the corresponding rectangle is found using the principle of similar triangles. The obtained corresponding points are substituted into the mapping transformation model, and the transformation model parameters are obtained through the least squares method, thereby achieving the registration of the point cloud image and the optical image. Since the accuracy of this method depends on the minimum bounding rectangle of the building outlines in the scene and has high requirements for semantic perception accuracy, the semantic segmentation effect is poor and the mismatch rate is high in some complex application scenarios (such as scenarios where the foreground and background are difficult to separate in the image), which in turn affects the accuracy of control point extraction.

[0080] The examples from the two scenarios above reveal that both line feature extraction and semantic segmentation scenarios may suffer from low registration accuracy and high misregistration rate between point cloud images and 2D images, thus affecting the accuracy of control point extraction. To address this, this application proposes an image processing method that can improve the accuracy of control point extraction and has wide applicability.

[0081] It should be noted that in this image processing method, the electronic device uses the second image as a bridge to achieve high-precision matching between video image feature points (e.g., feature points of the first image) and point cloud images (e.g., the third image), thereby improving the accuracy of control point extraction. For an explanation of the first image, the second image, and the third image, please refer to the relevant descriptions below, which will not be repeated here.

[0082] To better understand the embodiments of this application, the structure of the electronic devices involved in the embodiments of this application will be described below.

[0083] The electronic devices involved in the embodiments of this application may include, but are not limited to, at least one of the following: foldable electronic devices, tablet computers, desktop computers, laptop computers, handheld computers, ultra-mobile personal computers (UMPCs), and artificial intelligence (AI) devices. The embodiments of this application do not impose any special limitations on the specific type of the electronic device.

[0084] Figure 1 A schematic diagram of an electronic device 100 to which the above-described image processing method is applied is shown.

[0085] Electronic device 100 may include processor 110, external memory interface 120, internal memory 121, universal serial bus (USB) connector 130, charging management module 140, power management module 141, battery 142, antenna 1, antenna 2, mobile communication module 150, wireless communication module 160, display screen 170, and camera 171, etc.

[0086] It is understood that the structures illustrated in the embodiments of this application do not constitute a specific limitation on the electronic device 100. In other embodiments of this application, the electronic device 100 may include more than Figure 1 More or fewer components, or combining some components, or splitting some components, or different component arrangements. Figure 1 The components can be implemented in hardware, software, or a combination of both.

[0087] Processor 110 may include one or more processing units, such as application processors (APs), modem processors, graphics processing units (GPUs), image signal processors (ISPs), controllers, video codecs, digital signal processors (DSPs), baseband processors, and / or neural network processing units (NPUs). These different processing units may be independent devices or integrated into one or more processors.

[0088] The processor can generate operation control signals based on the instruction opcode and timing signals to control the instruction fetching and execution.

[0089] The processor 110 may also include a memory for storing instructions and data. In some embodiments, the memory in the processor 110 may be a cache memory. This memory can store instructions or data that the processor 110 has used or that are used frequently. If the processor 110 needs to use the instruction or data, it can directly retrieve it from this memory. This avoids repeated accesses, reduces the waiting time of the processor 110, and thus improves the efficiency of the system.

[0090] In some embodiments, the processor 110 may include one or more interfaces. These interfaces may include an inter-integrated circuit (I2C) interface, an inter-integrated circuit sound (I2S) interface, a pulse code modulation (PCM) interface, a universal asynchronous receiver / transmitter (UART) interface, a mobile industry processor interface (MIPI), a general-purpose input / output (GPIO) interface, a subscriber identity module (SIM) interface, and / or a universal serial bus (USB) interface, etc. The processor 110 can connect to modules such as wireless communication modules and displays (screens) through at least one of these interfaces.

[0091] It is understood that the interface connection relationships between the modules illustrated in the embodiments of this application are merely illustrative and do not constitute a structural limitation on the electronic device 100. In other embodiments of this application, the electronic device 100 may also employ different interface connection methods or combinations of multiple interface connection methods as described in the above embodiments.

[0092] USB connector 130 is a USB standard-compliant interface used to connect electronic device 100 to peripheral devices, specifically a Mini USB connector, Micro USB connector, USB Type-C connector, etc. USB connector 130 can be used to connect a charger, allowing the charger to charge the electronic device 100, or to connect other terminal devices, enabling data transfer between the electronic device 100 and other terminal devices. This connector can also be used to connect other terminal devices, such as VR devices. In some embodiments, the Universal Serial Bus standard specification can be USB 1.x, USB 2.0, USB 3.x, and USB 4.

[0093] The charging management module 140 receives charging input from a charger. The charger can be a wireless charger or a wired charger. In some wired charging embodiments, the charging management module 140 receives charging input from the wired charger via a USB connector 130. In some wireless charging embodiments, the charging management module 140 receives wireless charging input via the wireless charging coil of the electronic device 100. While charging the battery 142, the charging management module 140 can also supply power to the terminal device via the power management module 141.

[0094] The power management module 141 connects the battery 142, the charging management module 140, and the processor 110. The power management module 141 receives input from the battery 142 and / or the charging management module 140, and supplies power to the processor 110, internal memory 121, display screen 170, and wireless communication module 160, etc. The power management module 141 can also monitor parameters such as battery capacity, battery cycle count, and battery health status (leakage current, impedance). In some other embodiments, the power management module 141 may also be located within the processor 110. In other embodiments, the power management module 141 and the charging management module 140 may be located in the same device.

[0095] The wireless communication function of electronic device 100 can be realized through antenna 1, antenna 2, mobile communication module 150, wireless communication module 160, modem processor and baseband processor, etc.

[0096] Antenna 1 and antenna 2 are used to transmit and receive electromagnetic wave signals. Each antenna in electronic device 100 can be used to cover one or more communication frequency bands. Different antennas can also be multiplexed to improve antenna utilization. For example, antenna 1 can be multiplexed as a diversity antenna for a wireless local area network. In some other embodiments, the antennas can be used in conjunction with tuning switches.

[0097] The mobile communication module 150 can provide solutions for wireless communication, including 2G / 3G / 4G / 5G, applied to the electronic device 100. The mobile communication module 150 may include at least one filter, switch, power amplifier, low noise amplifier (LNA), etc. The mobile communication module 150 can receive electromagnetic waves via antenna 1, and perform filtering, amplification, and other processing on the received electromagnetic waves before transmitting them to a modem processor for demodulation. The mobile communication module 150 can also amplify the signal modulated by the modem processor and convert it into electromagnetic waves for radiation via antenna 1. In some embodiments, at least some functional modules of the mobile communication module 150 may be housed in the processor 110. In some embodiments, at least some functional modules of the mobile communication module 150 and at least some modules of the processor 110 may be housed in the same device.

[0098] The wireless communication module 160 can provide solutions for wireless communication applications on the electronic device 100, including wireless local area networks (WLAN) (such as wireless fidelity (Wi-Fi) networks), Bluetooth (BT), Bluetooth Low Energy (BLE), ultra-wideband (UWB), global navigation satellite system (GNSS), frequency modulation (FM), near field communication (NFC), and infrared (IR) technologies. The wireless communication module 160 can be one or more devices integrating at least one communication processing module. The wireless communication module 160 receives electromagnetic waves via antenna 2, performs frequency modulation and filtering of the electromagnetic wave signals, and sends the processed signal to processor 110. The wireless communication module 160 can also receive signals to be transmitted from processor 110, perform frequency modulation and amplification, and convert them into electromagnetic waves for radiation via antenna 2.

[0099] In some embodiments, antenna 1 of electronic device 100 is coupled to mobile communication module 150, and antenna 2 is coupled to wireless communication module 160, enabling electronic device 100 to communicate with networks and other terminal devices via wireless communication technology. This wireless communication technology may include Global System for Mobile Communications (GSM), General Packet Radio Service (GPRS), Code Division Multiple Access (CDMA), Wideband Code Division Multiple Access (WCDMA), Time-Division Code Division Multiple Access (TD-SCDMA), Long Term Evolution (LTE), BT, GNSS, WLAN, NFC, FM, and / or IR technologies, etc. The GNSS may include the Global Positioning System (GPS), the Global Navigation Satellite System (GLONASS), the BeiDou Navigation Satellite System (BDS), the Quasi-Zenith Satellite System (QZSS), and / or satellite-based augmentation systems (SBAS).

[0100] Electronic device 100 can implement display functions through a GPU, display screen 170, and application processor. The GPU is a microprocessor for image processing, connected to the display screen 170 and the application processor. The GPU is used to perform mathematical and geometric calculations and for graphics rendering. Processor 110 may include one or more GPUs, which execute program instructions to generate or modify display information.

[0101] The external storage interface 120 can be used to connect an external memory card, such as a Micro SD card, to expand the storage capacity of the electronic device 100. The external memory card communicates with the processor 110 through the external storage interface 120 to perform data storage functions. For example, it can save video files, pictures, and other files to the external memory card, or transfer video files, pictures, and other files from the electronic device to the external memory card.

[0102] Internal memory 121 can be used to store computer executable program code, including instructions. Internal memory 121 may include a program storage area and a data storage area. The program storage area may store the operating system and at least one application program required for a given function.

[0103] (For example, video playback function, image playback function, etc.). The storage data area can store data created during the use of the electronic device 100 (e.g., video data, etc.). In addition, the internal memory 121 may include high-speed random access memory, and may also include non-volatile memory, such as at least one disk storage device, flash memory device, universal flash storage (UFS), etc. The processor 110 executes various functional methods or data processing of the electronic device 100 by running instructions stored in the internal memory 121 and / or instructions stored in memory set in the processor.

[0104] The display screen 170 is used to display data such as videos, images, and image processing results. For example, the display screen 170 can be used to display images and videos captured by the camera 171. The display screen 170 includes a display panel. The display panel can be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode (AMOLED), a flexible light-emitting diode (FLED), a Mini LED, a MicroLED, a Micro-OLED, a quantum dot light-emitting diode (QLED), etc. In some embodiments, the electronic device 100 may include one or more display screens 170.

[0105] Camera 171 is used to capture images or videos. An object is projected onto a photosensitive element by generating an optical image through the lens. The photosensitive element can be a charge-coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor. The photosensitive element converts the light signal into an electrical signal, which is then transmitted to an ISP (Internet Service Provider) for conversion into a digital image signal. The ISP outputs the digital image signal to a DSP (Digital Signal Processor) for processing. The DSP converts the digital image signal into image signals in standard formats such as RGB and YUV. In some embodiments, the electronic device 100 may include one or N cameras 171, where N is a positive integer greater than 1.

[0106] It is understood that the structures illustrated in the embodiments of this application do not constitute a specific limitation on the electronic device 100. In other embodiments of this application, the electronic device 100 may also include... Figure 1 More or fewer components, or combining some components, or splitting some components, or different component arrangements. Figure 1 The components can be implemented in hardware, software, or a combination of both.

[0107] The software system of the aforementioned electronic device 100 can adopt a layered architecture or a service architecture, etc. This embodiment of the invention uses the layered architecture of the Android operating system as an example to exemplify the software architecture of the electronic device 100. It should be understood that the solution provided in this application can also be applied to other types of operating systems such as HarmonyOS, Apple operating systems, and Windows operating systems.

[0108] Figure 2 A schematic diagram of the software architecture of the electronic device 100 provided in an embodiment of this application is shown. For example... Figure 2 As shown, the layered architecture of the electronic device 100 divides the software into several layers, each with a clear role and division of labor. Layers communicate with each other through software interfaces. In some embodiments, the software components, from top to bottom, are the application (APP) layer, the application framework (FW) layer, the Android runtime (ART) and native C / C++ libraries, the hardware abstraction layer (HAL), and the kernel layer.

[0109] The application layer, also known as the application layer, can include a series of application packages. For example, application layer packages may include gallery applications, video applications, image editing applications, and settings applications. When these application packages are run, they can access the various service modules provided by the application framework layer through application programming interfaces (APIs) and execute corresponding intelligent business logic.

[0110] The application framework layer (FWK) provides application programming interfaces (APIs) and a programming framework for applications in the application layer. The application framework layer includes some predefined functions. For example... Figure 2 As shown, the application framework layer can include a window manager, content providers, a view system, a resource manager, a notification manager, an activity manager, and an input manager. The window manager manages all windows in the system; the content provider stores and retrieves data (such as videos and images) and makes this data accessible to the application; the view system includes visual controls, such as controls for displaying videos and controls for displaying images. The display interface can consist of one or more views. For example, the display interface including the SMS notification icon can include a view for displaying text and a view for displaying images; the resource manager provides the application with various resources, such as images and video files; and the notification manager manages the notification information in the phone's top status bar.

[0111] The Android runtime consists of the core libraries and the Android runtime itself. The Android runtime is responsible for converting source code into machine code. The Android runtime primarily employs ahead-of-time (AOT) compilation and just-in-time (JIT) compilation techniques.

[0112] The core library primarily provides basic Java class library functionalities, such as libraries for fundamental data structures, mathematics, I / O, tools, databases, and networking. It also provides APIs for users to develop Android applications.

[0113] Native C / C++ libraries can include multiple functional modules. Examples include a surface manager and a media framework. The surface manager manages the display subsystem and provides blending of 2D and 3D layers for multiple applications. The media framework supports playback and recording of various common audio and video formats, as well as still image files.

[0114] The Hardware Abstraction Layer (HAL) runs in user space, encapsulates kernel-level drivers, and provides calling interfaces to the upper layers. The HAL includes modules such as display, Bluetooth, and Wi-Fi.

[0115] The kernel layer is the layer between hardware and software. At a minimum, the kernel layer contains display drivers, monitoring drivers, and Wi-Fi drivers to power the display, camera, Bluetooth, and Wi-Fi.

[0116] The following uses Figure 1 and Figure 2 Taking the electronic device with the structure shown as an example, combined with Figure 3 The illustrated software architecture diagram of the electronic device exemplifies the overall flow of the electronic device performing the above-described image processing method.

[0117] Figure 3 In this context, image editing applications run on the operating system of electronic devices. In image editing application scenarios, the human-computer interaction (HCI) module receives video images, point cloud images, etc., input from the user, and displays the image processing results to the user. This HCI module includes an image display module and a control point display module. The image display module displays the user-input video image and the processed image. The control point display module displays the visual positioning control points (i.e., control points) automatically extracted by the control point extraction module, allowing the user to view the distribution of the control points. Furthermore, the HCI module can send the user-input video image to the control point processing module for image processing. Correspondingly, the control point processing module can send the processing results of the video image to the HCI module, which then displays the results through the control point display module.

[0118] The control point processing module includes an image similarity retrieval module, a feature extraction and matching module, a map registration module, a cross-modal feature matching module, and a control point extraction module; the functions of these modules are as follows:

[0119] The image similarity retrieval module is used to retrieve the image (or video) with the highest similarity to the video image input by the user from a basic image library (also known as a basic video library). The image with the highest similarity can also be called the most relevant basic image (or most relevant basic video). It should be noted that both the video image and the most relevant basic image are two-dimensional (2D) images; the method for determining the image with the highest similarity can be found in the relevant description below, and will not be repeated here.

[0120] The feature extraction and matching module is used to extract visual features (such as point features) between video images and the most relevant base images, and to establish matching relationships based on the similarity of visual features and imaging constraints.

[0121] The map registration module is used to establish the correspondence between each visual feature on the most relevant base image and each point cloud data in the point cloud image based on the three-dimensional (3D) point cloud image (referred to as point cloud image) and coordinate transformation model. Individual point cloud data can be represented using 3D coordinates.

[0122] The cross-modal feature matching module is used to determine (or infer) the 3D coordinates corresponding to the visual features of the video image based on the relationship between the visual features of the video image and the visual features of the most associated base image, as well as the correspondence between the visual features of the most associated base image and the 3D coordinates of the point cloud data of the point cloud image.

[0123] The control point extraction module is used to extract control points (i.e. visual positioning control points) that meet the requirements from the visual features of video images according to certain rules (such as the KNN algorithm).

[0124] After obtaining the control points, the control point processing module can send a control point display request to the driver module. The driver module generates a control point display instruction based on the control point display request and sends the control point display instruction to the display screen. The display screen displays the control points in the video image according to the control point display instruction, so that the user can view the distribution of the control points on the video image. For details, please refer to the interface embodiment below, which will not be elaborated here.

[0125] The above describes a software architecture diagram for an electronic device. Below, we will combine... Figure 4This paper introduces an application system architecture applicable to this application. In this architecture, the server can interact with the user through a human-computer interaction interface. For example, the user can input a video image (e.g., the first image below) through the human-computer interaction interface. The server retrieves the video image from the human-computer interaction interface, processes it to obtain control point data, and then displays the control point data to the user through the human-computer interaction interface. The user can view the distribution of control point data on the video image on the human-computer interaction interface to determine whether the control point data extracted by the server meets the requirements. The server includes a basic image library (also called a basic image database) and a point cloud image library (also called a point cloud database). After receiving the video image, the server initiates an image similarity retrieval function and retrieves the image with the highest similarity from the basic image library. The video image serves as the most relevant base image (e.g., the second image below). Based on this most relevant base image, the correspondence between the video image and the corresponding point cloud image in the point cloud image library (e.g., the third image below) is determined. Then, based on the correspondence between the video image and the most relevant base image, and the correspondence between the most relevant base image and the point cloud image, a one-to-one correspondence between each visual feature in the video image and each point cloud data point in the point cloud image is derived, thus obtaining the 3D coordinates of each visual feature in the video image. Finally, the control point extraction function is activated to extract the required visual features from each visual feature in the video image as control points, and the control point data is displayed to the user through a human-computer interaction interface. It should be noted that since each visual feature in the video image already corresponds to a 3D coordinate, the extracted control points also have 3D coordinates.

[0126] The above example illustrates an application system architecture and electronic device software architecture applicable to this application. In actual image processing scenarios, the application system architecture and electronic device software architecture may be the same as or partially the same as the above architecture (e.g., improvements based on the above structure), and this application does not limit them. Before detailing the image processing method proposed in this application below, let's first combine... Figure 5A and Figure 5B The overall flow of the image processing method proposed in this application is illustrated by way of example.

[0127] Figure 5A and Figure 5B The diagrams illustrate the flowcharts of two image processing methods. The difference between the two diagrams is that... Figure 5A The overall processing flow of this method is shown. Figure 5B The data flow relationships between the various images involved in different processing steps of this method are illustrated. It should be noted that... Figure 5A and Figure 5BThe execution entity in the image processing method can be an electronic device, a GPU within the electronic device, or other software or hardware functional modules within the electronic device. The following example illustrates the execution entity of the image processing method using an electronic device, but this does not limit the scope of this application.

[0128] Figure 5A The flowchart of the image processing method shown is as follows:

[0129] Step 501: Acquire video image. Electronic devices can acquire a single frame of video image from surveillance video (e.g., the first image below), such as... Figure 5B As shown in 501.

[0130] Step 502: Obtain the most relevant base image. The electronic device performs an image similarity search in the base image library and obtains the image with the highest similarity to the video image as the most relevant base image (e.g., the second image below), such as... Figure 5B As shown in 501.

[0131] Step 503: Extract visual features from the video image and the most associated base image. The electronic device extracts visual features (e.g., point features) from the video image and the most associated base image respectively, and matches each visual feature in the video image with each visual feature in the most associated base image (e.g., ...). Figure 5B As shown in 502 (in the image), to establish a feature matching association, such as... Figure 5B As shown in 501.

[0132] Step 504: Register the base associated image with the point cloud image. The electronic device determines the correspondence (also called the matching relationship) between each visual feature in the base associated image and each point cloud data in the point cloud image (e.g., the third image below) through a coordinate transformation model. Figure 5B As shown in 503 (in the diagram), this process can also be understood as registering the most correlated base image and the point cloud image through a coordinate transformation model, such as... Figure 5B As shown in 504.

[0133] Step 505: Cross-modal feature matching. This cross-modal feature matching can be understood as: based on the correspondence between the visual features of the video image and the visual features of the most associated base image in Step 502, and the correspondence between the visual features of the most associated base image and each point cloud data in the point cloud image in Step 503, derive the one-to-one correspondence between each visual feature in the video image and each point cloud data in the point cloud image (e.g., ...). Figure 5B As shown in 505), the 3D coordinates of each visual feature in the video image are derived, such as... Figure 5B As shown in 506.

[0134] Step 506: Extract control points. Based on the needs of visual positioning, the electronic device extracts some visual features as control points from the visual features of the video image with matched 3D coordinates (e.g., set R below) according to certain principles (e.g., uniformity principle). This results in a set of control points, which the electronic device can display on at least one of the video image, the most associated base image, or the point cloud image.

[0135] For example, electronic devices can display each control point in a set of control points in a video image, such as... Figure 5B As shown in 507; for example, an electronic device can display each control point in the control point set on the most associated base image, such as Figure 5B As shown in 508; for example, an electronic device can display each control point in the control point set on a point cloud image, such as... Figure 6 As shown in 509.

[0136] The above section outlines the relevant process of the image processing method provided in this application, combining software architecture and application system architecture. The following section details the image processing method in conjunction with different embodiments (such as interface embodiments).

[0137] It should be noted that the subject executing this image processing method can be an electronic device, or a chip, chip system, or processor applied to an electronic device, or a logic module or software that can realize all or part of the functions of an electronic device.

[0138] The following embodiments are described using an electronic device as an example to illustrate the execution of the image processing method 600. Figure 7 The diagram shown is a flowchart of an image processing method 600 provided in an embodiment of this application. The method 600 includes steps S601 to S604, which will be described in detail below.

[0139] S601, the electronic device acquires a first image, which is a two-dimensional image.

[0140] It should be noted that the first image can also be called the first video; the first image can be a frame of image obtained by the electronic device from local video, or a frame of image extracted from a video stream received from another device (such as a monitoring device), or an image manually input by the user. This application does not limit the way the electronic device obtains the first image.

[0141] For example, if an electronic device has a surveillance video stored locally, the electronic device can extract a frame from the surveillance video as the first image.

[0142] In one alternative implementation, the electronic device may also acquire the first image from the video.

[0143] For example, indoor monitoring equipment (such as surveillance cameras) and electronic devices can communicate with each other. The electronic devices can obtain a video clip from the monitoring equipment and extract a frame from the video as the first image.

[0144] S602, the electronic device determines a second image and a third image based on the first image, wherein the second image is a two-dimensional image that matches the first image from at least one two-dimensional image, and the third image is a three-dimensional image that is registered with the second image.

[0145] The second image is the image that matches the first image, also known as the most relevant base image. The third image is a type of 3D point cloud data. It should be noted that in some scenarios, the first, second, and third images are images from different perspectives within the same shooting scene. The matching between the first and second images can be measured using similarity calculations. For example, with a similarity threshold of 90%, when the similarity between the first and second images is greater than or equal to 90%, it indicates that the first and second images match (or are similar); when the similarity is less than 90%, it indicates that the first and second images do not match (or are dissimilar). Electronic devices can select the image with the highest similarity to the first image from a base image library as the second image through similarity calculations.

[0146] It should be noted that in some scenarios, there may be alternative descriptions regarding the "matching of the first image and the second image." For example, when the similarity between the first image and the second image is greater than or equal to a similarity threshold, it can also be considered that the first image and the second image are registered (or aligned or correspond). When the similarity between the first image and the second image is less than a similarity threshold, it can also be considered that the first image and the second image are not registered (or misaligned or non-corresponding).

[0147] In some embodiments, the second image is the two-dimensional image that has the highest similarity to the first image among at least one two-dimensional image.

[0148] For example, the basic image library stores at least one two-dimensional image. The electronic device can retrieve the two-dimensional image with the highest similarity to the first image from the basic image library based on the first image. For example, if the electronic device calculates that the similarity values ​​of three basic images and the first image are all greater than the similarity threshold, the electronic device can sort these three similarity values. The basic image with the largest similarity value to the first image is taken as the second image, thus obtaining the second image that matches the first image.

[0149] After the electronic device identifies the second image that matches the first image, it can extract the visual features of the first image and the second image respectively, and establish the matching relationship between each visual feature in the first image and each visual feature in the second image.

[0150] It should be noted that the aforementioned visual features may refer to point features or other features that are allowed to be processed as point features (such as line features, surface features, etc.), and this application does not limit them.

[0151] In some scenarios, the third image is a 3D image registered with the second image, or alternatively, it can be described as a 3D image that matches (or aligns with or corresponds to) the second image. In some embodiments, since the coordinate transformation model between the acquisition device of the second image and the acquisition device of the third image is known, the second image can establish an association (or correspondence) between visual features of the second image and the third image through the coordinate transformation model, so as to achieve a one-to-one correspondence between the visual features of the second image and the visual features of the third image. In other words, the second image and the third image can be understood as pre-registered (or pre-aligned or pre-matched or pre-calibrated). For an explanation of the coordinate transformation model, please refer to the description in the relevant embodiments below, which will not be repeated here.

[0152] Since the second image is registered with the third image, the higher the accuracy of the electronic device in retrieving the second image that matches the first image through the similarity algorithm, the more beneficial it is to improve the accuracy of matching each visual feature in the first image with the corresponding visual features in the third image.

[0153] For example, such as Figure 7As shown in (a), taking the image acquisition device 700 acquiring the image of scene A as an example, the images acquired by the camera 701 and the lidar device 702 in the image acquisition device 700 are synchronized and acquired from the same angle; the image acquisition device 700 can enable the camera 701 to simultaneously capture images of scene A to obtain at least one two-dimensional image 703; the at least one two-dimensional image 703 can be stored in a local folder or in a basic image library deployed on a remote cloud server 705; in some scenarios, the above-mentioned second image can be one of the at least one two-dimensional image 703 acquired by the camera 703; the basic image library (or local folder) stores the above-mentioned at least one two-dimensional image 703 (or at least one image 703); at the same time, the lidar device 702 is enabled to acquire a 3D image of scene A. The monitoring device 706 can acquire video images of scene A at different times from different angles to obtain a video 707. The monitoring device 706 can send the video 707 to the electronic device 708. The electronic device 708 can obtain a frame from the video 707 as a first image and perform an image similarity search in the basic image library (or local folder) to retrieve a second image that matches the first image. After determining the second image, the electronic device 708 can determine a third image that is registered (or matched) with the second image from the point cloud image library (or local folder).

[0154] It should be noted that since the coordinate transformation model between the image captured by camera 701 in image acquisition device 700 and the point cloud image captured by lidar device 702 is known, the visual features (e.g., point features) of the second image can be mapped one-to-one with the point cloud data in the third image through the known coordinate transformation model. In other words, the second image and the third image are registered (or aligned). For details, please refer to the relevant explanation of coordinate model transformation below, which will not be elaborated here. Electronic device 708 can determine the matching second image from the basic image library based on the first image, and determine the matching third image from the point cloud image library based on the second image and the coordinate transformation model. Finally, using the second image as a bridge, based on the correspondence between the first and second images and the correspondence between the second and third images, the one-to-one correspondence between each visual feature in the first image and each point cloud data in the third image is derived, thereby obtaining the 3D coordinates of each visual feature in the first image.

[0155] For example, such as Figure 7As shown in (b), since the first image and the second image are matched, the electronic device 708 can determine the correspondence between each visual feature (e.g., feature point S1) in the first image and each visual feature (e.g., feature point S2) in the second image; since the second image and the third image are registered, the electronic device 708 can also determine the correspondence between each visual feature (e.g., feature point S2) in the second image and the 3D coordinates (e.g., S3(x,y,z)) of each point cloud data (e.g., feature point S3) in the third image; based on the correspondence between the visual features in the first image and the second image (e.g., S1 corresponds to S2) and the correspondence between the second image and the third image (e.g., S2 corresponds to S3), the electronic device 708 can deduce the one-to-one correspondence between each visual feature in the first image and each point cloud data in the third image, thereby obtaining the 3D coordinates (e.g., S1(x,y,z)) of each visual feature in the first image.

[0156] S603, the electronic device determines a matching second pixel from the second image based on a first pixel in the first image.

[0157] It should be noted that in this application, a single visual feature point (or a single visual feature) can be represented by a single pixel; the aforementioned first pixel may refer to a visual feature point corresponding to a pixel in the first image, and the second pixel is a visual feature point corresponding to a pixel in the second image, which is a visual feature point in the second image corresponding to the first pixel.

[0158] Electronic devices can extract visual feature points corresponding to each pixel in a first image (or second image) according to visual feature extraction algorithms, including but not limited to SIFT, SURF, and ORB algorithms. Accordingly, visual feature points can be at least one of SIFT, SURF, or ORB feature points; specifically, visual feature points obtained by processing the first image (or second image) using the SIFT algorithm are SIFT feature points, visual feature points obtained by processing the first image (or second image) using the SURF algorithm are SURF feature points, and visual feature points obtained by processing the first image (or second image) using the ORB algorithm are ORB feature points.

[0159] In some embodiments, the electronic device may use the SIFT algorithm (SURF algorithm or ORB algorithm) to extract visual features from the pixels of the first image (or the second image) to obtain at least one SIFT feature point (SURF feature point or ORB feature point).

[0160] For example, taking the SIFT algorithm to extract point features from an image as an example, an electronic device can extract point features from a first image using the SIFT algorithm to obtain a first SIFT point feature set, and then extract point features from a second image using the SIFT algorithm to obtain a second SIFT point feature set; wherein, the first pixel is a SIFT point feature in the first SIFT point feature set, and the second pixel is a SIFT point feature in the second SIFT point feature set that matches the first pixel.

[0161] For example, an electronic device extracts point features from a first image using the SIFT algorithm, obtaining a first SIFT point feature set F, F = {f1, f2, ..., f...}. M Similarly, the electronic device extracts point features from the second image using the SIFT algorithm, obtaining a second SIFT point feature set F', where F' = {f'1, f'2, ..., f'}. N The electronic device performs a one-to-one matching between set F and its elements (e.g., point features f). a With point feature f' a Matching, point features f b With point feature f' b (e.g., matching) to establish a feature matching relationship L between set F and set F, L = {(f a f' a ), (f b f' b ), ..., (f K f' K )}, where f a f b , ..., f K ∈F,f' a f' b ,…,f' K ∈F, where M, N and K are all integers.

[0162] For example, the first pixel is f a The second pixel is f' a The first pixel f a With the second pixel f' a These are mutually matching (or mutually corresponding) visual features (or mutually matching pixels). Electronic devices extract point features (e.g., first or second pixels) from a first (or second) image using the SIFT algorithm with higher reliability.

[0163] It should be noted that the processing method of SURF algorithm (or ORB algorithm) for extracting visual features is similar to that of SIFT algorithm for extracting visual features (e.g., extracting point features from the first image or the second image), and will not be repeated here.

[0164] Therefore, in different application scenarios, electronic devices can use different feature processing algorithms (such as SURF, SIFT or ORB algorithms) to extract visual features from the pixels of an image to ensure the accuracy of image matching based on visual feature points in different application scenarios.

[0165] S604, the electronic device determines the registered third pixel from the third image based on the second pixel.

[0166] The third pixel is a point cloud data or a three-dimensional coordinate point (i.e., a 3D coordinate point) in the third image. Since the first pixel matches the second pixel, and the second pixel matches the third pixel, the pixel corresponding to the first pixel in the third image is the third pixel. In other words, the first pixel matches the third pixel, and the third pixel is the 3D coordinate corresponding to the first pixel.

[0167] For example, in S603, the electronic device has already determined the second pixel. Since the second image and the third image are registered with each other, the electronic device can determine the third pixel that is registered with the second pixel from the third image. Since the first pixel corresponds to the second pixel, the pixel in the third image that corresponds to the second pixel is the third pixel. Therefore, the pixel corresponding to the first pixel in the third image is the third pixel. And since the third pixel is a 3D coordinate point, the 3D coordinates corresponding to the first pixel are the 3D coordinates of the third pixel.

[0168] In some embodiments, the electronic device determines a registered third pixel from a third image based on the second pixel, including: determining the third pixel based on the second pixel and a coordinate transformation model.

[0169] In some scenarios, electronic devices can determine the point cloud data (e.g., the third pixel) corresponding to the second pixel in the third image based on a coordinate transformation model. For example, electronic devices can make a one-to-one correspondence between each visual feature (e.g., the second pixel) in the second image and each point cloud data (e.g., the third pixel) in the third image based on a coordinate transformation model to obtain the 3D coordinates corresponding to each visual feature (e.g., the second pixel) in the second image. For example, the 3D coordinates corresponding to the second pixel are the 3D coordinates of the third pixel.

[0170] It should be noted that since the coordinate transformation model between the second and third images is fixed and known, it can be understood that the second and third images are registered (or aligned) in the same coordinate system. Electronic devices can obtain the coordinate transformation model between the second and third images and establish the correspondence between each visual feature (e.g., SIFT point feature) in the second image and each point cloud data in the third image through the coordinate transformation model.

[0171] In this embodiment, the electronic device registers (or corresponds) the second pixel with the third pixel using a coordinate transformation model, which has the following advantages: 1) The coordinate transformation model aligns the second and third pixels in the same coordinate system, thereby achieving data fusion. This cross-dimensional image information integration can provide more comprehensive and accurate data support for subsequent control point extraction operations; 2) The coordinate transformation model can accurately register the second and third pixels, reducing registration errors caused by inconsistencies in coordinate systems, thus helping to improve the accuracy and precision of subsequent control point extraction; 3) Since the coordinate transformation model is a technique based on mathematical algorithms and computational optimization, it can achieve efficient data processing and transformation. Therefore, registering the second and third pixels using the coordinate transformation model can reduce the amount of computation and data processing time, which is beneficial to improving the overall efficiency of image processing.

[0172] In other embodiments, the first acquisition module and the second acquisition module meet the following requirements during the image acquisition process: synchronization and position fixation, wherein the first acquisition module is used to acquire the second image, the second acquisition module is used to acquire the third image, and the coordinate transformation model is related to synchronization and position fixation.

[0173] Synchronicity (also expressed as simultaneity) can refer to the fact that the second and third images are acquired simultaneously (or almost simultaneously), that is, the two images capture the scene at the same or similar time. Positional fixity (also expressed as positional stability or relative positional invariance) can refer to the fact that the second and third images are acquired from the same angle (i.e., viewpoint) or the same position, that is, they capture the scene from the same angle (i.e., viewpoint).

[0174] By ensuring synchronicity and fixed position between the first and second acquisition modules during image acquisition, it is guaranteed that the second and third images are acquired at the same time and angle. The advantages of this synchronous, same-angle image acquisition method are as follows: 1) Since the second and third images are acquired synchronously at the same angle, each pixel in both images can be accurately matched. This accurate correspondence makes the measurement results more accurate, improving the accuracy of subsequent control point extraction. 2) Because the second and third images are acquired from the same angle, they can provide information from different dimensions of the same scene (such as texture or depth information), thus complementing each other and improving the situation of insufficient single-view information caused by occlusion and other issues. 3) Compared with asynchronous or different-angle image acquisition methods, synchronous, same-angle image acquisition does not require complex scene matching or angle transformation during preprocessing, which helps reduce computational load and improve the overall efficiency of image processing.

[0175] The statement that "the coordinate transformation model is related to synchronization and positional fixation" can be understood as follows: the determination of the coordinate transformation model is closely linked to the synchronization and positional fixation of the first and second acquisition modules during image acquisition. Specifically, if the first and second acquisition modules fail to achieve synchronization during image acquisition, it will directly affect the time parameters in the coordinate transformation model that are closely related to the acquisition time, thus causing changes in the coordinate transformation model and consequently affecting the registration (or correspondence) of each pixel in the second and third images. Similarly, if the positions of the two acquisition modules change during image acquisition, the acquisition angle will change, thereby affecting the coordinate parameters in the coordinate transformation model that are related to the acquisition angle, thus causing changes in the coordinate transformation model and consequently affecting the registration (or correspondence) of each pixel in the second and third images. Therefore, ensuring that the first and second acquisition modules maintain a high degree of synchronization and positional fixation during image acquisition is crucial for achieving the acquisition of the second and third images at the same time and angle. This approach ensures that the coordinate transformation model remains fixed, eliminating the need for frequent adjustments due to changes in time and angle, thereby improving the efficiency and accuracy of registration (or alignment) between the two images.

[0176] The first acquisition module (or the second acquisition module) can be an acquisition device, software module, or chip, etc., and this application does not limit it.

[0177] Furthermore, the first acquisition module (or the second acquisition module) can be integrated into the same device or belong to different devices. For example, when both belong to the same device (such as electronic device A), they can capture a specific scene synchronously and from the same perspective. However, when they belong to different devices, these devices need to maintain the synchronization of acquisition time to capture the scene at the same or similar time during the acquisition of the second and third images respectively. On the other hand, they also need to maintain the fixity of the device position (i.e., the stability of the device position) to ensure that the image is acquired from the same angle (i.e., the perspective).

[0178] For example, such as Figure 8 As shown in (a), taking a LiDAR device 702 with a camera 701 as the first acquisition module and an image acquisition device 700 as the second acquisition module as an image acquisition device 700 as an example, the camera 701 and the LiDAR device 702 are two parts of the same image acquisition device 700; the camera 701 is used to acquire the second image (i.e., an example of the most relevant base image), and the LiDAR device 702 is used to acquire the third image (i.e., an example of a point cloud image); the acquisition time and acquisition angle of the camera 701 and the LiDAR device 702 are the same during the image acquisition process, and the coordinate transformation model between the camera 701 and the LiDAR device 702 is P. u'v’ =kMP w Electronic devices can be modeled as P through coordinate transformation. u'v' Establish a matching relationship between the SIFT point features of the second image and the point cloud data (or 3D coordinates) of the third image, where M represents the coordinate transformation model, k represents the scale parameter, (u', v') represents the pixel position corresponding to point feature f' in the second SIFT point feature set F', and P w The 3D coordinates (x, y, z) of the point cloud (or point cloud data); the feature matching relationship L between the above sets F and F' is {f' a f' b ,…,f' K The set of 3D point cloud coordinates corresponding to each SIFT point feature is P = {p1, p2, ..., P}. K That is, the electronic device uses a coordinate transformation model to match the SIFT point features (i.e., an example of the second pixel) in the second image with the corresponding 3D coordinates, thereby obtaining a new coordinate set P = {p1, p2, ..., P}. K}

[0179] If the point cloud data M(x, y, z) in the third image matches the second pixel, then the 3D coordinates of the first pixel in the third image are M(x, y, z). By repeating method 600, each visual feature in the first image can find its corresponding 3D coordinates in the third image; ultimately, a set of feature points with matched 3D coordinates, R = {R1, R2, R3, ..., R...}, is obtained. i , ...}, where R i For example, R is a visual feature in the first image whose 3D coordinates have been matched. i It can be the first pixel point that has been matched with 3D coordinates.

[0180] Therefore, in method 600, the electronic device performs same-dimensional matching between a pixel in the first image (e.g., the first pixel) and the corresponding pixel in the second image (e.g., the second pixel). This matching is not only relatively easy but also highly accurate. Furthermore, since the second and third images are already registered, each pixel (or visual feature) in the second image has a corresponding spatial position in the third image. Thus, the registered third pixel can be determined from the third image based on the second pixel. Also, since the first and second images are matched (or registered), and the second and third images are registered, the registration of the first and third images can be indirectly achieved. This method of indirectly registering the first and third images through an intermediate image (e.g., the second image) not only improves the accuracy of registration but also improves the accuracy of control point extraction.

[0181] In an alternative implementation, the above method 600 further includes: S605;

[0182] S605: The electronic device determines the coordinates of the first control point based on the third pixel.

[0183] Wherein, the third pixel is the pixel corresponding to the first pixel in the third image (i.e., an example of point cloud data), the 3D coordinates of the first pixel are the 3D coordinates of the third pixel; the coordinates of the first control point are the coordinates of the first pixel, and the first control point is the first pixel.

[0184] As described above, an electronic device can determine the 3D coordinates of the first pixel based on the third pixel. Therefore, when the electronic device determines the first pixel in the first image as the first control point based on the clustering algorithm, it can determine the coordinates of the first control point based on the third pixel.

[0185] In other words, by executing S604, the electronic device can match each visual feature (e.g., the first pixel) in the first image with each point cloud data (e.g., the third pixel) in the third image to obtain the aforementioned set R; finally, according to the needs of visual positioning, it extracts several elements from the set R as control points, such as the first control point.

[0186] It should be noted that, in some embodiments, the electronic device can process the third pixel point using clustering algorithms, uniformity principles, or other methods to determine the coordinates of the first control point.

[0187] For example, the electronic device determines the coordinates of the first control point based on the third pixel, including: clustering the third pixel using a clustering algorithm to determine the first control point, wherein the first control point includes the first pixel; and the electronic device determines the coordinates of the first control point based on the coordinates of the third pixel.

[0188] It should be noted that the electronic device determines the first control point by clustering set R using a clustering algorithm. Since set R includes the 3D coordinates of the first pixel, and the 3D coordinates of the first pixel are determined based on the third pixel, the clustering of set R by the electronic device can also be understood as the clustering of the third pixel. In other words, the electronic device can cluster set R, which includes the third pixel, to extract the first pixel as the first control point. At this time, the coordinates of the first control point are the coordinates of the first pixel (i.e., the coordinates of the third pixel).

[0189] In some examples, after obtaining a set R, the electronic device can select some elements from set R as control points according to the needs of visual positioning. For example, the electronic device uses a clustering algorithm to cluster the elements in set R and determine X elements to be extracted as X control points. These X elements include the first pixel, meaning the X control points include the first control point, and the coordinates of the first control point are the 3D coordinates of the first pixel. The clustering algorithms include, but are not limited to, the KNN algorithm and the K-means algorithm.

[0190] For example, electronic devices can use the KNN clustering algorithm to cluster elements in a set R to extract multiple elements as control points; for example, these control points may include {control point 1 (uv... a p1), control point 2 (uv) b p2), ..., control point N(uv) N P N )}, where uv i Representing feature f i The corresponding pixel position (or the coordinates of the pixel), i∈{a, b, ..., N}.

[0191] In other examples, electronic devices can also extract a subset of elements from set R as control points using the principle of uniformity. For example, the elements in set R can be evenly divided into N blocks, and one element can be extracted from each block as a control point.

[0192] In some other examples, after obtaining a set R, the electronic device can also use all elements in set R as control points.

[0193] In this embodiment, the user can automatically select appropriate control points (e.g., the first control point) through a clustering algorithm according to the needs of visual positioning, without the need for manual selection, which is efficient and accurate.

[0194] The method 600 and its possible implementations have been described in detail above. The following section takes an electronic device as a terminal device (e.g., a laptop computer) as an example and combines interface embodiments for different application scenarios to further explain the method 600 and its possible implementations. It should be noted that the following interface embodiments are only used to illustrate the method 600 and should not be construed as limiting the application scenarios of this application. In addition, it should be noted that the terminal device may include a display screen, and the user can perform operations such as image processing on the display interface shown on the display screen.

[0195] like Figure 9 As shown in (a), the user can double-click the shortcut icon of "Image Editing 802" on the terminal's display interface 801 to launch the image editing software. At this time, the user can see the main interface 803 of the image editing software. The main interface 803 includes a toolbar 804, an image display area 805, and an operation bar 806. The toolbar 804 includes operation tools such as text editing, deletion, and favorites. The image display area 805 is used to display the video image (also known as the image to be processed) (i.e., an example of the first image) input by the user and the image processing result (i.e., the processing result of the video image). It should be noted that the image displayed in the image display area 805 may be different in different scenarios. For details, please refer to the interface embodiment below, which will not be repeated here. The operation bar 806 includes an upload button 8061, a split display button 8062, a one-click processing button 8063, and a step-by-step viewing button 8064.

[0196] The "Upload 8061" button allows users to upload video images that need processing. It should be noted that after clicking the "Upload 8061" button, users can select the video images to be processed locally or log in to a remote cloud server to obtain the video images to be processed from the cloud server.

[0197] The function of the above-mentioned column display 8062 button is to control the display mode of the image display area 805. For example, the image display area 805 can be displayed in a single column, or displayed in columns side by side (or multiple columns side by side). For details, please refer to the interface embodiment below, which will not be described in detail here.

[0198] The function of the aforementioned one-click processing button 8063 is to perform one-click processing on video images uploaded by users that require processing. For example, if a user clicks the one-click processing button 8063, the terminal will execute the entire process of the aforementioned method 600 and display the control point extraction result on the image display area 805 so that the user can view the control point extraction result.

[0199] It should be noted that the control point extraction results are used to display multiple control points extracted by the electronic device. These control points can be displayed on at least one of the following images: video image, point cloud image, and most correlated base image. Here, we only take the display of the distribution of multiple control points on a video image using the control point extraction results as an example.

[0200] The function of the step-by-step viewing button 8064 is different from that of the one-click processing button 8063. The function of the step-by-step viewing button 8064 is to view the processing status of the video image at each step during the execution of method 600. If the user is not satisfied with the processing result of a certain step, he / she can trigger the terminal to re-execute the step through the step-by-step viewing button 8064. For details, please refer to the interface embodiment below, which will not be described in detail here.

[0201] For example, such as Figure 9 As shown in (a), taking the user uploading video image 901 (i.e., an example of the first image) from the local computer as an example, the user clicks the upload 8061 button to upload the video image 901 that needs to be processed; after uploading the video image 901, the user can click the one-click processing 8063 button to start the one-click processing function; at this time, the terminal will execute S601 to S604 in the above method 600, and display the extraction result of the control point (e.g., black dot 902) in the corresponding area of ​​the image display area 805, such as... Figure 10 As shown in (b).

[0202] For example, such as Figure 10 As shown in (a), the user clicks the upload 8061 button to upload video image 901 from the local machine; after uploading video image 901, the user can first click the split-column display 8062 button to set the display format of the image display area 805; for example, the user first clicks the split-column display 8062 button and selects three-column display 1002 in the split-column display 8062 option box 1001, as shown in (a). Figure 10 As shown in (a); then click the "One-Click Processing 8063" button to activate the one-click processing function, as shown in (a).Figure 10 As shown in (b) above; at this time, the terminal will also execute S601 to S604 of the above method 600; since the user has set a three-column display, after the terminal processes the video image 901, it can display the video image 901, the point cloud image (i.e., an example of the third image), and the control point extraction results in a three-column arrangement in the image display area 805. The left column 1003 is used to display the video image 901, the middle column 1004 is used to display the point cloud image, and the right column 1005 is used to display the control point extraction results, as shown below. Figure 11 As shown in (b).

[0203] For example, such as Figure 11 As shown in (a), the user clicks the upload 8061 button to upload video images 901 from the local machine; if the user first clicks the split-column display 8062 button and selects four-column display 1101 in the split-column display 8062 option box 1001; and then clicks the one-click processing 8063 button to start the one-click processing function, as shown in (a), the user uploads ... Figure 11 As shown in (b) above; at this time, the terminal will also execute S601 to S604 of the above method 600; since the user has set a four-column display, after the terminal processes the video image 901, the terminal will display the video image 901, the most associated base image (i.e., an example of the second image), the point cloud image, and the control point extraction results in the corresponding areas of the image display area 805 in a four-column manner. From left to right, the first column 1102 is used to display the video image 901, the second column 1103 is used to display the most associated base image, the third column 1104 is used to display the point cloud image, and the fourth column 1105 is used to display the control point extraction results, as shown below. Figure 11 As shown in (b). From Figure 12 As shown in image (b), the first pixel A in the video image corresponds to the second pixel B in the most associated base image; the second pixel B in the most associated base image corresponds to the third pixel C in the point cloud image; by executing S604, the terminal can determine that the first pixel A corresponds to the third pixel C in the point cloud image; by executing S605, the terminal extracts the first pixel A as a control point. At this time, the user can see from the control point extraction result that the first pixel A appears as the first control point on the video image 901, and the user can click on the first control point (i.e., the first pixel A) in the control point extraction result to see the 3D coordinates of the first control point (i.e., the first pixel A), namely A(x1,y1,z1).

[0204] For example, such as Figure 12As shown in (a), the user clicks the upload 8061 button to upload video images 901 from the local machine; if the user first clicks the split display 8062 button and selects five-column display 1201 in the split display 8062 option box 1001; then clicks the one-click processing 8063 button to start the one-click processing function, as shown in (a). Figure 12 As shown in (b); at this time, the terminal will also execute S601 to S604 of the above method 600; since the user has set a five-column display, after the terminal processes the video image 901, the terminal will display the video image 901, the most associated base image (i.e., an example of the second image), the point cloud image (i.e., an example of the third image), the 2D-3D matching result, and the control point extraction result in the corresponding areas of the image display area 805 in a five-column manner. Among them, from the left, the first column 1202 is used to display the video image 901, the second column 1203 is used to display the most associated base image, the third column 1204 is used to display the point cloud image, the fourth column 1205 is used to display the 2D-3D matching result, and the fifth column 1206 is used to display the control point extraction result, as shown. Figures 13-17 As shown in (b).

[0205] It should be noted that the 2D-3D matching result is used to display the image processing results performed by the terminal in the following steps: Based on the correspondence between visual features in video image 901 and the most associated base image (e.g., the first pixel A corresponds to the second pixel B), and the correspondence between the most associated base image and the point cloud image (e.g., the second pixel B corresponds to the third pixel C), the terminal can deduce the one-to-one correspondence between each visual feature in video image 901 and each point cloud data in the point cloud image, thereby obtaining the 3D coordinates of each visual feature in video image 901 (e.g., the 3D coordinates of the first pixel A are (x, y, z)). The 2D-3D matching result displays the 3D coordinates of each visual feature in video image 901. For example, if a user clicks on the first pixel A in the 2D-3D matching result, they can see the 3D coordinates of pixel A, i.e., A(x1, y1, z1). It should also be noted that the following... Figure 12 In the image display area 805, the interpretation of the first pixel point A, the second pixel point B, and the third pixel point C identified on different images is as follows: Figure 12 The situation shown in (b) is similar and can be referred to. Figure 13 The explanation of (b) will not be repeated below.

[0206] The following example illustrates how users can view the image processing results at different stages of method 600 by selecting five columns to display the results.

[0207] For example, such as Figure 13As shown in (a), after the terminal processes the video image, it displays the processing results of the images at different processing stages in a five-column format in the image display area 805. From left to right, the first column 1301 displays the video image (an example of the first image), the second column 1302 displays the most relevant base image (an example of the second image), the third column 1303 displays the point cloud image (an example of the third image), the fourth column 1304 displays the 2D-3D matching results, and the fifth column 1305 displays the control point extraction results. At this time, the user can click the "Step-by-Step View 8064" button and select "Similarity Search 1307" in the option box 1306 of "Step-by-Step View 8064" (equivalent to triggering the terminal (an example of an electronic device) to start the image similarity search module mentioned above). At this time, the second column 1302 becomes selected (as shown by the thick black box). Figure 13 As shown in (b), the processing results of similarity search 1307 corresponding to the second column 1302 are displayed to the user so that the user can view the processing results (i.e., the most relevant base image); if the user believes that the current similarity search results do not meet the requirements, the user can click the "Re-execute 1308" button after the "Similarity Search" option (e.g., Figure 13 As shown in (a) above, the terminal re-executes the processing steps corresponding to the similarity retrieval 1307 (for example, re-executes the electronic device in method 600 above to retrieve a second image matching the first image from the base image library). Simultaneously, the fifth column 1305 also becomes selected, just like the second column 1302 (i.e., shown by the thick black box). When the most relevant base image displayed in the second column 1302 is updated, the control point extraction results displayed in the fifth column 1305 will also be updated accordingly, such as... Figure 14 As shown in (b) of the diagram.

[0208] For example, such as Figure 14As shown in (a), the user clicks the "Step-by-Step View 8064" button and selects "Associated Feature Matching 1401" in the option box 1306 of "Step-by-Step View 8064". At this time, the first column 1301, the second column 1302, and the third column 1303 are all selected (i.e., shown by the thick black box), to show the user the processing results of the associated feature matching corresponding to the first column 1301, the second column 1302, and the second column 1303, so that the user can view the processing results (i.e., the matching results of the video image and the most associated base image and the matching results of the most associated base image and the point cloud image); if the user believes that the current associated feature matching results do not meet the requirements, then, Users can click the "Re-execute 1402" button next to the "Associated Feature Matching" option to re-execute the processing steps corresponding to the associated feature matching (for example, re-execute S602 and S603 executed by the electronic device in method 600 above). Simultaneously, column 1305, like columns 1301, 1302, and 1303, becomes selected (shown by the thick black box). When the visual features on the most associated base image displayed in column 1302 and the point cloud data in the point cloud image displayed in column 1303 are updated, the control point extraction results displayed in column 1305 will also be updated accordingly. Figure 15 As shown in (b) of the diagram.

[0209] For example, such as Figure 15 As shown in (a), when the user clicks the "Step-by-Step View 8064" button and selects "Cross-Membrane Feature Matching 1501" in the option box 1306 of "Cross-Step View 8064", the fourth column 1304 becomes selected (i.e., shown by the thick black box), to show the user that the 2D-3D matching result displayed in the fourth column 1304 corresponds to the processing result of cross-membrane feature matching, so that the user can view the 2D-3D matching result (i.e., the matching result between the video image and the point cloud image); if the user believes that the current cross-membrane feature matching does not meet the requirements, then... Users can click the "Re-execute 1502" button next to the "Cross-membrane feature matching" option to re-execute the processing steps corresponding to the cross-membrane feature matching (for example, re-execute S603 and S604 executed by the electronic device in method 600 above). Simultaneously, column 1305 will also become selected, just like column 1304 (shown by the thick black box). When the visual features in the 2D-3D matching results displayed in column 1304 are updated, the control point extraction results displayed in column 1305 will also be updated accordingly, such as... Figure 16 As shown in (b) of the diagram.

[0210] For example, such as Figure 16As shown in (a), the user clicks the "Step-by-Step View 8064" button and selects "Control Point Extraction 1601" in the option box 1306 of "Step-by-Step View 8064". At this time, the fifth column 1305 becomes selected (i.e., shown by the thick black box), to show the user that the control point extraction result displayed in the fifth column 1305 corresponds to the processing result of control point extraction 1601, so that the user can view the processing result (for example, the terminal extracts visual features (i.e., control points) that meet the requirements based on the 2D-3D matching result using a clustering algorithm). If the user believes that the current control point extraction result does not meet the requirements, the user can click the "Re-execute 1602" button after the "Control Point Extraction 1601" option, so that the terminal re-executes the processing step corresponding to control point extraction 1601 (for example, re-executes S605 executed by the electronic device in method 600 above). When the terminal re-executes the processing step corresponding to control point extraction 1601, the control point extraction result displayed in the fifth column 1305 will also be updated accordingly. Figure 17 As shown in (b) of the diagram.

[0211] It should be noted that if users want to examine the processing results of a specific stage of the image more closely, they can use the zoom-in function to view the image; for example, ... Figure 17 As shown in (a), the user right-clicks on the area containing the control point extraction results to bring up the tools tab 1701, and selects the "Magnify Display" function 1702 in tab 1701. At this time, the terminal will generate a magnified display window 1703 on the main interface 803, and magnify the control point extraction results in window 1703, as shown in the example. Figure 18 As shown in (b) in the figure; users can view the control point extraction results in window 1703; of course, users can also adjust the scaling effect of the control point extraction results in window 1703 using scaling tools (such as zoom in or zoom out tools).

[0212] The foregoing has detailed examples of the image processing methods provided in this application. It should be noted that, in order to achieve the above functions, the electronic device includes corresponding hardware structures and / or software modules for performing each function. Those skilled in the art should readily recognize that, based on the modules and algorithm steps of the examples described in conjunction with the embodiments disclosed herein, this application can be implemented in hardware or a combination of hardware and computer software. Whether a function is executed in hardware or by computer software driving hardware depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0213] This application can divide the image processing method into functional modules based on the above method examples. For example, each function can be divided into its own functional module, or two or more functions can be integrated into one module. These modules can be implemented in hardware or as software functional modules. It should be noted that the module division in this application is illustrative and represents only one logical functional division; in actual implementation, other division methods may be used.

[0214] Figure 6 This illustration shows a schematic diagram of an image processing apparatus 1800 provided in this application; the apparatus 1800 may be a terminal device, a server, an image processing system, or a chip (system). The apparatus 1800 includes a first acquisition module 1801, a second acquisition module 1802, an acquisition module 1803, and a processing module 1804;

[0215] The acquisition module 1803 is used to acquire a first image, which is a two-dimensional image;

[0216] The processing module 1804 is configured to determine a second image and a third image based on a first image, wherein the second image is a two-dimensional image that matches the first image from at least one two-dimensional image, and the third image is a three-dimensional image registered with the second image; it is also configured to determine a matching second pixel from the second image based on a first pixel from the first image; and it is also configured to determine a registered third pixel from the third image based on the second pixel.

[0217] For details on the method by which device 1800 performs image processing and the beneficial effects thereof, please refer to [link to relevant documentation]. Figure 18 The relevant descriptions in the method embodiments shown.

[0218] exist Figure 18 In the device 1800 shown, the first pixel is a visual feature point corresponding to a pixel in the first image, and the second pixel is a visual feature point corresponding to a pixel in the second image. The beneficial effects of this part can be seen in the above embodiments.

[0219] exist Figure 18 In the device 1800 shown, the visual feature points are one of SIFT feature points, SURF feature points, or ORB feature points. The beneficial effects of this part can be seen in the above embodiments.

[0220] exist Figure 18 In the apparatus 1800 shown, the processing module 1804 is specifically used to determine the third pixel point based on the second pixel point and the coordinate transformation model. The beneficial effects of the processing module 1804 performing this step can be seen in the above embodiments.

[0221] exist Figure 18In the illustrated device 1800, the first acquisition module 1801 and the second acquisition module 1802 meet the following requirements during image acquisition: synchronization and position fixation. Specifically, the first acquisition module 1801 is used to acquire the second image, and the second acquisition module 1802 is used to acquire the third image. The coordinate transformation model is related to synchronization and position fixation. The beneficial effects of this part can be seen in the above embodiments.

[0222] exist Figure 18 In the apparatus 1800 shown, the processing module 1804 is further configured to: determine the coordinates of the first control point based on the third pixel. The beneficial effects of the processing module 1804 performing this step can be seen in the above embodiments.

[0223] exist Figure 18 In the apparatus 1800 shown, the processing module 1804 is further configured to: cluster the third pixel points using a clustering algorithm to determine a first control point, the first control point including the first pixel point; and determine the coordinates of the first control point based on the coordinates of the third pixel point. The beneficial effects of the processing module 1804 performing this step can be seen in the above embodiments.

[0224] exist Figure 18 In the apparatus 1800 shown, the second image is a two-dimensional image that matches the first image from at least one two-dimensional image, including: the second image is the two-dimensional image with the highest similarity to the first image from at least one two-dimensional image. The beneficial effects of this part can be seen in the above embodiments.

[0225] exist Figure 19 In the apparatus 1800 shown, the acquisition module 1803 is specifically used to: acquire a first image from the video. The beneficial effects of the acquisition module 1803 performing this step can be seen in the above embodiments.

[0226] Figure 19 A schematic diagram of the structure of an electronic device 1900 provided in this application is shown. Figure 6 The dashed lines indicate that the unit or module is optional. Electronic device 1900 can be used to implement the methods described in the above method embodiments. Electronic device 1900 can be a server or a chip (system).

[0227] Electronic device 1900 includes one or more processors 1901, which enable electronic device 1900 to implement ​The method described in the corresponding method embodiment. Processor 1901 can be a general-purpose processor or a dedicated processor. For example, processor 1901 can be a central processing unit (CPU). The CPU can be used to control electronic device 1900, execute software programs, and process data from the software programs. Electronic device 1900 may also include a communication unit 1905 for implementing signal input (reception) and output (transmission).

[0228] The aforementioned electronic device 1900 may be a chip (system) including a memory and a processor, wherein the processor is configured to execute a computer program stored in the memory to implement the methods shown in the various embodiments above.

[0229] The communication unit 1905 may be an input and / or output circuit of the chip (system), or the communication unit 1905 may be a communication interface of the chip (system), which may be a component of the electronic device 1900.

[0230] For example, the communication unit 1905 may be a transceiver of the electronic device 1900, or the communication unit 1905 may be a transceiver circuit of the electronic device 1900.

[0231] The electronic device 1900 may include one or more memories 1902 storing a program 1904. The program 1904 can be executed by a processor 1901 to generate instructions 1903, causing the processor 1901 to perform the methods described in the above method embodiments according to the instructions 1903. Optionally, the memory 1902 may also store data. Optionally, the processor 1901 may also read data stored in the memory 1902, which may be stored at the same memory address as the program 1904, or it may be stored at a different memory address.

[0232] The processor 1901 and memory 1902 can be configured separately or integrated together, for example, integrated on a system-on-chip (SOC) of an electronic device.

[0233] For details on how processor 1901 performs image processing, please refer to the relevant description in the method embodiments.

[0234] It should be understood that the steps of the above method embodiments can be implemented by hardware logic circuits or software instructions in the processor 1901. The processor 1901 can be a CPU, a digital signal processor (DSP), a field programmable gate array (FPGA), or other programmable logic devices, such as discrete gate, transistor logic devices, or discrete hardware components.

[0235] This application also provides a computer program product that, when executed by processor 1901, implements the method of any method embodiment of this application.

[0236] The computer program product can be stored in memory 1902, for example, program 1904. Program 1904 is finally converted into an executable object file that can be executed by processor 1901 after processing such as preprocessing, compilation, assembly and linking.

[0237] This application also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a computer, implements the method of any of the method embodiments of this application. The computer program may be a high-level language program or an executable object program.

[0238] The computer-readable storage medium is, for example, memory 1902. Memory 1902 can be volatile memory or non-volatile memory, or memory 1902 can include both volatile and non-volatile memory. The non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. The volatile memory can be random access memory (RAM), which is used as an external cache. By way of example, but not limitation, many forms of RAM are available, such as Static RAM (SRAM), Dynamic RAM (DRAM), Synchronous DRAM (SDRAM), Double Data Rate SDRAM (DDR SDRAM), Enhanced Synchronous DRAM (ESDRAM), Synchronous Link DRAM (SLDRAM), and Direct Rambus RAM (DRRAM).

[0239] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working process and technical effects of the above-described apparatus and equipment can be referred to the corresponding processes and technical effects in the foregoing method embodiments, and will not be repeated here.

[0240] The systems, apparatuses, and methods disclosed in the embodiments provided in this application can be implemented in other ways. For example, some features of the method embodiments described above may be omitted or not performed. The apparatus embodiments described above are merely illustrative; the division of units is only a logical functional division, and in actual implementation, there may be other division methods. Multiple units or components may be combined or integrated into another system. Furthermore, the coupling between units or components can be direct or indirect, including electrical, mechanical, or other forms of connection.

[0241] The above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit it. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the idea and scope of the technical solutions of the embodiments of this application, and should all be included within the protection scope of this application.

[0242] Finally, the above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any changes or substitutions within the technical scope disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A method of image processing, characterized by, The method comprises: acquiring a first image, the first image being a two-dimensional image; determining a second image and a third image according to the first image, the second image being a two-dimensional image matching the first image in at least one two-dimensional image, and the third image being a three-dimensional image registered with the second image; determining a matching second pixel point from the second image according to a first pixel point in the first image; determining a registered third pixel point from the third image according to the second pixel point.

2. The method of claim 1, wherein, The first pixel point is a visual feature point corresponding to a pixel point of the first image, and the second pixel point is a visual feature point corresponding to a pixel point of the second image.

3. The method of claim 2, wherein, The visual feature point is one of a scale-invariant feature transform (SIFT) feature point, a speeded up robust features (SURF) feature point, or an oriented FAST and rotated BRIEF (ORB) feature point.

4. The method according to any one of claims 1 to 3, characterized in that, The determining of the registered third pixel point from the third image according to the second pixel point comprises: determining the third pixel point according to the second pixel point and a coordinate conversion model.

5. The method of claim 4, wherein, The first acquisition module and the second acquisition module meet the following requirements in an image acquisition process: synchronization and positional fixity, wherein the first acquisition module is configured to acquire the second image, the second acquisition module is configured to acquire the third image, and the coordinate conversion model has a correlation with the synchronization and the positional fixity.

6. The method according to any one of claims 1 to 5, characterized in that, The method further comprises: determining a coordinate of a first control point according to the third pixel point.

7. The method of claim 6, wherein, The determining of the coordinate of the first control point according to the third pixel point comprises: determining the first control point by clustering the third pixel point, the first control point including the first pixel point; determining the coordinate of the first control point according to the coordinate of the third pixel point.

8. The method according to any one of claims 1 to 7, characterized in that, The second image being a two-dimensional image matching the first image in at least one two-dimensional image comprises: The second image is a two-dimensional image having the highest similarity to the first image in the at least one two-dimensional image.

9. The method according to any one of claims 1 to 8, characterized in that, The acquiring of the first image comprises: acquiring the first image from a video.

10. An apparatus for image processing, characterized by The apparatus comprises an acquisition module and a processing module; The acquisition module is configured to acquire a first image, the first image being a two-dimensional image; The processing module is configured to determine a second image and a third image according to the first image, the second image being a two-dimensional image matching the first image in at least one two-dimensional image, and the third image being a three-dimensional image registered with the second image; to determine a matching second pixel point from the second image according to a first pixel point in the first image; and to determine a registered third pixel point from the third image according to the second pixel point.

11. The apparatus of claim 10, wherein, The first pixel point is a visual feature point corresponding to a pixel point of the first image, and the second pixel point is a visual feature point corresponding to a pixel point of the second image.

12. The apparatus of claim 11, wherein, The visual feature point is one of a scale-invariant feature transform (SIFT) feature point, a speeded up robust features (SURF) feature point, or an oriented FAST and rotated BRIEF (ORB) feature point.

13. The apparatus of any one of claims 10-12, wherein, The processing module is specifically configured to: determine the third pixel point according to the second pixel point and a coordinate conversion model.

14. The apparatus of claim 13, wherein, The device also includes a first acquisition module and a second acquisition module, wherein the first acquisition module and the second acquisition module meet the following requirements in the image acquisition process: synchronization and positional fixity, wherein the first acquisition module is used to acquire the second image, the second acquisition module is used to acquire the third image, and the coordinate conversion model has an association relationship with the synchronization and the positional fixity.

15. The apparatus of any one of claims 10-14, wherein, The processing module is further used to: determine the coordinates of the first control point according to the third pixel point.

16. The apparatus of claim 15, wherein, The processing module is specifically further used to: cluster the third pixel point through a clustering algorithm to determine the first control point, and the first control point includes the first pixel point; determine the coordinates of the first control point according to the coordinates of the third pixel point.

17. The apparatus of any one of claims 10-16, wherein, The second image is at least one two-dimensional image that matches the first image, including: the second image is a two-dimensional image with the highest similarity to the first image in the at least one two-dimensional image.

18. The apparatus of any one of claims 10-17, wherein, The acquisition module is specifically used to: acquire the first image from a video.

19. An electronic device, comprising: The electronic device includes a processor and a memory, the memory is used to store a computer program, and the processor is used to call and run the computer program from the memory, so that the electronic device executes the method in any one of claims 1 to 9.

20. A computer-readable storage medium, characterized in that, The computer readable storage medium stores a computer program, and when the computer program is executed by a processor, the processor executes the method in any one of claims 1 to 9.

21. A chip system, characterized by The chip system includes a memory and a processor, and the processor is configured to execute a computer program stored in the memory to implement the method in any one of claims 1 to 9.

Images