Image processing method, electronic device, and computer storage medium

Abstract

An image processing method, an electronic device, and a computer storage medium. The method comprises: acquiring a first image, wherein the first image is a two-dimensional image; determining a second image and a third image on the basis of the first image, wherein the second image is a two-dimensional image, among at least one two-dimensional image, matching the first image, and the third image is a three-dimensional image registered with the second image; on the basis of a first pixel point in the first image, determining a matched second pixel point from the second image; and on the basis of the second pixel point, determining a registered third pixel point from the third image. The method can improve the accuracy of control point extraction.

Description

Method, electronic device and computer storage medium for image processing

[0001] The present application claims priority to the Chinese patent application No. 202410891821.2, filed on July 2, 2024, and entitled "Method, electronic device and computer storage medium for image processing", the content of which is incorporated herein by reference in its entirety. TECHNICAL FIELD

[0002] The present application relates to the technical field of image processing, and in particular to a method, an electronic device and a computer storage medium for image processing. BACKGROUND

[0003] With the rapid development of video monitoring technology, a large number of monocular cameras are deployed in vertical industries such as industry, medical care and mining due to the need for daily monitoring. More than 80% of the target scenes such as steel enterprises and smart industrial parks have realized video coverage. Therefore, it is of great significance to improve the safety management level of enterprises, parks and the like by fully utilizing the existing camera resources of enterprises to locate active objects such as pedestrians and vehicles.

[0004] In the prior art, when high-precision positioning of objects such as pedestrians and vehicles is performed according to the image of a monocular camera, a visual positioning control point (hereinafter referred to as a control point) needs to be selected according to the features of the video image and the features of the three-dimensional point cloud image. However, in some scenarios (such as scenarios with weak texture, poor separation between foreground and background, etc.), the accuracy of video image registration with three-dimensional point cloud is low, and the misregistration rate is high, which affects the accuracy of control point extraction and further affects the accuracy of visual positioning. Therefore, how to improve the accuracy of control point extraction is a problem that needs to be solved at present. SUMMARY

[0005] The present application provides a method, an electronic device and a computer storage medium for image processing, which can improve the accuracy of control point extraction.

[0006] In a first aspect, a method for image processing is provided. The method can be executed by an electronic device, a module (such as a processor, a chip, or a chip system, etc.) applied in the electronic device, a logic node, a logic module or software capable of realizing all or part of the functions of the electronic device. The method comprises the following steps:

[0007] obtaining a first image, the first image being a two-dimensional image; determining a second image and a third image according to the first image, the second image being at least one two-dimensional image matched with the first image, and the third image being a three-dimensional image registered with the second image; determining a matched second pixel point from the second image according to a first pixel point in the first image; and determining a registered third pixel point from the third image according to the second pixel point.

[0008] In the application scenario of image registration, one method is to directly perform cross-modal matching of visual features of a two-dimensional image (such as the first image) and pixel points (or point cloud data) of a three-dimensional image (such as the third image), but this cross-modal matching method is often accompanied by a high false matching rate; the reason is that the first image and the third image may have great differences due to factors such as dimension, viewing angle, illumination, and scale change, and directly matching these feature points or pixel points with great differences will increase the difficulty and false matching rate of matching. Therefore, the present application proposes an image processing method, which performs same-dimension matching (which can also be understood as same-dimension registration) of a certain pixel point (such as the first pixel point) in a two-dimensional image (such as the first image) and a corresponding pixel point (such as the second pixel point) in another two-dimensional image (such as the second image). Since both images are two-dimensional, the matching is relatively easy and accurate; in addition, since the second image and the third image are registered images, each pixel point (or visual feature) in the second image has a corresponding spatial position in the third image, so the registered third pixel point can be determined from the third image according to the second pixel point; and since the first image and the second image are matched (which can also be understood as registered), and the second image and the third image are registered, the registration of the first image and the third image can be indirectly realized. This indirect registration method through an intermediate image (such as the second image) can more accurately determine the correspondence between the two-dimensional image (such as the first image) and the three-dimensional image (such as the third image), which is beneficial to the accuracy of subsequent control point extraction.

[0009] In an optional implementation, the first pixel point is a visual feature point corresponding to a pixel point of the first image, and the second pixel point is a visual feature point corresponding to a pixel point of the second image.

[0010] In some scenarios, extracting visual feature points of an image can quickly realize matching between images (such as the first image and the second image).

[0011] In a possible implementation, the visual feature point is at least one of a scale-invariant feature transform (SIFT) feature point, a speeded up robust features (SURF) feature point, or an oriented FAST and rotated BRIEF (ORB) feature point.

[0012] In different application scenarios, different feature processing algorithms (such as a SURF algorithm, a SIFT algorithm, or an ORB algorithm) can be used to extract visual features of pixel points of an image to ensure the accuracy of image matching through different visual feature points in different application scenarios.

[0013] In an optional implementation, determining the registered third pixel point from the third image according to the second pixel point includes determining the third pixel point according to the second pixel point and a coordinate conversion model.

[0014] In some scenarios, the second pixel point and the third pixel point are registered (or corresponded) through the coordinate conversion model, which has the following advantages: 1) Through the coordinate conversion model, the second pixel point and the third pixel point can be aligned in the same coordinate system, thereby realizing data fusion. This cross-dimension image information integration can provide more comprehensive and accurate data support for subsequent control point extraction operations; 2) The second pixel point and the third pixel point can be accurately registered through the coordinate conversion model, reducing registration errors caused by inconsistent coordinate systems, thereby helping to improve the accuracy and accuracy of subsequent control point extraction; 3) Since the coordinate conversion model is a technology based on mathematical algorithms and computational optimization, it can realize efficient data processing and conversion. Therefore, registering the second pixel point and the third pixel point through the coordinate conversion model can reduce the amount of calculation and data processing time, which is conducive to improving the overall efficiency of image processing.

[0015] In a possible implementation, the first acquisition module and the second acquisition module meet the following requirements during image acquisition: synchronization and position fixation, wherein the first acquisition module is configured to acquire the second image, the second acquisition module is configured to acquire the third image, and the coordinate conversion model has an association relationship with the synchronization and the position fixation.

[0016] In some scenarios, synchronization can mean that the second image and the third image are collected at the same time (or almost at the same time), that is, two images are collected at the same time or at a similar time. The position fixation can mean that the second image and the third image are collected from the same angle (i.e., the viewing angle) or the same position, that is, they collect the scene under the same viewing angle. Since the two images are collected at the same time and the same angle, the coordinate conversion model between them is fixed and does not need to be adjusted with the change of time and angle, thereby facilitating the efficiency and accuracy of registration (or alignment) between the two images. Specifically, the advantages of this synchronous and same-angle image acquisition method are as follows: 1) Since the second image and the third image are collected synchronously and at the same angle, each pixel point in the second image and the third image can be accurately corresponded. This accurate correspondence makes the measurement results more accurate, which is conducive to improving the accuracy of subsequent control point extraction. 2) Since the second image and the third image are collected from the same angle, they can provide different dimension information (such as texture information or depth information) of the same scene, thereby realizing mutual complementation of information, and further improving the situation of insufficient single-view information caused by occlusion and the like; 3) Compared with the method of asynchronous or different-angle image acquisition, the method of synchronous and same-angle image acquisition does not need to perform complex scene matching or angle conversion during preprocessing, which is conducive to reducing the amount of calculation and improving the overall efficiency of image processing.

[0017] In an optional implementation, the method further includes determining the coordinates of the first control point according to the third pixel point.

[0018] In an optional implementation, the determining of the coordinates of the first control point according to the third pixel point includes clustering the third pixel points by a clustering algorithm to determine the first control point, the first control point including the first pixel point; and determining the coordinates of the first control point according to the coordinates of the third pixel points.

[0019] In some embodiments, the user can automatically select a suitable control point (for example, the first control point) according to the requirements of visual positioning by a clustering algorithm, without manual selection, which is efficient and accurate.

[0020] In an optional implementation, the second image is a two-dimensional image that is most similar to the first image among at least one two-dimensional image, including: the second image is a two-dimensional image that is most similar to the first image among at least one two-dimensional image.

[0021] Since the second image is an image registered with the third image, determining the second image that is most similar to the first image helps to improve the accuracy of matching each pixel point in the first image with the corresponding pixel point in the third image.

[0022] In an optional implementation, the obtaining of the first image includes: obtaining the first image from a video.

[0023] In a second aspect, an image processing apparatus is provided, the apparatus including an obtaining module and a processing module.

[0024] The obtaining module is configured to obtain a first image, the first image being a two-dimensional image.

[0025] The processing module is configured to determine a second image and a third image according to the first image, the second image being a two-dimensional image that is matched with the first image among at least one two-dimensional image, and the third image being a three-dimensional image registered with the second image; and further configured to determine a matched second pixel point from the second image according to a first pixel point in the first image; and further configured to determine a registered third pixel point from the third image according to the second pixel point.

[0026] In an optional implementation, the first pixel point is a visual feature point corresponding to a pixel point of the first image, and the second pixel point is a visual feature point corresponding to a pixel point of the second image.

[0027] In an optional implementation, the visual feature point is one of a scale-invariant feature transform (SIFT) feature point, a speeded up robust features (SURF) feature point, or an oriented FAST and rotated BRIEF (ORB) feature point.

[0028] In an optional implementation, the processing module is specifically configured to determine the third pixel point according to the second pixel point and the coordinate conversion model.

[0029] In an optional implementation, the apparatus further includes a first acquisition module and a second acquisition module, wherein the first acquisition module and the second acquisition module satisfy the following requirements during image acquisition: synchronization and positional fixity, the first acquisition module is configured to acquire the second image, the second acquisition module is configured to acquire the third image, and the coordinate conversion model has an association relationship with the synchronization and the positional fixity.

[0030] In an optional implementation, the processing module is further configured to determine the coordinate of the first control point according to the third pixel point.

[0031] In an optional implementation, the processing module is specifically further configured to: cluster the third pixel points by using a clustering algorithm to determine the first control point, the first control point includes the first pixel point; and determine the coordinate of the first control point according to the coordinates of the third pixel points.

[0032] In an optional implementation, the second image is at least one two-dimensional image that matches the first image, including: the second image is a two-dimensional image with the highest similarity to the first image among at least one two-dimensional image.

[0033] In an optional implementation, the acquisition module is specifically configured to: acquire the first image from a video.

[0034] In a third aspect, an electronic device is provided, the electronic device including a processor and a memory, the memory being configured to store a computer program, and the processor being configured to invoke and run the computer program from the memory, so that the electronic device executes the method of any one of the first aspect.

[0035] In a fourth aspect, a computer readable storage medium is provided, the computer readable storage medium storing a computer program, when the computer program is executed by a processor, the processor executes the method of any one of the first aspect.

[0036] In a fifth aspect, a computer program product is provided, the computer program product including: computer program code, when the computer program code is executed by an electronic device, the electronic device executes the method of any one of the first aspect.

[0037] In a sixth aspect, a chip system is provided, the chip system including a memory and a processor, the processor being configured to execute a computer program stored in the memory to implement the method of any one of the first aspect.

[0038] The beneficial effects of the technical solutions of the second aspect, the third aspect, the fourth aspect, the fifth aspect and the sixth aspect of the present application are the same as those of the technical solution of the first aspect, and will not be repeated here. BRIEF DESCRIPTION OF DRAWINGS

[0039] FIG. 1 is a structural schematic diagram of an electronic device 100 provided by an embodiment of the present application;

[0040] FIG. 2 is a software structure schematic diagram of an electronic device 100 provided by an embodiment of the present application;

[0041] FIG. 3 is a software architecture schematic diagram of an electronic device suitable for the present application provided by an embodiment of the present application;

[0042] FIG. 4 is an application system architecture schematic diagram suitable for the present application provided by an embodiment of the present application;

[0043] FIGS. 5A and 5B show flow schematic diagrams of two image processing methods;

[0044] FIG. 6 is a flow schematic diagram of an image processing method 600 provided by an embodiment of the present application;

[0045] FIG. 7 is an application scenario schematic diagram provided by an embodiment of the present application;

[0046] FIG. 8 is an image editing application scenario schematic diagram provided by an embodiment of the present application;

[0047] FIG. 9 is another image editing application scenario schematic diagram provided by an embodiment of the present application;

[0048] FIG. 10 is still another image editing application scenario schematic diagram provided by an embodiment of the present application;

[0049] FIG. 11 is still another image editing application scenario schematic diagram provided by an embodiment of the present application;

[0050] FIG. 12 is still another image editing application scenario schematic diagram provided by an embodiment of the present application;

[0051] FIG. 13 is still another image editing application scenario schematic diagram provided by an embodiment of the present application;

[0052] FIG. 14 is still another image editing application scenario schematic diagram provided by an embodiment of the present application;

[0053] FIG. 15 is still another image editing application scenario schematic diagram provided by an embodiment of the present application;

[0054] FIG. 16 is still another image editing application scenario schematic diagram provided by an embodiment of the present application;

[0055] FIG. 17 is a schematic diagram of another image editing application scenario according to an embodiment of the present application;

[0056] FIG. 18 is a schematic diagram of an image processing device 1800 according to an embodiment of the present application;

[0057] FIG. 19 is a schematic diagram of an electronic device 1900 according to an embodiment of the present application. DETAILED DESCRIPTION

[0058] In order to clearly describe the technical solutions of the embodiments of the present application, in the embodiments of the present application, the terms "first", "second", etc. are used to distinguish the same or similar items or items with basically the same function and effect. For example, the first terminal device and the second terminal device are only used to distinguish different terminal devices, and do not limit the order. Those skilled in the art can understand that the terms "first", "second", etc. do not limit the number and execution order, and the terms "first", "second", etc. do not necessarily mean different.

[0059] It should be noted that in the present application, the words "exemplary" or "for example" are used to mean serving as an example, instance, or illustration. Any embodiment or design solution described as "exemplary" or "for example" in the present application should not be interpreted as being more preferred or having more advantages than other embodiments or design solutions. Rather, the use of the words "exemplary" or "for example" is intended to present the relevant concept in a specific manner.

[0060] In the present application, "at least one" means one or more, and "multiple" means two or more. The association relationship between the associated objects is described as "and / or", which means that there can be three relationships, for example, A and / or B, which means that A exists alone, A and B exist together, and B exists alone, where A and B can be singular or plural. The character " / " generally represents an "or" relationship between the associated objects. "At least one of the following" or similar expressions means any combination of these items, including any combination of single or multiple items. For example, at least one of a, b, or c, can mean a, b, c, a-b, a-c, b-c, or a-b-c, where a, b, and c can be single or multiple.

[0061] In order to facilitate understanding of the present application, some technical terms related to the present application are explained below.

[0062] (1) Scale-invariant feature transform (SIFT) algorithm

[0063] SIFT algorithm is a kind of machine vision algorithm, which is used to detect and describe local feature points in an image; it finds extreme points in spatial scale and extracts the position, scale and rotation invariant number. SIFT can keep local feature points invariant in the case of image scaling and rotation. In addition, by matching SIFT feature points in different images, accurate image registration can be achieved.

[0064] (2) Oriented FAST and Rotated BRIEF (ORB) algorithm

[0065] ORB algorithm is a feature point extraction algorithm widely used in computer vision and image processing field, which can be used to quickly create feature vectors for key points in an image, and these feature vectors can be used to identify objects in the image; the ORB algorithm combines oriented features from accelerated segment test (oFAST) algorithm and rotated binary robust independent elementary features (rBRIEF) algorithm to form oriented FAST and rotated BRIEF algorithm; in the ORB algorithm, the oFAST algorithm is first used to extract feature points in the image, and a direction is defined for each feature point to achieve rotation invariance of the feature points; then, the rBRIEF algorithm is used to describe each feature point to generate a binary string as the descriptor of the feature point.

[0066] (3) Speed up Robust Features (SURF)

[0067] SURF algorithm is a robust local feature point detection and description algorithm; SURF algorithm includes three parts: extraction of local feature points, description of feature points and matching of feature points.

[0068] (4) Image control point

[0069] Image control point, also known as visual positioning control point or visual control point or control point, refers to the reference point selected on the image to establish a geometric transformation function. For example, in image geometric correction, projection transformation and image registration, the reference point selected on the image to establish a geometric transformation function.

[0070] (5) Three-dimensional point cloud image

[0071] A three-dimensional point cloud image (or point cloud image or point cloud image) can also be understood as three-dimensional point cloud data, which can refer to a set of points in a three-dimensional coordinate system; each point can be represented by a three-dimensional coordinate, which can carry some information about the attributes of the point, such as color, reflectivity, intensity, etc. In general, point cloud images (i.e. point cloud data) can be obtained by laser scanners (also known as lidar), cameras, three-dimensional scanners and other devices for three-dimensional modeling, scene reconstruction, virtual reality and augmented reality applications. It should be noted that the point cloud image can directly represent the shape, surface and texture of the object in space. The processing method of the point cloud image includes but is not limited to filtering, registration, segmentation, reconstruction, identification and classification.

[0072] (6) K-Nearest Neighbors (KNN) algorithm

[0073] The KNN algorithm is a basic classification and regression method that can classify by measuring the distance between different feature values; the three basic elements of K-Nearest Neighbors are the selection of k value, distance measurement and classification decision rule.

[0074] (7) Coordinate conversion model

[0075] The coordinate conversion model is usually used to establish the relationship between the source coordinates and the target coordinates, and different coordinate conversion models are used to realize the conversion between the source coordinates and the target coordinates for different types of coordinates.

[0076] (8) Line segment detector (LSD)

[0077] The LSD algorithm is a line segment detection algorithm, which accurately detects the position and direction of a straight line segment in an image, and has high precision and high robustness. The LSD algorithm is based on the detection and analysis of image edges, such as extracting a straight line segment by using the direction and intensity information of edge pixels. The main idea is to construct a global branch and a simulated partial branch graph on the edge pixels, and extract a straight line segment based on the graph. It should be noted that the LSD algorithm mainly includes the following steps: edge detection, edge pixel sorting, global branch, simulated partial branch, and non-maximum suppression. The edge detection refers to extracting the edges of an image using an edge detection algorithm. The edge pixel sorting refers to sorting the edge pixels in descending order of intensity. The global branch refers to constructing a global branch graph on the edge pixels, that is, starting from the edge pixel with the highest intensity, calculating the direction difference of the adjacent edge pixels, and constructing a global branch graph according to the direction difference. The simulated partial branch refers to simulating a partial branch graph based on the global branch graph, that is, starting from the strongest global branch, calculating the simulated partial branch graph according to the direction and intensity information of the edge pixels. The non-maximum suppression refers to performing a non-maximum suppression operation according to the direction and intensity information of the edge pixels to obtain the final straight line segment.

[0078] The technical problems to be solved in the present application are introduced below in combination with actual application scenarios.

[0079] In some scenarios, when a point cloud image and an image (or an optical image) are registered, the LSD algorithm can be used to extract the line features of the texture image, and the vanishing point detection algorithm can be used to extract the image vanishing point. For the extraction of line features in the laser point cloud, an adaptive segmentation method is used to extract the line features, and a Gaussian mixture model clustering algorithm is used to cluster the extracted line features. The image line features and the point cloud line features extracted by the above steps are registered. Since the accuracy of this method depends on the number of line features in the scene, the misregistration rate is high in scenes with weak texture and high repetition, which further affects the accuracy of the control point extraction.

[0080] In yet some scenarios, when the point cloud image is registered with the image (or optical image), the registration can be achieved based on a semantic segmentation manner, for example, a depth image is first generated by using the elevation information of the point cloud image, wherein the gray value of each pixel point in the depth image represents the distance of a certain point in the scene from the camera, so as to reflect the geometric shape and height of the surface of the ground object; then a deep learning algorithm is used to segment and extract the building surface in the depth image and the optical image of the laser point cloud, and the minimum circumscribed rectangle of the building contour in the two types of images is constructed respectively; finally, based on the aspect ratio constraint of the minimum circumscribed rectangle, the center points of the homonymous rectangles are found by using the principle of similar triangles, so as to obtain the homonymous points, and the transformation model parameters are obtained by the least square method through the mapping conversion model, so as to realize the registration of the point cloud image and the optical image; since the accuracy of the registration method depends on the minimum circumscribed rectangle of the building contour in the scene, and the semantic perception accuracy is high, in some complex application scenarios (for example, the scene in which the foreground and background in the image are not easy to separate), the semantic segmentation effect is poor and the misregistration rate is high, thereby affecting the accuracy of the control point extraction.

[0081] It is found through the above two scenarios that, whether in the line feature extraction scenario or in the semantic segmentation scenario, there may be problems of low registration accuracy of the point cloud image and the two-dimensional image, high misregistration rate, and thus affecting the accuracy of the control point extraction. Therefore, the present application proposes an image processing method, which can improve the accuracy of the control point extraction and is widely applicable.

[0082] It should be noted that, in the image processing method, the electronic device uses the second image as a bridge to realize high-precision matching of the feature points of the video image (such as the feature points of the first image) and the point cloud image (such as the third image), thereby improving the accuracy of the control point extraction. The descriptions about the first image, the second image and the third image can be referred to the relevant descriptions below, and will not be described here in detail.

[0083] In order to better understand the embodiments of the present application, the structure of the electronic device involved in the embodiments of the present application is introduced as follows.

[0084] The electronic device involved in the embodiments of the present application can include but is not limited to at least one of a foldable electronic device, a tablet computer, a desktop computer, a laptop computer, a handheld computer, a notebook computer, an ultra-mobile personal computer (UMPC), and an artificial intelligence (AI) device. The specific type of the electronic device is not specially limited in the embodiments of the present application.

[0085] FIG. 1 shows a structural schematic diagram of an electronic device 100 applicable to the above-mentioned image processing method.

[0086] The electronic device 100 can include a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (USB) joint 130, a charge management module 140, a power management module 141, a battery 142, an antenna 1, an antenna 2, a mobile communication module 150, a wireless communication module 160, a display screen 170, a camera 171, and the like.

[0087] It can be understood that the structure shown in the embodiments of the present application does not constitute a specific limitation on the electronic device 100. In other embodiments of the present application, the electronic device 100 can include more or fewer components than Figure 1, or combine certain components, or split certain components, or different component arrangements. The components in Figure 1 can be implemented in hardware, software, or a combination of software and hardware.

[0088] The processor 110 can include one or more processing units, for example: the processor 110 can include an application processor (AP), a modem processor, a graphics processing unit (GPU), an image signal processor (ISP), a controller, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural-network processing unit (NPU), etc. Different processing units can be independent devices, or can be integrated into one or more processors.

[0089] The processor can generate operation control signals according to instruction operation codes and timing signals, and complete the control of fetching and executing instructions.

[0090] The memory in the processor 110 can also be provided for storing instructions and data. In some embodiments, the memory in the processor 110 can be a cache memory. The memory can save instructions or data that have been used or used frequently by the processor 110. If the processor 110 needs to use the instructions or data, it can directly call from the memory. Avoiding repeated access, reducing the waiting time of the processor 110, thus improving the efficiency of the system.

[0091] In some embodiments, the processor 110 can include one or more interfaces. The interfaces can include an inter-integrated circuit (I2C) interface, an inter-integrated circuit sound (I2S) interface, a pulse code modulation (PCM) interface, a universal asynchronous receiver / transmitter (UART) interface, a mobile industry processor interface (MIPI), a general-purpose input / output (GPIO) interface, a subscriber identity module (SIM) interface, and / or a universal serial bus (USB) interface, etc. The processor 110 can connect modules such as a wireless communication module, a display (screen), etc. through at least one of the above interfaces.

[0092] It can be understood that the interface connection relationship between the modules shown in the embodiments of the present application is only illustrative and does not constitute a structural limitation of the electronic device 100. In some other embodiments of the present application, the electronic device 100 can also use different interface connection modes or a combination of multiple interface connection modes in the above embodiments.

[0093] The USB connector 130 is an interface that conforms to the USB standard specification and can be used to connect the electronic device 100 and a peripheral device. Specifically, the USB connector 130 can be a Mini USB connector, a Micro USB connector, a USB Type C connector, etc. The USB connector 130 can be used to connect a charger to charge the electronic device 100, and can also be used to connect other terminal devices to transmit data between the electronic device 100 and the other terminal devices. The connector can also be used to connect other terminal devices, such as VR devices, etc. In some embodiments, the standard specification of the universal serial bus can be USB 1.x, USB 2.0, USB 3.x, and USB 4.

[0094] The charging management module 140 is configured to receive charging input from a charger. The charger can be a wireless charger or a wired charger. In some embodiments of wired charging, the charging management module 140 can receive charging input from a wired charger through the USB connector 130. In some embodiments of wireless charging, the charging management module 140 can receive wireless charging input through a wireless charging coil of the electronic device 100. The charging management module 140 can charge the battery 142 and supply power to the terminal device through the power management module 141.

[0095] The power management module 141 is configured to connect the battery 142, the charging management module 140, and the processor 110. The power management module 141 receives input from the battery 142 and / or the charging management module 140 to supply power to the processor 110, the internal memory 121, the display screen 170, the wireless communication module 160, and the like. The power management module 141 can also be configured to monitor parameters such as battery capacity, battery cycle count, battery health status (leakage, impedance), and the like. In some other embodiments, the power management module 141 can also be disposed in the processor 110. In some other embodiments, the power management module 141 and the charging management module 140 can also be disposed in the same device.

[0096] The wireless communication function of the electronic device 100 can be implemented through the antenna 1, the antenna 2, the mobile communication module 150, the wireless communication module 160, the modem processor, and the baseband processor, and the like.

[0097] The antenna 1 and the antenna 2 are configured to transmit and receive electromagnetic wave signals. Each antenna in the electronic device 100 can be configured to cover a single or multiple communication frequency bands. Different antennas can also be multiplexed to improve the utilization rate of the antennas. For example, the antenna 1 can be multiplexed as a diversity antenna for a wireless local area network. In some other embodiments, the antennas can be used in combination with a tuning switch.

[0098] The mobile communication module 150 can provide a solution for wireless communication including 2G / 3G / 4G / 5G and the like applied to the electronic device 100. The mobile communication module 150 can include at least one filter, a switch, a power amplifier, a low noise amplifier (LNA), and the like. The mobile communication module 150 can receive electromagnetic waves from the antenna 1, filter, amplify, and the like the received electromagnetic waves, and transmit the processed electromagnetic waves to the modem processor for demodulation. The mobile communication module 150 can also amplify signals modulated by the modem processor and radiate the signals as electromagnetic waves through the antenna 1. In some embodiments, at least part of the function modules of the mobile communication module 150 can be disposed in the processor 110. In some embodiments, at least part of the function modules of the mobile communication module 150 and at least part of the modules of the processor 110 can be disposed in the same device.

[0099] The wireless communication module 160 can provide a wireless communication solution applied to the electronic device 100, including wireless local area networks (WLAN) such as wireless fidelity (Wi-Fi) networks, Bluetooth (BT), Bluetooth low energy (BLE), ultra wide band (UWB), global navigation satellite system (GNSS), frequency modulation (FM), near field communication (NFC), infrared (IR) technology, and the like. The wireless communication module 160 can be one or more devices that integrate at least one communication processing module. The wireless communication module 160 receives electromagnetic waves via the antenna 2, frequency-modulates and filters the electromagnetic wave signals, and transmits the processed signals to the processor 110. The wireless communication module 160 can also receive signals to be transmitted from the processor 110, frequency-modulate them, amplify them, and radiate them as electromagnetic waves via the antenna 2.

[0100] In some embodiments, the antenna 1 and the mobile communication module 150 of the electronic device 100 are coupled, and the antenna 2 and the wireless communication module 160 are coupled, so that the electronic device 100 can communicate with a network and other terminal devices through wireless communication technology. The wireless communication technology can include global system for mobile communications (GSM), general packet radio service (GPRS), code division multiple access (CDMA), wideband code division multiple access (WCDMA), time-division code division multiple access (TD-SCDMA), long term evolution (LTE), BT, GNSS, WLAN, NFC, FM, and / or IR technology, etc. The GNSS can include a global positioning system (GPS), a global navigation satellite system (GLONASS), a beidu navigation satellite system (BDS), a quasi-zenith satellite system (QZSS), and / or a satellite-based augmentation system (SBAS).

[0101] The electronic device 100 can implement a display function through a GPU, a display screen 170, and an application processor, etc. The GPU is a microprocessor for image processing, connected to the display screen 170 and the application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering. The processor 110 can include one or more GPUs that execute program instructions to generate or change display information.

[0102] The external memory interface 120 can be used to connect an external memory card, such as a Micro SD card, to expand the storage capacity of the electronic device 100. The external memory card communicates with the processor 110 through the external memory interface 120 to implement a data storage function. For example, files such as videos, pictures, etc. are saved in the external memory card. Or files such as videos, pictures, etc. are transmitted from the electronic device to the external memory card.

[0103] The internal memory 121 can be configured to store computer-executable program code including instructions. The internal memory 121 can include a program storage area and a data storage area. The program storage area can store an operating system, at least one application program (e.g., a video playing function, a picture playing function, etc.) required by at least one function, etc. The data storage area can store data (e.g., video data, etc.) created during use of the electronic device 100, etc. In addition, the internal memory 121 can include a high-speed random access memory, and can further include a nonvolatile memory such as at least one of a magnetic disk storage device, a flash memory device, a universal flash storage (UFS), etc. The processor 110 performs various functions, methods, or data processing of the electronic device 100 by executing instructions stored in the internal memory 121 and / or instructions stored in a memory disposed in the processor.

[0104] The display screen 170 is configured to display data such as videos, pictures, picture processing results, etc. For example, the display screen 170 can be configured to display pictures and videos captured by the camera 171. The display screen 170 includes a display panel. The display panel can be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode (AMOLED), a flex light-emitting diode (FLED), a Miniled, a MicroLed, a Micro-oLed, a quantum dot light emitting diode (QLED), etc. In some embodiments, the electronic device 100 can include one or more display screens 170.

[0105] The camera 171 is configured to capture images or videos. An object projects an optical image through a lens to a photosensitive element. The photosensitive element can be a charge coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor. The photosensitive element converts the optical signal into an electrical signal, and then transmits the electrical signal to an ISP to convert into a digital image signal. The ISP outputs the digital image signal to a DSP for processing. The DSP converts the digital image signal into a standard image signal in RGB, YUV, or the like. In some embodiments, the electronic device 100 can include one or N cameras 171, where N is a positive integer greater than 1.

[0106] It can be understood that the structure shown in the embodiments of the present application does not constitute a specific limitation on the electronic device 100. In some other embodiments of the present application, the electronic device 100 can further include more or fewer components than those in FIG. 1, or combine some components, or split some components, or different arrangement of components. The components in FIG. 1 can be implemented in hardware, software, or a combination of software and hardware.

[0107] The software system of the electronic device 100 described above can adopt a layered architecture or a service architecture, and the embodiments of the present application take the Android operating system with a layered architecture as an example to exemplarily illustrate the software architecture of the electronic device 100. It should be understood that the scheme provided by the present application can also be applied to other types of operating systems such as the Harmony system, the Apple operating system, the Windows operating system, and the like.

[0108] FIG. 2 shows a software architecture diagram of the electronic device 100 provided by the embodiments of the present application. As shown in FIG. 2, the layered architecture of the electronic device 100 divides the software into several layers, each layer has a clear role and division of labor. The layers communicate with each other through a software interface. In some embodiments, the software architecture includes, from top to bottom, an application (APP) layer, an application framework (FW) layer, an Android runtime (ART) and a native C / C++ library, a hardware abstraction layer (HAL), and a kernel layer.

[0109] The application layer, also referred to as the application layer, can include a series of application packages. For example, the application layer package can include a gallery application, a video application, an image editing application, and a settings application, etc. When the above application package is run, each service module provided by the application framework layer can be accessed through an application programming interface (application programming interface, API), and the corresponding intelligent business is executed.

[0110] The application framework layer (framework, FWK) provides an application programming interface (application programming interface, API) and a programming framework for the application of the application layer. The application framework layer includes some pre-defined functions. As shown in FIG. 2, the application framework layer can include a window manager, a content provider, a view system, a resource manager, a notification manager, an activity manager, an input manager, etc. Among them, the window manager is used to manage all windows in the system; the content provider is used to store and obtain data (such as video, picture, etc.), and makes these data can be accessed by the application; the view system includes visual controls, for example, controls for displaying videos, controls for displaying pictures, etc. The display interface can be composed of one or more views. For example, the display interface including the short message notification icon can include a view for displaying text and a view for displaying pictures; the resource manager provides various resources for the application, such as picture, video file, etc.; the notification manager is responsible for managing the prompt information of the top status bar of the mobile phone.

[0111] The Android runtime includes the core library and the Android runtime. The Android runtime is responsible for converting the source code into machine code. The Android runtime mainly includes the ahead of time (AOT) compilation technology and the just in time (JIT) compilation technology.

[0112] The core library is mainly used to provide the functions of the basic Java class library, such as the basic data structure, mathematics, IO, tools, database, network, etc. The core library provides an API for the user to develop the Android application.

[0113] The native C / C++ library can include a plurality of function modules. For example: a surface manager, a media framework, etc. Among them, the surface manager is used to manage the display subsystem, and provides the fusion of 2D and 3D layers for multiple applications. The media framework supports a variety of commonly used audio, video format playback and recording, and static image files, etc.

[0114] The hardware abstraction layer runs in a user space, encapsulates kernel layer drivers, and provides an invocation interface to an upper layer; the hardware abstraction layer includes a display module, a Bluetooth module, a Wifi module, and the like.

[0115] The kernel layer is a layer between hardware and software. The kernel layer at least includes display drivers, monitoring drivers, Wifi drivers, and the like to drive a display screen, a camera, Bluetooth, and Wifi to work.

[0116] Next, taking an electronic device with the structure shown in FIG. 1 and FIG. 2 as an example, and in combination with a software architecture diagram of the electronic device shown in FIG. 3, the overall flow of the method for the electronic device to perform the image processing is exemplarily described.

[0117] In FIG. 3, an image editing application runs on an operating system of the electronic device. In the image editing application scenario, a human-computer interaction module is used to receive a video image, a point cloud image, and the like input from a user, and to show the user a result of image processing; the human-computer interaction module includes an image display module and a control point display module, wherein the image display module is used to display a video image input by the user and a processed image; the control point display module is used to display a visual positioning control point (i.e., a control point) automatically extracted by a control point extraction module, so as to facilitate the user to view a distribution of the control points; in addition, the human-computer interaction module can send the video image input by the user to a control point processing module, so as to facilitate the control point processing module to perform image processing on the video image input by the user; correspondingly, the control point processing module can send a processing result of the video image to the human-computer interaction module, so as to facilitate the human-computer interaction module to display the processing result through the control point display module.

[0118] The control point processing module includes an image similarity retrieval module, a feature extraction and matching module, a map registration module, a cross-modal feature matching module, and a control point extraction module; the functions of these modules are as follows:

[0119] The image similarity retrieval module is used to retrieve, from a basic image library (also referred to as a basic video library), an image (or a video) with the highest similarity to a video image input by a user end, wherein the image with the highest similarity can also be referred to as a most associated basic image (or a most associated basic video). It should be noted that the video image and the most associated basic image are both two-dimensional (2D) images; the determination manner of the image with the highest similarity can be referred to relevant description below, which is not described herein again.

[0120] The feature extraction and matching module is used to extract visual features (such as point features, etc.) of the video image and the most associated basic image, and to establish a matching relationship according to the similarity of the visual features and imaging constraints.

[0121] The map registration module is configured to establish a corresponding relationship between each visual feature on the most relevant base image and each point cloud data in the point cloud image according to a three-dimensional (3D) point cloud image and a coordinate conversion model. Each point cloud data can be represented by a 3D coordinate.

[0122] The cross-modal feature matching module is configured to determine (or infer) the 3D coordinate corresponding to the visual feature of the video image according to the relationship between the visual feature of the video image and the visual feature of the most relevant base image, and the corresponding relationship between the visual feature of the most relevant base image and the 3D coordinate of the point cloud data of the point cloud image.

[0123] The control point extraction module is configured to extract a control point (i.e., a visual positioning control point) meeting a requirement from the visual feature of the video image according to a certain rule (e.g., a KNN algorithm).

[0124] After the control point processing module obtains the control point, the control point processing module can send a control point display request to the driving module. The driving module generates a control point display instruction according to the control point display request, and sends the control point display instruction to the display screen. The display screen displays the control point in the video image according to the control point display instruction, so that the user can view the distribution of the control point on the video image. For details, refer to the interface embodiment below, which will not be described here.

[0125] The software architecture of the electronic device is introduced above, and the application system architecture suitable for the present application is introduced below in combination with FIG. 4. In the application system architecture, the server can interact with the user through the human-computer interaction interface. For example, the user can input a video image (for example, the first image below) through the human-computer interaction interface, the server obtains the video image from the human-computer interaction interface, processes the video image, obtains control point data, and displays the control point data to the user through the human-computer interaction interface; the user can view the distribution of the control point data on the video image on the human-computer interaction interface to determine whether the control point data extracted by the server meets the requirements; wherein the server includes a basic image library (also referred to as a basic image database) and a point cloud image library (also referred to as a point cloud database); after the server receives the video image, the image similarity retrieval function is started, and the image with the highest similarity to the video image is retrieved from the basic image library as the most associated basic image (for example, the second image below); then the corresponding relationship between the video image and the corresponding point cloud image (for example, the third image below) in the point cloud image library is determined according to the most associated basic image; thereafter, according to the corresponding relationship between the video image and the most associated basic image and the corresponding relationship between the most associated basic image and the point cloud image, the one-to-one correspondence between each visual feature in the video image and each point cloud data in the point cloud image is deduced, thereby obtaining the 3D coordinates of each visual feature in the video image; finally, the control point extraction function is started to extract the visual feature meeting the requirements from each visual feature in the video image as the control point, and the control point data is displayed to the user through the human-computer interaction interface. It should be noted that since each visual feature in the video image has been associated with 3D coordinates, each control point extracted also has 3D coordinates.

[0126] The above illustrates an application system architecture suitable for the present application and a software architecture of an electronic device. In actual image processing scenarios, the application system architecture and the software architecture of the electronic device can be the same as the above-mentioned architecture, or can be partially the same (for example, improvement is made on the basis of the above-mentioned structure), and the present application does not limit this. Before the image processing method proposed by the present application is introduced in detail below, the overall flow of the image processing method proposed by the present application is exemplarily described in combination with FIGS. 5A and 5B.

[0127] FIGS. 5A and 5B show flowcharts of two image processing methods. The difference between the two figures is that FIG. 5A shows the overall processing flow of the method, and FIG. 5B shows the data flow relationship between the images involved in different processing steps of the method. It should be noted that the execution subject in the image processing method of FIGS. 5A and 5B can be an electronic device, a GPU in the electronic device, or other software or hardware functional modules in the electronic device. The following illustrates the electronic device as the execution subject of the image processing method, which does not limit the present application.

[0128] The flow steps of the image processing method shown in FIG. 5A are as follows:

[0129] Step 501: Obtain a video image. The electronic device can obtain a frame of video image (for example, the first image below) from the monitoring video, as shown in 501 of FIG. 5B.

[0130] Step 502: Obtain the most associated basic image. The electronic device performs image similarity search in the basic image library to obtain the image with the highest similarity to the video image as the most associated basic image (for example, the second image below), as shown in 501 of FIG. 5B.

[0131] Step 503: Extract the visual features of the video image and the basic associated image. The electronic device extracts the visual features (for example, point features, etc.) of the video image and the most associated basic image respectively, and matches each visual feature in the video image with each visual feature in the most associated basic image (as shown in 502 of FIG. 5B), to establish an association relationship of feature matching, as shown in 501 of FIG. 5B.

[0132] Step 504: Register the basic associated image and the point cloud image. The electronic device determines the correspondence (also referred to as matching relationship) between each visual feature in the most associated basic image and each point cloud data in the point cloud image (for example, the third image below) through the coordinate conversion model (as shown in 503 of FIG. 5B), which can also be understood as registering the most associated basic image and the point cloud image through the coordinate conversion model, as shown in 504 of FIG. 5B.

[0133] Step 505: Cross-modal feature matching. The cross-modal feature matching can be understood as follows: according to the correspondence between the visual features of the video image and the visual features of the most associated basic image in step 502, and the correspondence between the visual features of the most associated basic image and each point cloud data in the point cloud image in step 503, the one-to-one correspondence between each visual feature in the video image and each point cloud data in the point cloud image is deduced (as shown in 505 of FIG. 5B), so as to deduce the 3D coordinates of each visual feature in the video image, as shown in 506 of FIG. 5B.

[0134] Step 506: extracting control points. The electronic device extracts part of the visual features (for example, the set R below) from the visual features of the video image whose 3D coordinates have been matched as control points according to certain principles (for example, the principle of uniformity, etc.) according to the needs of visual positioning, so as to obtain a control point set, wherein the electronic device can display the control point set on at least one of the video image, the most relevant base image or the point cloud image respectively.

[0135] For example, the electronic device can display each control point in the control point set on the video image, as shown as 507 in FIG. 5B; for another example, the electronic device can display each control point in the control point set on the most relevant base image, as shown as 508 in FIG. 5B; for yet another example, the electronic device can display each control point in the control point set on the point cloud image, as shown as 509 in FIG. 5B.

[0136] The above outlines the related processes of the image processing method provided by the present application in combination with software architecture, application system architecture, etc. The image processing method will be described in detail in combination with different embodiments (for example, interface embodiments, etc.).

[0137] It should be noted that the execution subject of the image processing method can be an electronic device, a chip, a chip system or a processor applied to the electronic device, or a logic module or software capable of realizing all or part of the functions of the electronic device.

[0138] The following embodiments take the electronic device as an example to describe the execution subject of the image processing method 600. As shown in FIG. 6, the flow step schematic diagram of the image processing method 600 provided by the embodiments of the present application is shown, which includes S601 to S604, and the steps will be described in detail below.

[0139] S601, the electronic device acquires a first image, and the first image is a two-dimensional image.

[0140] It should be noted that the first image can also be referred to as the first video; the first image can be an image frame acquired by the electronic device from a local video, an image frame extracted from a video stream received from another device (for example, a monitoring device, etc.), or an image manually input by a user, and the present application does not limit the manner in which the electronic device acquires the first image.

[0141] For example, the electronic device locally stores a monitoring video, and the electronic device can extract an image frame from the monitoring video as the first image.

[0142] In an optional implementation manner, the electronic device can also acquire the first image from the video.

[0143] For example, the indoor monitoring device (such as a monitoring camera) and the electronic device can communicate with each other, and the electronic device can obtain a video from the monitoring device, and extract an image frame from the video as the first image.

[0144] In S602, the electronic device determines a second image and a third image according to the first image, the second image is at least one two-dimensional image that matches the first image, and the third image is a three-dimensional image that is registered with the second image.

[0145] The second image is an image that matches the first image, also known as the most associated base image or the most associated base image, and the third image is a 3D point cloud data. It should be noted that in some scenarios, the first image, the second image and the third image are images of different perspectives in the same shooting scene; the matching of the first image and the second image can be measured by similarity calculation, for example, the similarity threshold is 90%, when the similarity of the first image and the second image is greater than or equal to 90%, it means that the first image and the second image match (or similar); when the similarity of the first image and the second image is less than 90%, it means that the first image and the second image do not match (or not similar); the electronic device can select the image with the highest similarity to the first image from the base image library as the second image through similarity calculation.

[0146] It should be noted that in some scenarios, there may be some alternative descriptions about "the matching of the first image and the second image", for example, when the similarity of the first image and the second image is greater than or equal to the similarity threshold, it can also be considered that the first image and the second image are registered (or aligned or corresponding). When the similarity of the first image and the second image is less than the similarity threshold, it can also be considered that the first image and the second image are not registered (or not aligned or not corresponding).

[0147] In some embodiments, the second image is at least one two-dimensional image that has the highest similarity to the first image.

[0148] For example, the base image library stores at least one two-dimensional image, and the electronic device can retrieve the two-dimensional image with the highest similarity to the first image in the base image library according to the first image, for example, the electronic device calculates that the similarity values of three base images to the first image are all greater than the similarity threshold, at this time, the electronic device can sort the three similarity values, and the base image with the largest similarity value calculated with the first image is taken as the second image, thereby obtaining the second image that matches the first image.

[0149] After the electronic device determines the second image matching the first image, the visual features of the first image and the second image can be extracted respectively, and a matching relationship between the visual features in the first image and the visual features in the second image can be established.

[0150] It should be noted that the visual features described above can refer to point features or other features (such as line features, surface features, etc.) that can be processed as point features, and the present application does not limit the same.

[0151] In some scenarios, the third image is a three-dimensional image registered with the second image, which can also be described as the third image is a three-dimensional image matching (or aligned or corresponding) with the second image; in some embodiments, since the coordinate conversion model between the acquisition device of the second image and the acquisition device of the third image is known, the second image can establish an association (or correspondence) between the visual features through the coordinate conversion model and the third image, to achieve one-to-one correspondence between the visual features of the second image and the visual features of the third image, in other words, the second image and the third image can be understood as pre-registered (or pre-aligned or pre-matched or pre-calibrated); for the explanation of the coordinate conversion model, reference can be made to the description in the related embodiments below, which will not be described here in detail.

[0152] Since the second image is an image registered with the third image, the higher the accuracy of the electronic device retrieving the second image matching the first image through the similarity algorithm, the more conducive to improving the accuracy of matching the visual features in the first image with the corresponding visual features in the third image.

[0153] For example, as shown in (a) of FIG. 7, taking the image acquisition device 700 acquiring the picture of scene A as an example, the camera 701 and the lidar device 702 in the image acquisition device 700 acquire picture images synchronously and at the same angle; the image acquisition device 700 can enable the camera 701 to synchronously shoot the image of scene A to obtain at least one two-dimensional image 703; the at least one two-dimensional image 703 can be stored in a local folder or in a basic image library deployed on a remote cloud server 705; in some scenarios, the second image can be one of the at least one two-dimensional image 703 acquired by the camera 703; the basic image library (or the local folder) stores the at least one two-dimensional image 703 (or at least one image 703); at the same time, the lidar device 702 is enabled to acquire the 3D image (i.e., 3D point cloud data) of scene A to obtain a set of 3D images 704; the set of 3D images 704 can be stored in a local folder or in a point cloud image library deployed on a remote cloud server 705; similarly, the monitoring device 706 can also acquire video images of scene A at different time periods from different angles to obtain a video 707; the monitoring device 706 can send the video 707 to an electronic device 708; the electronic device 708 can obtain a frame image from the video 707 as a first image and perform image similarity retrieval in the basic image library (or the local folder) to retrieve a second image matching the first image; after determining the second image, the electronic device 708 can determine a third image registered (or matched) with the second image from the point cloud image library (or the local folder) according to the second image.

[0154] It should be noted that since the coordinate conversion model between the image shot by the camera 701 in the image acquisition device 700 and the point cloud image shot by the lidar device 702 is known, the visual features (such as point features) of the second image can be one-to-one corresponding to the point cloud data in the third image through the known coordinate conversion model, in other words, the second image and the third image are registered (or aligned), which can be specifically referred to the related explanation of the coordinate model conversion below, which will not be described here. The electronic device 708 can determine the second image matching itself from the basic image library according to the first image, and determine the third image matching itself from the point cloud image library according to the second image and the coordinate conversion model, and finally, taking the second image as a bridge, according to the corresponding relationship between the first image and the second image and the corresponding relationship between the second image and the third image, the one-to-one corresponding relationship between each visual feature in the first image and each point cloud data in the third image is obtained, so as to obtain the 3D coordinates of each visual feature in the first image.

[0155] For example, as shown in (b) of FIG. 7, since the first image and the second image are matching images, the electronic device 708 can determine the correspondence between each visual feature (e.g., feature point S1) in the first image and each visual feature (e.g., feature point S2) in the second image; since the second image and the third image are registered, the electronic device 708 can also determine the correspondence between each visual feature (e.g., feature point S2) in the second image and the 3D coordinates (e.g., S3(x, y, z)) of each point cloud data (e.g., feature point S3) in the third image; according to the correspondence between the visual features between the first image and the second image (e.g., S1 and S2 correspond), and the correspondence between the second image and the third image (e.g., S2 and S3 correspond), the electronic device 708 can deduce the one-to-one correspondence between each visual feature in the first image and each point cloud data in the third image, thereby obtaining the 3D coordinates (e.g., S1(x, y, z)) of each visual feature in the first image.

[0156] In S603, the electronic device determines a matching second pixel point from the second image according to a first pixel point in the first image.

[0157] It should be noted that in this application, a single visual feature point (or a single visual feature) can be represented by a single pixel point; the above-mentioned first pixel point can refer to a visual feature point corresponding to a pixel point of the first image, and the second pixel point can refer to a visual feature point corresponding to a pixel point of the second image, which is a visual feature point corresponding to the first pixel point in the second image.

[0158] The electronic device can extract visual feature points corresponding to each pixel point in the first image (or the second image) according to a visual feature extraction algorithm, where the visual feature extraction algorithm includes but is not limited to SIFT algorithm, SURF algorithm and ORB algorithm. Correspondingly, the visual feature point can be at least one of SIFT feature point, SURF feature point or ORB feature point; where the visual feature point obtained by processing the first image (or the second image) using the SIFT algorithm is the SIFT feature point, the visual feature point obtained by processing the first image (or the second image) using the SURF algorithm is the SURF feature point, and the visual feature point obtained by processing the first image (or the second image) using the ORB algorithm is the ORB feature point.

[0159] In some embodiments, the electronic device can use the SIFT algorithm (SURF algorithm or ORB algorithm) to perform visual feature extraction on the pixel points of the first image (or the second image) to obtain at least one SIFT feature point (SURF feature point or ORB feature point).

[0160] For example, taking the SIFT algorithm to extract point features from an image as an example, an electronic device can extract point features from a first image using the SIFT algorithm to obtain a first SIFT point feature set, and then extract point features from a second image using the SIFT algorithm to obtain a second SIFT point feature set; wherein, the first pixel is a SIFT point feature in the first SIFT point feature set, and the second pixel is a SIFT point feature in the second SIFT point feature set that matches the first pixel.

[0161] For example, an electronic device extracts point features from a first image using the SIFT algorithm, obtaining a first SIFT point feature set F, where F = {f1, f2, ..., f...} M Similarly, the electronic device extracts point features from the second image using the SIFT algorithm, obtaining a second SIFT point feature set F', where F' = {f'1, f'2, ..., f'}. N The electronic device will match the elements in set F and set F' one by one (e.g., point features f). a With point feature f' a Matching, point feature fb and point feature f' b (e.g., matching) to establish a feature matching relationship L between set F and set F', where L = {(f a ,f' a ), (f b ,f' b ), ..., (f K ,f' K )}, where f a f b , ..., f K ∈F,f' a , f' b ,…,f' K ∈F', where M, N and K are all integers.

[0162] For example, the first pixel is f a The second pixel is f' a The first pixel f a With the second pixel f' a These are mutually matching (or mutually corresponding) visual features (or mutually matching pixels). Electronic devices extract point features (e.g., first or second pixels) from a first (or second) image using the SIFT algorithm with higher reliability.

[0163] It should be noted that the processing method of SURF algorithm (or ORB algorithm) for extracting visual features is similar to that of SIFT algorithm for extracting visual features (e.g., extracting point features from the first image or the second image), and will not be repeated here.

[0164] Therefore, in different application scenarios, the electronic device can use different feature processing algorithms (such as SURF algorithm, SIFT algorithm, or ORB algorithm) to perform visual feature extraction on the pixel points of the image, to ensure the accuracy of image matching through visual feature points in different application scenarios.

[0165] In S604, the electronic device determines a third pixel point registered with the second pixel point from the third image.

[0166] The third pixel point is a point cloud data or a three-dimensional coordinate point (3D coordinate point) in the third image. Since the first pixel point is matched with the second pixel point, and the second pixel point is matched with the third pixel point, the pixel point corresponding to the first pixel point in the third image is the third pixel point, that is, the first pixel point is matched with the third pixel point, or in other words, the third pixel point is the 3D coordinate corresponding to the first pixel point.

[0167] For example, in S603, the electronic device has determined the second pixel point. Since the second image and the third image are registered with each other, the electronic device can determine the third pixel point registered with the second pixel point from the third image. Since the first pixel point corresponds to the second pixel point, the pixel point corresponding to the second pixel point in the third image is the third pixel point, and the third pixel point is a 3D coordinate point, therefore, the 3D coordinate corresponding to the first pixel point is the 3D coordinate of the third pixel point.

[0168] In some embodiments, the electronic device determines the third pixel point registered with the second pixel point from the third image, including: determining the third pixel point according to the second pixel point and the coordinate conversion model.

[0169] In some scenarios, the electronic device can determine the point cloud data (such as the third pixel point) corresponding to the second pixel point in the third image according to the coordinate conversion model; for example, the electronic device can one-to-one correspond each visual feature (such as the second pixel point) in the second image with each point cloud data (such as the third pixel point) in the third image according to the coordinate conversion model, to obtain the 3D coordinate corresponding to each visual feature (such as the second pixel point) in the second image, such as the 3D coordinate corresponding to the second pixel point is the 3D coordinate of the third pixel point.

[0170] It should be noted that since the coordinate conversion model between the second image and the third image is fixed and known, it can be understood that the second image and the third image are registered (or aligned) in the same coordinate system; the electronic device can obtain the coordinate conversion model between the second image and the third image, and establish the correspondence between each visual feature (such as SIFT point feature) in the second image and each point cloud data in the third image through the coordinate conversion model.

[0171] In the present embodiment, the electronic device registers (or corresponds) the second pixel point and the third pixel point through the coordinate conversion model, which has the following advantages: 1) Through the coordinate conversion model, the second pixel point and the third pixel point can be aligned in the same coordinate system, thereby realizing the fusion of data. This cross-dimension image information integration can provide more comprehensive and accurate data support for subsequent control point extraction operations; 2) The coordinate conversion model can accurately register the second pixel point and the third pixel point, reducing the registration error caused by inconsistent coordinate systems, thereby helping to improve the accuracy and accuracy of subsequent control point extraction; 3) Since the coordinate conversion model is a technology based on mathematical algorithms and computational optimization, it can realize efficient data processing and conversion. Therefore, registering the second pixel point and the third pixel point through the coordinate conversion model can reduce the amount of calculation and data processing time, which is conducive to improving the overall efficiency of image processing.

[0172] In some other embodiments, the first acquisition module and the second acquisition module meet the following requirements during image acquisition: synchronicity and positional fixity, wherein the first acquisition module is used to acquire the second image, the second acquisition module is used to acquire the third image, and the coordinate conversion model has an associated relationship with the synchronicity and the positional fixity.

[0173] The synchronicity (which can also be expressed as simultaneity) can mean that the second image and the third image are collected at the same time (or almost at the same time), that is, two image acquisitions are of the same time or similar time scenes; the positional fixity (which can also be expressed as positional stability or positional relative invariability) can mean that the second image and the third image are collected from the same angle (that is, the same view angle) or the same position, that is, they collect the scene under the same angle (that is, the same view angle).

[0174] The synchronization and position fixity of the first acquisition module and the second acquisition module in the image acquisition process can ensure that the second image and the third image are acquired at the same time and at the same angle, and the advantages of the image acquisition method in the same angle and at the same time are as follows: 1) Since the second image and the third image are acquired in the same angle and at the same time, each pixel point in the second image and the third image can be accurately corresponded; and the accurate correspondence makes the measurement result more accurate, and is beneficial to improve the accuracy of the subsequent control point extraction. 2) Since the second image and the third image are acquired from the same angle, they can provide different dimensional information (such as texture information or depth information) of the same scene, so as to realize mutual complementation of information, and further improve the situation of insufficient single-view information caused by occlusion and the like; 3) Compared with the image acquisition method in different angles and at different times, the image acquisition method in the same angle and at the same time does not need to perform complex scene matching or angle conversion and the like in the preprocessing, which is beneficial to reduce the calculation amount and improve the overall efficiency of image processing.

[0175] The above-mentioned "the coordinate conversion model is related to the synchronization and position fixity" can be understood as: the determination of the coordinate conversion model is closely related to the synchronization and position fixity of the first acquisition module and the second acquisition module in the image acquisition process; specifically, if the first acquisition module and the second acquisition module fail to realize synchronization in the image acquisition process, the time parameter in the coordinate conversion model closely related to the acquisition time will be directly affected, so that the coordinate conversion model changes, and then the registration (or correspondence) of each pixel point in the second image and the third image is affected; similarly, if the position of the two acquisition modules changes in the image acquisition process, the acquisition angle will change, and then the coordinate parameter in the coordinate conversion model related to the acquisition angle will be affected, so that the coordinate conversion model changes, and then the registration (or correspondence) of each pixel point in the second image and the third image is affected. Therefore, ensuring that the first acquisition module and the second acquisition module maintain high synchronization and position fixity in the image acquisition process is the key to realizing the acquisition of the second image and the third image at the same time and at the same angle; this method can ensure that the coordinate conversion model remains fixed, and does not need to be frequently adjusted due to the change of time and angle, so as to be beneficial to improve the efficiency and accuracy of the registration (or alignment) between the two images.

[0176] The first acquisition module (or the second acquisition module) can be an acquisition device, a software module or a chip, and the present application does not limit the same.

[0177] In addition, the first acquisition module (or the second acquisition module) can be integrated in the same device, or can be in different devices; for example, when they belong to the same device (such as electronic device A), they can capture a specific scene at the same time and at the same angle; when they belong to different devices, the devices need to keep the acquisition time synchronized when acquiring the second image and the third image, on the one hand, to acquire the scene at the same time or at a similar time; on the other hand, the device position needs to be fixed (i.e., stable) to ensure that the image is acquired from the same angle (i.e., perspective).

[0178] For example, as shown in (a) of FIG. 7, taking the camera 701 as the first acquisition module and the laser radar device 702 as the second acquisition module as an example, where the camera 701 and the laser radar device 702 belong to two parts of the same image acquisition device 700; the camera 701 is used to acquire the second image (i.e., an example of the most relevant basic image), and the laser radar device 702 is used to acquire the third image (i.e., an example of the point cloud image); the camera 701 and the laser radar device 702 have the same acquisition time and acquisition angle when acquiring the image, and the coordinate conversion model between the camera 701 and the laser radar device 702 is P u’v’ = kM P w ; the electronic device can establish a matching relationship between the SIFT point features of the second image and the point cloud data (or 3D coordinates) of the third image through the coordinate conversion model P u’v’ , where M represents the coordinate conversion model, k represents the scale parameter, (u', v') represents the pixel position corresponding to the point feature f' in the second SIFT point feature set F', and P w is the 3D coordinates (x, y, z) of the point cloud (or point cloud data); the 3D point cloud coordinate set P = {p1, p2, …, p a} corresponding to each SIFT point feature in the feature matching relationship L of the above set F and set F' is {f b , f K , …, f K}; that is, the electronic device matches the SIFT point features (i.e., an example of the second pixel point) in the second image with the corresponding 3D coordinates through the coordinate conversion model, thereby obtaining a new coordinate set P = {p1, p2, …, p K}.

[0179] If the point cloud data M(x, y, z) in the third image matches the second pixel point, the 3D coordinate corresponding to the first pixel point in the third image is M(x, y, z). In this way, the method 600 can make each visual feature in the first image find a corresponding 3D coordinate in the third image; and finally obtain a feature point set R = {R1, R2, R3, …, Rn} of matched 3D coordinates, where R1, R2, R3, …, and Rn are one visual feature in the first image with matched 3D coordinates, for example, R1 can be the first pixel point with matched 3D coordinates. i i i

[0180] As can be seen, in the method 600, the electronic device performs same-dimension matching on a certain pixel point (such as the first pixel point) in the first image and a corresponding pixel point (such as the second pixel point) in the second image, which is not only relatively easy to match but also highly accurate; in addition, since the second image and the third image are registered images, each pixel point (or visual feature) in the second image has a corresponding spatial position in the third image, so that the registered third pixel point can be determined from the third image according to the second pixel point; and since the first image and the second image are matched (which can also be understood as registered), and the second image and the third image are registered, the registration of the first image and the third image can be indirectly realized; such a method of indirectly registering the first image and the third image through an intermediate image (such as the second image) is not only conducive to improving the accuracy of registration, but also conducive to improving the accuracy of control point extraction.

[0181] In an optional implementation, the above method 600 further includes S605.

[0182] S605: The electronic device determines the coordinate of the first control point according to the third pixel point.

[0183] wherein the third pixel point is a pixel point (i.e., an example of point cloud data) corresponding to the first pixel point in the third image, and the 3D coordinate of the first pixel point is the 3D coordinate of the third pixel point; and the coordinate of the first control point is the coordinate of the first pixel point, and the first control point is the first pixel point.

[0184] As described above, the electronic device can determine the 3D coordinate of the first pixel point according to the third pixel point, so when the electronic device determines to extract the first pixel point in the first image as the first control point according to the clustering algorithm, the coordinate of the first control point can be determined according to the third pixel point.

[0185] ​​​In other words, the electronic device can correspond each visual feature (e.g., the first pixel point) in the first image to each point cloud data (e.g., the third pixel point) in the third image to obtain the set R by performing S604; finally, a plurality of elements are extracted from the set R as control points according to the requirement of visual positioning, such as the first control point.

[0186] It should be noted that in some embodiments, the electronic device can process the third pixel point by a clustering algorithm, a uniformity principle, or the like to determine the coordinates of the first control point.

[0187] For example, the electronic device determines the coordinates of the first control point according to the third pixel point, including: clustering the third pixel point by a clustering algorithm to determine the first control point, and the first control point includes the first pixel point; the electronic device determines the coordinates of the first control point according to the coordinates of the third pixel point.

[0188] It should be noted that the electronic device determines the first control point by clustering the set R by a clustering algorithm, and since the set R includes the 3D coordinates of the first pixel point, and the 3D coordinates of the first pixel point are determined according to the third pixel point, the clustering of the set R by the electronic device can also be understood as the clustering of the third pixel point; in other words, the electronic device can cluster the set R including the third pixel point to extract the first pixel point as the first control point, and at this time, the coordinates of the first control point are the coordinates of the first pixel point (i.e., the coordinates of the third pixel point).

[0189] In some examples, after the electronic device obtains the set R, it can select part of the elements in the set R as control points according to the requirement of visual positioning; for example, the electronic device clusters the elements in the set R by a clustering algorithm to determine that X elements are extracted as X control points, wherein the X elements include the first pixel point, that is, the X control points include the first control point, and the coordinates of the first control point are the 3D coordinates of the first pixel point. The clustering algorithm includes but is not limited to KNN algorithm and K-means algorithm.

[0190] For example, the electronic device can cluster the elements in the set R by a KNN clustering algorithm to extract a plurality of elements as control points; for example, the control points can include {control point 1 (uv a , p1), control point 2 (uv b , p2), …, control point N (uv N , p N )}, wherein uv i represents the pixel position (or the coordinates of the pixel point) corresponding to the feature f i , i∈{a, b, …, N}.

[0191] In some examples, the electronic device can extract part of the elements in the set R as control points according to the uniformity principle, for example, evenly divide the elements in the set R into N blocks, and extract one element from each block as a control point.

[0192] In some examples, the electronic device can extract all the elements in the set R as control points after obtaining the set R.

[0193] In this embodiment, the user can automatically select a suitable control point (for example, the first control point) according to the requirements of visual positioning through the clustering algorithm, without manual selection, which is efficient and accurate.

[0194] The method 600 and possible implementation manners are described in detail above, and the electronic device is taken as a terminal device (for example, a notebook computer) in the following, and the method 600 and possible implementation manners are further described in combination with interface embodiments of different application scenarios; it should be noted that the following interface embodiments are only used to illustrate the method 600, and should not be understood as a limitation on the application scenarios of the present application; in addition, it should be noted that the terminal device can include a display screen, and the user can perform image processing and other operations on the display interface displayed by the display screen.

[0195] As shown in (a) of FIG. 8, the user can double-click the shortcut icon of “image editing 802” on the display interface 801 of the terminal to start the image editing software; at this time, the user can see the main interface 803 of the image editing software; the main interface 803 includes a toolbar 804, an image display area 805, and an operation bar 806, wherein the toolbar 804 includes text editing, deletion, collection, and other operation tools; the image display area 805 is used to display video images (which can also be referred to as to-be-processed images) input by the user (that is, an example of a first image) and image processing results (that is, processing results of the video images), and the like; it should be noted that the images displayed by the image display area 805 can be different in different scenarios, and specific details can be referred to the interface embodiments below, which will not be described here again.

[0196] The function of the upload button 8061 is to upload the video image to be processed by the user; it should be noted that after the upload button 8061 is clicked, the user can select the video image to be processed from the local, or log in to a remote cloud server to obtain the video image to be processed from the cloud server.

[0197] The function of the column display 8062 button is to control the display mode of the image display area 805. For example, the image display area 805 can be displayed in a single column or in multiple columns. Details will be described below in the interface embodiment, and will not be described here in detail.

[0198] The function of the one-key processing 8063 button is to perform one-key processing on the video image uploaded by the user. For example, if the user clicks the one-key processing 8063 button, the terminal will execute the entire process of the method 600, and display the control point extraction result on the image display area 805, so that the user can view the control point extraction result.

[0199] It should be noted that the control point extraction result is used to display the control points extracted by the electronic device. The control points can be displayed on at least one of the video image, the point cloud image, and the most relevant base image. Here, only the case where the control point extraction result is used to display the distribution of the control points on the video image is taken as an example.

[0200] The function of the step-by-step viewing 8064 button is different from that of the one-key processing 8063 button. The function of the step-by-step viewing 8064 button is to view the processing of the video image by the terminal at each step in the process of executing the method 600. If the user is not satisfied with the processing result of a certain step, the terminal can be triggered to re-execute the step through the step-by-step viewing 8064 button. Details will be described below in the interface embodiment, and will not be described here in detail.

[0201] For example, as shown in (a) of FIG. 9, taking the case where the user uploads a video image 901 (which is an example of the first image) from the local as an example, the user clicks the upload 8061 button to upload the video image 901 to be processed. After uploading the video image 901, the user can click the one-key processing 8063 button to start the one-key processing function. At this time, the terminal will execute S601 to S604 in the method 600, and display the control point (such as the black point 902) extraction result in the corresponding area of the image display area 805, as shown in (b) of FIG. 9.

[0202] For another example, as shown in (a) of FIG. 10, the user clicks the upload 8061 button to upload the video image 901 locally; after uploading the video image 901, the user can first click the column display 8062 button to set the display form of the image display area 805; for example, the user first clicks the column display 8062 button and selects three-column display 1002 in the option box 1001 of the column display 8062, as shown in (a) of FIG. 10; then clicks the one-key processing 8063 button to start the one-key processing function, as shown in (b) of FIG. 10; at this time, the terminal will also execute S601 to S604 in the method 600; since the user sets three-column display, when the terminal processes the video image 901, the video image 901, the point cloud image (i.e. one example of the third image) and the control point extraction result can be displayed in the image display area 805 in a three-column manner, wherein the left column 1003 is used to display the video image 901, the middle column 1004 is used to display the point cloud image, and the right column 1005 is used to display the control point extraction result, as shown in (b) of FIG. 10.

[0203] For another example, as shown in (a) of FIG. 11, the user clicks the upload button 8061 to upload a video image 901 locally; if the user first clicks the column display button 8062 and selects four-column display 1101 in the option box 1001 of the column display 8062; then clicks the one-key processing button 8063 to start the one-key processing function, as shown in (b) of FIG. 11; at this time, the terminal will also perform S601 to S604 in the method 600; since the user sets four-column display, when the terminal processes the video image 901, the terminal will display the video image 901, the most relevant base image (which is an example of the second image), the point cloud image, and the control point extraction result in the corresponding area in the image display area 805 in four-column mode, wherein, from left to right, the first column 1102 is used to display the video image 901, the second column 1103 is used to display the most relevant base image, the third column 1104 is used to display the point cloud image, and the fourth column 1105 is used to display the control point extraction result, as shown in (b) of FIG. 11. As can be seen from the image shown in (b) of FIG. 11, the first pixel point A in the video image corresponds to the second pixel point B in the most relevant base image; the second pixel point B in the most relevant base image corresponds to the third pixel point C in the point cloud image; the terminal can determine that the first pixel point A in the video image corresponds to the third pixel point C in the point cloud image by performing S604; the terminal extracts the first pixel point A as a control point by performing S605, at this time, the user can see from the control point extraction result that the first pixel point A appears as the first control point on the video image 901, and the user can click the first control point (i.e. the first pixel point A) on the control point extraction result to see the 3D coordinates of the first control point (i.e. the first pixel point A), i.e. A(x1, y1, z1).

[0204] For another example, as shown in (a) of FIG. 12, the user clicks the upload button 8061 to upload the video image 901 locally; if the user first clicks the column display button 8062 and selects five-column display 1201 in the option box 1001 of the column display 8062; and then clicks the one-key processing button 8063 to start the one-key processing function, as shown in (b) of FIG. 12; at this time, the terminal will also perform S601 to S604 in the method 600; since the user sets five-column display, when the terminal processes the video image 901, the terminal displays the video image 901, the most relevant basic image (i.e., an example of the second image), the point cloud image (i.e., an example of the third image), the 2D-3D matching result, and the control point extraction result in the corresponding area in the image display area 805 in the form of five columns, wherein, from left to right, the first column 1202 is used to display the video image 901, the second column 1203 is used to display the most relevant basic image, the third column 1204 is used to display the point cloud image, the fourth column 1205 is used to display the 2D-3D matching result, and the fifth column 1206 is used to display the control point extraction result, as shown in (b) of FIG. 12.

[0205] It should be noted that the 2D-3D matching result is used to display the image processing result of the terminal performing the following steps: according to the correspondence between the visual features of the video image 901 and the most relevant basic image (for example, the first pixel point A corresponds to the second pixel B), and the correspondence between the most relevant basic image and the point cloud image (for example, the second pixel point B corresponds to the third pixel point C), the terminal can deduce the one-to-one correspondence between each visual feature in the video image 901 and each point cloud data in the point cloud image, thereby obtaining the 3D coordinates of each visual feature in the video image 901 (for example, the 3D coordinates of the first pixel point A are (x, y, z)); the 2D-3D matching result displayed is the 3D coordinates of each visual feature in the video image 901, for example, the user clicks the first pixel point A on the 2D-3D matching result to see the 3D coordinates of the pixel point A, i.e., A(x1, y1, z1). It should also be noted that in FIGS. 13 to 17 below, the explanations of the first pixel point A, the second pixel point B, and the third pixel point C marked on different images in the image display area 805 are similar to the case shown in (b) of FIG. 12, and can refer to the related explanations in (b) of FIG. 12, which will not be described separately below.

[0206] Next, taking the user's selection of the five-column display of the processing results of the images at different stages as an example, the user's method of viewing the processing results of the images at different stages in the execution process of the method 600 through the step-by-step viewing button 8064 is explained.

[0207] For example, as shown in (a) of FIG. 13, after the terminal processes the video image, the terminal displays the processing results of the images in different processing stages in the image display area 805 in a five-column manner; wherein, from left to right, the first column 1301 is used to display the video image (i.e. an example of the first image), the second column 1302 is used to display the most relevant base image (i.e. an example of the second image), the third column 1303 is used to display the point cloud image (i.e. an example of the third image), the fourth column 1304 is used to display the 2D-3D matching result, and the fifth column 1305 is used to display the control point extraction result; at this time, the user can click the step-by-step viewing 8064 button and select the similarity search 1307 in the step-by-step viewing 8064 option box 1306 (equivalent to triggering the terminal (i.e. an example of an electronic device) to start the image similarity search module described above), at this time, the second column 1302 becomes the selected state (i.e. shown by the solid thick black frame), as shown in (b) of FIG. 13, to show the user the processing result of the similarity search 1307 corresponding to the second column 1302, so as to facilitate the user to view the processing result (i.e. the most relevant base image); if the user thinks that the current similarity search result does not meet the requirements, at this time, the user can click the "re-execute 1308" button behind the "similarity search" option (as shown in (a) of FIG. 13), so that the terminal re-executes the processing steps corresponding to the similarity search 1307 (for example, re-executes the step of searching the second image matching the first image from the base image library in the electronic device in the method 600 described above), and at the same time, the fifth column 1305 also becomes the selected state (i.e. shown by the solid thick black frame) as the second column 1302; when the most relevant base image displayed by the second column 1302 is updated, the control point extraction result displayed by the fifth column 1305 will also be updated accordingly, as shown in (b) of FIG. 13.

[0208] For another example, as shown in (a) of FIG. 14, the user clicks the step-by-step viewing 8064 button, and selects the associated feature matching 1401 in the option box 1306 of the step-by-step viewing 8064, at this time, the first column 1301, the second column 1302 and the third column 1303 all become selected (i.e. shown as a solid thick black frame), so as to show the user the processing results of the associated feature matching corresponding to the first column 1301, the second column 1302 and the third column 1303, so as to facilitate the user to view the processing results (i.e. the matching results of the video image and the most associated basic image, and the matching results of the most associated basic image and the point cloud image); if the user thinks that the current associated feature matching result does not meet the requirements, at this time, the user can click the "re-execute 1402" button behind the "associated feature matching" option, so that the terminal re-executes the processing steps corresponding to the associated feature matching (for example, re-executes S602 and S603 executed by the electronic device in the method 600 above), and at the same time, the fifth column 1305 also becomes selected (i.e. shown as a solid thick black frame) like the first column 1301, the second column 1302 and the third column 1303; when the visual features on the most associated basic image displayed by the second column 1302 and the point cloud data on the point cloud image displayed by the third column 1303 are updated, the control point extraction result displayed by the fifth column 1305 will also be updated correspondingly, as shown in (b) of FIG. 14.

[0209] For another example, as shown in (a) of FIG. 14, the user clicks the step-by-step viewing 8064 button, and selects the associated feature matching 1401 in the option box 1306 of the step-by-step viewing 8064, at this time, the first column 1301, the second column 1302 and the third column 1303 all become selected (i.e. shown as a solid thick black frame), so as to show the user the processing results of the associated feature matching corresponding to the first column 1301, the second column 1302 and the third column 1303, so as to facilitate the user to view the processing results (i.e. the matching results of the video image and the most associated basic image, and the matching results of the most associated basic image and the point cloud image); if the user thinks that the current associated feature matching result does not meet the requirements, at this time, the user can click the "re-execute 1402" button behind the "associated feature matching" option, so that the terminal re-executes the processing steps corresponding to the associated feature matching (for example, re-executes S602 and S603 executed by the electronic device in the method 600 above), and at the same time, the fifth column 1305 also becomes selected (i.e. shown as a solid thick black frame) like the first column 1301, the second column 1302 and the third column 1303; when the visual features on the most associated basic image displayed by the second column 1302 and the point cloud data on the point cloud image displayed by the third column 1303 are updated, the control point extraction result displayed by the fifth column 1305 will also be updated correspondingly, as shown in (b) of FIG. 14.

[0210] For example, as shown in (a) of FIG. 16, the user clicks the step-by-step viewing 8064 button, and selects the control point extraction 1601 in the option box 1306 of the step-by-step viewing 8064, at this time, the fifth column 1305 becomes the selected state (i.e. shown by the thick solid line and black frame), so as to show the user that the control point extraction result displayed in the fifth column 1305 corresponds to the processing result of the control point extraction 1601, so as to facilitate the user to view the processing result (for example, the terminal extracts the visual features (i.e. control points) meeting the requirements based on the 2D-3D matching result by using the clustering algorithm); if the user thinks that the current control point extraction result does not meet the requirements, at this time, the user can click the "re-execute 1602" button behind the "control point extraction 1601" option, so that the terminal re-executes the processing step corresponding to the control point extraction 1601 (for example, re-executes S605 in the method 600 executed by the electronic device); after the terminal re-executes the processing step corresponding to the control point extraction 1601, the control point extraction result displayed in the fifth column 1305 will also be updated, as shown in (b) of FIG. 16.

[0211] It should be noted that if the user wants to carefully view the processing result of a certain stage image, the user can use the zoom display function to view the picture; for example, as shown in (a) of FIG. 17, the user right-clicks in the area where the control point extraction result is located to call out the tool tab 1701, and selects the "zoom display" function 1702 in the tab 1701, at this time, the terminal will generate a zoom display window 1703 in the main interface 803, and zoom display the control point extraction result in the window 1703, as shown in (b) of FIG. 17; the user can view the control point extraction result in the window 1703; of course, the user can also adjust the zoom effect of the control point extraction result in the window 1703 by using the zoom tool (for example, the zoom-in or zoom-out tool).

[0212] The above describes an example of the image processing method provided by the present application in detail. It should be noted that in order to realize the above functions, the electronic device comprises a hardware structure and / or a software module corresponding to the execution of each function. Those skilled in the art should easily realize that, in combination with the modules and algorithm steps of each example described in the embodiments disclosed in the present application, the present application can be realized in the form of hardware or a combination of hardware and computer software. Whether a certain function is executed by hardware or computer software driven hardware depends on the specific application and design constraints of the technical solution. Professional technicians can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the present application.

[0213] The present application can divide the functions of the image processing method according to the above method examples, for example, each function can be divided into a functional module, or two or more functions can be integrated into one module. The above modules can be realized in the form of hardware or in the form of software functional modules. It should be noted that the division of the modules in the present application is illustrative, and is only a logical function division. When actually implemented, there can be another division manner.

[0214] FIG. 18 shows a structural schematic diagram of an image processing apparatus 1800 provided by the present application. The apparatus 1800 can be a terminal device, a server, an image processing system or a chip (system). The apparatus 1800 comprises a first acquisition module 1801, a second acquisition module 1802, an acquisition module 1803 and a processing module 1804.

[0215] The acquisition module 1803 is configured to acquire a first image, the first image being a two-dimensional image.

[0216] The processing module 1804 is configured to determine a second image and a third image according to the first image, the second image being at least one two-dimensional image matched with the first image, and the third image being a three-dimensional image registered with the second image; and to determine a matched second pixel point from the second image according to a first pixel point in the first image; and to determine a registered third pixel point from the third image according to the second pixel point.

[0217] The specific manners in which the apparatus 1800 performs the image processing method and the beneficial effects produced thereby can be seen in the related descriptions of the method embodiments shown in FIG. 6.

[0218] In the apparatus 1800 shown in FIG. 18, the first pixel point is a visual feature point corresponding to a pixel point of the first image, and the second pixel point is a visual feature point corresponding to a pixel point of the second image. The beneficial effects of this part can be seen in the above embodiments.

[0219] In the apparatus 1800 shown in FIG. 18, the visual feature point is one of a SIFT feature point, a SURF feature point or an ORB feature point. The beneficial effects of this part can be seen in the above embodiments.

[0220] In the apparatus 1800 shown in FIG. 18, the processing module 1804 is specifically configured to determine the third pixel point according to the second pixel point and a coordinate conversion model. The beneficial effects of the processing module 1804 in performing this step can be seen in the above embodiments.

[0221] In the apparatus 1800 shown in FIG. 18, the first acquisition module 1801 and the second acquisition module 1802 meet the following requirements during image acquisition: synchronization and positional fixity, wherein the first acquisition module 1801 is configured to acquire the second image, the second acquisition module 1802 is configured to acquire the third image, and the coordinate conversion model is associated with the synchronization and the positional fixity. The beneficial effects of this part can be seen from the above embodiments.

[0222] In the apparatus 1800 shown in FIG. 18, the processing module 1804 is further configured to determine the coordinates of the first control point according to the third pixel point. The beneficial effects of the processing module 1804 performing this step can be seen from the above embodiments.

[0223] In the apparatus 1800 shown in FIG. 18, the processing module 1804 is specifically further configured to: cluster the third pixel points by using a clustering algorithm, determine the first control point, and the first control point includes the first pixel point; and determine the coordinates of the first control point according to the coordinates of the third pixel point. The beneficial effects of the processing module 1804 performing this step can be seen from the above embodiments.

[0224] In the apparatus 1800 shown in FIG. 18, the second image is at least one two-dimensional image that matches the first image, including: the second image is at least one two-dimensional image that has the highest similarity with the first image. The beneficial effects of this part can be seen from the above embodiments.

[0225] In the apparatus 1800 shown in FIG. 18, the acquisition module 1803 is specifically configured to: acquire the first image from a video. The beneficial effects of the acquisition module 1803 performing this step can be seen from the above embodiments.

[0226] FIG. 19 shows a structural schematic diagram of an electronic device 1900 provided in the present application. The dashed line in FIG. 19 indicates that the unit or the module is optional. The electronic device 1900 can be used to implement the method described in the above method embodiments. The electronic device 1900 can be a server or a chip (system).

[0227] The electronic device 1900 includes one or more processors 1901, which can support the electronic device 1900 to implement the method in the method embodiments corresponding to FIG. 6. The processor 1901 can be a general-purpose processor or a special-purpose processor. For example, the processor 1901 can be a central processing unit (CPU). The CPU can be used to control the electronic device 1900, execute software programs, and process data of the software programs. The electronic device 1900 can also include a communication unit 1905 to realize input (reception) and output (transmission) of signals.

[0228] The electronic device 1900 can be a chip (system) including a memory and a processor, wherein the processor is configured to execute a computer program stored in the memory to implement the method shown in each of the above embodiments.

[0229] The communication unit 1905 can be an input and / or output circuit of the chip (system), or the communication unit 1905 can be a communication interface of the chip (system), which can be a component of the electronic device 1900.

[0230] For another example, the communication unit 1905 can be a transceiver of the electronic device 1900, or the communication unit 1905 can be a transceiver circuit of the electronic device 1900.

[0231] The electronic device 1900 can include one or more memories 1902, in which programs 1904 can be stored, and the programs 1904 can be run by the processor 1901 to generate instructions 1903, so that the processor 1901 executes the method described in the above method embodiments according to the instructions 1903. Optionally, the memory 1902 can also store data. Optionally, the processor 1901 can also read the data stored in the memory 1902, and the data can be stored in the same storage address as the program 1904, or the data can be stored in a different storage address from the program 1904.

[0232] The processor 1901 and the memory 1902 can be separately arranged, or can be integrated together, for example, integrated on a system on chip (SOC) of the electronic device.

[0233] The specific manner in which the processor 1901 executes the method of image processing can be referred to the related description in the method embodiments.

[0234] It should be understood that each step of the above method embodiments can be completed by a logic circuit in the form of hardware in the processor 1901 or instructions in the form of software. The processor 1901 can be a CPU, a digital signal processor (DSP), a field programmable gate array (FPGA), or other programmable logic devices, for example, discrete gates, transistor logic devices, or discrete hardware components.

[0235] The present application also provides a computer program product, which is executed by the processor 1901 to implement the method of any method embodiment in the present application.

[0236] The computer program product can be stored in the memory 1902, for example, a program 1904, which is finally converted into an executable object file capable of being executed by the processor 1901 through preprocessing, compiling, assembling and linking and other processing procedures.

[0237] The computer program can be a high-level language program or an executable object program.

[0238] The computer readable storage medium is, for example, the memory 1902. The memory 1902 can be a volatile memory or a non-volatile memory, or the memory 1902 can include both volatile memory and non-volatile memory. The non-volatile memory can be a Read-Only Memory (ROM), a Programmable ROM (PROM), an Erasable PROM (EPROM), an Electrically EPROM (EEPROM) or a flash memory. The volatile memory can be a Random Access Memory (RAM) used as an external cache. By way of example and not limitation, many forms of RAM can be used, such as Static RAM (SRAM), Dynamic RAM (DRAM), Synchronous DRAM (SDRAM), Double Data Rate SDRAM (DDR SDRAM), Enhanced SDRAM (ESDRAM), SynchLink DRAM (SLDRAM) and Direct Rambus RAM (DRRAM).

[0239] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process of the above-described apparatus and device and the generated technical effects can refer to the corresponding process and technical effects in the foregoing method embodiments, which will not be described here.

[0240] In several embodiments provided in the present application, the disclosed system, device and method can be implemented in other ways. For example, some features of the above-described method embodiments can be omitted or not performed. The above-described device embodiments are merely illustrative, and the splitting of units is merely a logical function splitting, and actual implementation can have another splitting manner. Multiple units or components can be combined or integrated into another system. In addition, the coupling between units or the coupling between components can be direct coupling or indirect coupling, and the above coupling includes electrical, mechanical or other forms of connection.

[0241] The above embodiments are only used to illustrate the technical solutions of the present application, rather than limit them. Although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that they can still modify the technical solutions recorded in the foregoing embodiments, or make equivalent replacements to some technical features, and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the ideas and scope of the technical solutions of the embodiments of the present application, and should be included in the protection scope of the present application.

[0242] Finally, the above is merely a specific implementation of the present application, but the protection scope of the present application is not limited to this. Any changes or replacements within the technical scope disclosed in the present application should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

Claims

1. A method of image processing, characterized by, The method comprises: acquiring a first image, the first image being a two-dimensional image; determining a second image and a third image according to the first image, the second image being a two-dimensional image matching the first image in at least one two-dimensional image, and the third image being a three-dimensional image registered with the second image; determining a matching second pixel point from the second image according to a first pixel point in the first image; determining a registered third pixel point from the third image according to the second pixel point.

2. The method of claim 1, wherein, The first pixel point is a visual feature point corresponding to a pixel point of the first image, and the second pixel point is a visual feature point corresponding to a pixel point of the second image.

3. The method of claim 2, wherein, The visual feature point is one of a scale-invariant feature transform (SIFT) feature point, a speeded up robust features (SURF) feature point, or an oriented FAST and rotated BRIEF (ORB) feature point.

4. The method according to any one of claims 1 to 3, characterized in that, The determining of the registered third pixel point from the third image according to the second pixel point comprises: determining the third pixel point according to the second pixel point and a coordinate conversion model.

5. The method of claim 4, wherein, The first acquisition module and the second acquisition module meet the following requirements in an image acquisition process: synchronization and positional fixity, wherein the first acquisition module is configured to acquire the second image, the second acquisition module is configured to acquire the third image, and the coordinate conversion model has a correlation with the synchronization and the positional fixity.

6. The method according to any one of claims 1 to 5, characterized in that, The method further comprises: determining a coordinate of a first control point according to the third pixel point.

7. The method of claim 6, wherein, The determining of the coordinate of the first control point according to the third pixel point comprises: determining the first control point by clustering the third pixel point, the first control point including the first pixel point; determining the coordinate of the first control point according to the coordinate of the third pixel point.

8. The method according to any one of claims 1 to 7, characterized in that, The second image being a two-dimensional image matching the first image in at least one two-dimensional image comprises: The second image is a two-dimensional image having the highest similarity to the first image in the at least one two-dimensional image.

9. The method according to any one of claims 1 to 8, characterized in that, The acquiring of the first image comprises: acquiring the first image from a video.

10. An apparatus for image processing, characterized by The apparatus comprises an acquisition module and a processing module; The acquisition module is configured to acquire a first image, the first image being a two-dimensional image; The processing module is configured to determine a second image and a third image according to the first image, the second image being a two-dimensional image matching the first image in at least one two-dimensional image, and the third image being a three-dimensional image registered with the second image; to determine a matching second pixel point from the second image according to a first pixel point in the first image; and to determine a registered third pixel point from the third image according to the second pixel point.

11. The apparatus of claim 10, wherein, The first pixel point is a visual feature point corresponding to a pixel point of the first image, and the second pixel point is a visual feature point corresponding to a pixel point of the second image.

12. The apparatus of claim 11, wherein, The visual feature point is one of a scale-invariant feature transform (SIFT) feature point, a speeded up robust features (SURF) feature point, or an oriented FAST and rotated BRIEF (ORB) feature point.

13. The apparatus of any one of claims 10-12, wherein, The processing module is specifically configured to: determine the third pixel point according to the second pixel point and a coordinate conversion model.

14. The apparatus of claim 13, wherein, The device also includes a first acquisition module and a second acquisition module, wherein the first acquisition module and the second acquisition module meet the following requirements in the image acquisition process: synchronization and positional fixity, wherein the first acquisition module is used to acquire the second image, the second acquisition module is used to acquire the third image, and the coordinate conversion model has an association relationship with the synchronization and the positional fixity.

15. The apparatus of any one of claims 10-14, wherein, The processing module is further used to: determine the coordinates of the first control point according to the third pixel point.

16. The apparatus of claim 15, wherein, The processing module is specifically further used to: cluster the third pixel point through a clustering algorithm to determine the first control point, and the first control point includes the first pixel point; determine the coordinates of the first control point according to the coordinates of the third pixel point.

17. The apparatus of any one of claims 10-16, wherein, The second image is at least one two-dimensional image that matches the first image, including: the second image is a two-dimensional image with the highest similarity to the first image in the at least one two-dimensional image.

18. The apparatus of any one of claims 10-17, wherein, The acquisition module is specifically used to: acquire the first image from a video.

19. An electronic device, comprising: The electronic device includes a processor and a memory, the memory is used to store a computer program, and the processor is used to call and run the computer program from the memory, so that the electronic device executes the method in any one of claims 1 to 9.

20. A computer-readable storage medium, characterized in that, The computer readable storage medium stores a computer program, and when the computer program is executed by a processor, the processor executes the method in any one of claims 1 to 9.

21. A chip system, characterized by The chip system includes a memory and a processor, and the processor is configured to execute a computer program stored in the memory to implement the method in any one of claims 1 to 9.

Images