Image processing methods, apparatus, electronic devices and storage media

Abstract

This application relates to the field of image processing technology and provides an image processing method, apparatus, electronic device, and storage medium. The method includes: performing a first image processing on a target image frame to obtain a first sub-image, wherein the first image processing is real-time processing according to the image acquisition time corresponding to the target image frame; performing a second image processing on the target image frame to obtain a second sub-image, wherein the second image processing is non-real-time image processing performed on all image frames in the target image frame in a random order; and generating a target fusion sub-image based on the visual overlap area between the first and second sub-images, wherein the target fusion sub-image includes the visual overlap area. This embodiment effectively improves the success rate of global alignment and fusion of sub-images, thereby improving the accuracy of global positioning and modeling of the system.

Description

Technical Field

[0001] This application relates to the field of image processing technology, and in particular to an image processing method, apparatus, electronic device and storage medium. Background Technology

[0002] In related technologies, when a robot is in an unknown environment, it can generate a sub-map from images of the surrounding environment and determine a safe route for movement within that environment based on the generated sub-map. However, due to factors such as texture loss, dynamic blurring, and changes in lighting and viewing angle, the robot exhibits insufficient robustness and cannot operate continuously during visual navigation, making it difficult to meet the practical application requirements of navigation.

[0003] In related technologies, multiple image sub-maps are typically generated, and the fused sub-map is used to determine the safe route for the robot to move forward in this environment, thereby solving the problem of insufficient robustness and inability to work continuously in visual navigation.

[0004] However, in the process of generating a fused subgraph from multiple subgraphs in related technologies, the success rate of global alignment and fusion of multiple subgraphs is low, which leads to the failure of global positioning and modeling of the system and affects navigation. Summary of the Invention

[0005] In view of this, embodiments of this application provide an image processing method, apparatus, electronic device, and storage medium, which can effectively improve the success rate of global alignment and fusion of sub-images, thereby improving the accuracy of global positioning and modeling of the system.

[0006] A first aspect of this application provides a method comprising: performing a first image processing on a target image frame to obtain a first sub-image, wherein the first image processing is real-time processing according to the image acquisition time corresponding to the target image frame; performing a second image processing on the target image frame to obtain a second sub-image, wherein the second image processing is non-real-time image processing performed on all image frames in the target image frame in a random order; and generating a target fusion sub-image based on the visual overlap region between the first sub-image and the second sub-image, wherein the target fusion sub-image includes the visual overlap region.

[0007] In one exemplary embodiment, before performing the first image processing on the target image frame, the method further includes: acquiring a real-time image sequence, the real-time image sequence including a plurality of image frames; and filtering out the target image frame from the plurality of image frames.

[0008] In one exemplary embodiment, the step of selecting the target image frame from the plurality of image frames includes: calculating the image feature difference between every two adjacent image frames in the plurality of image frames based on the image features of each image frame in the plurality of image frames; and selecting the target image frame from the real-time image sequence based on the image feature difference.

[0009] In one exemplary embodiment, the plurality of image frames includes M image frames, where M is a positive integer. The step of calculating the image feature difference between any two adjacent image frames based on the image features of each image frame in the plurality of image frames includes: determining a first image feature of the i-th image frame and a second image feature of the (i+1)-th image frame, where i is a positive integer greater than or equal to 1 and less than or equal to M; calculating the image feature difference between the first image feature and the second image feature; and determining the image feature difference as the image feature difference between the i-th image frame and the (i+1)-th image frame.

[0010] In an exemplary embodiment, performing first image processing on the target image frame to obtain a first sub-image includes: identifying a first feature point of a first target object in the first image frame; identifying a second feature point of the first target object in a second image frame, wherein the first image frame and the second image frame are adjacent image frames in the target image frame; determining a first map point corresponding to the first target object based on the first feature point and the second feature point; and fusing the first image frame and the second image frame into the first sub-image based on the first map point.

[0011] In an exemplary embodiment, performing second image processing on the target image frame to obtain a second sub-image includes: identifying a third feature point of a second target object in a third image frame; identifying a fourth feature point of the second target object in a fourth image frame, wherein the third image frame and the fourth image frame are any two image frames in the target image frame; determining a second map point corresponding to the second target object based on the third feature point and the fourth feature point; and fusing the third image frame and the fourth image frame into the second sub-image based on the second map point.

[0012] In an exemplary embodiment, generating a target fusion sub-image based on the visual overlap region between the first sub-image and the second sub-image includes: calculating the visual overlap region between the first sub-image and the second sub-image; determining a target transformation relationship between a first coordinate system and a second coordinate system based on the visual overlap region, wherein the first coordinate system is the coordinate system corresponding to the first sub-image, and the second coordinate system is the coordinate system corresponding to the second sub-image; and generating the target fusion sub-image based on the target transformation relationship, the first sub-image, and the second sub-image.

[0013] In an exemplary embodiment, calculating the visual overlap region between the first sub-image and the second sub-image includes: searching for a fifth feature point included in the first sub-image and searching for a sixth feature point included in the second sub-image; and determining the regions corresponding to the same feature points among the fifth feature points and the sixth feature points as the visual overlap region.

[0014] A second aspect of this application provides an apparatus comprising: a first processing unit configured to perform a first image processing on a target image frame to obtain a first sub-image, wherein the first image processing is real-time processing according to the image acquisition time corresponding to the target image frame; a second processing unit configured to perform a second image processing on the target image frame to obtain a second sub-image, wherein the second image processing is non-real-time image processing performed on all image frames in the target image frame in a random order; and a generation unit configured to generate a target fusion sub-image based on the visual overlap region between the first sub-image and the second sub-image, wherein the target fusion sub-image includes the visual overlap region.

[0015] In one exemplary embodiment, the apparatus further includes: an acquisition unit, configured to acquire a real-time image sequence, the real-time image sequence including a plurality of image frames, before performing the first image processing on the target image frame; and a filtering unit, configured to filter out the target image frame from the plurality of image frames.

[0016] In an exemplary embodiment, the filtering unit includes: a first calculation module, configured to calculate the image feature difference between every two adjacent image frames in the plurality of image frames based on the image features of each image frame in the plurality of image frames; and a filtering module, configured to filter out the target image frame from the real-time image sequence based on the image feature difference.

[0017] In an exemplary embodiment, the plurality of image frames includes M image frames, where M is a positive integer. The first calculation module includes: a first determining submodule, configured to determine a first image feature of the i-th image frame and a second image feature of the (i+1)-th image frame, where i is a positive integer greater than or equal to 1 and less than or equal to M; a calculation submodule, configured to calculate the image feature difference between the first image feature and the second image feature; and a second determining submodule, configured to determine the image feature difference as the image feature difference between the i-th image frame and the (i+1)-th image frame.

[0018] In an exemplary embodiment, the first processing unit includes: a first identification module, configured to identify a first feature point of a first target object in a first image frame; a second identification module, configured to identify a second feature point of the first target object in a second image frame, wherein the first image frame and the second image frame are adjacent image frames in the target image frame; a first determination module, configured to determine a first map point corresponding to the first target object based on the first feature point and the second feature point; and a first fusion module, configured to fuse the first image frame and the second image frame into a first sub-image based on the first map point.

[0019] In one exemplary embodiment, the second processing unit includes: a third recognition module, configured to recognize a third feature point of a second target object in a third image frame; a fourth recognition module, configured to recognize a fourth feature point of the second target object in a fourth image frame, wherein the third image frame and the fourth image frame are any two image frames in the target image frame; a second determination module, configured to determine a second map point corresponding to the second target object based on the third feature point and the fourth feature point; and a second fusion module, configured to fuse the third image frame and the fourth image frame into a second sub-image based on the second map point.

[0020] In an exemplary embodiment, the generation unit includes: a second calculation module, configured to calculate the visual overlap region between the first sub-image and the second sub-image; a third determination module, configured to determine the target transformation relationship between a first coordinate system and a second coordinate system based on the visual overlap region, wherein the first coordinate system is the coordinate system corresponding to the first sub-image, and the second coordinate system is the coordinate system corresponding to the second sub-image; and a generation module, configured to generate the target fusion sub-image based on the target transformation relationship, the first sub-image, and the second sub-image.

[0021] In an exemplary embodiment, the second calculation module includes: a search submodule, configured to search for a fifth feature point included in the first sub-image and a sixth feature point included in the second sub-image; and a third determination submodule, configured to determine the regions corresponding to the same feature points among the fifth feature points and the sixth feature points as the visual overlap region.

[0022] A third aspect of this application provides an electronic device including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the method described in the first aspect above.

[0023] A fourth aspect of this application provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the method described in the first aspect above.

[0024] A fifth aspect of this application provides a computer program product that, when run on an electronic device, causes the electronic device to perform the method described in any one of the first aspects.

[0025] The beneficial effects of this application embodiment compared with the prior art are as follows: It generates a fused sub-image based on the visual overlap area between a first sub-image generated in real-time on the front end and a second sub-image generated in non-real-time on the back end. This is achieved by performing first image processing on the target image frame to obtain the first sub-image, wherein the first image processing is performed in real-time according to the image acquisition time corresponding to the target image frame; second image processing is performed on the target image frame to obtain the second sub-image, wherein the second image processing is performed in non-real-time on all image frames in the target image frame in a random order; and a target fused sub-image is generated based on the visual overlap area between the first and second sub-images, wherein the target fused sub-image includes the visual overlap area. Since the first and second sub-images are fused based on their common area, better alignment between them is achieved, thereby effectively improving the success rate of global alignment and fusion of sub-images, and thus improving the accuracy of global positioning and modeling of the system. Attached Figure Description

[0026] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments or the prior art will be briefly introduced below.

[0027] Figure 1 This is a schematic diagram of the hardware environment of an optional image processing method according to an embodiment of this application;

[0028] Figure 2 This is a schematic flowchart of an optional image processing method according to an embodiment of this application;

[0029] Figure 3 This is a schematic flowchart of another optional image processing method according to an embodiment of this application;

[0030] Figure 4 This is a schematic diagram of a biomimetic ruminant foreground and background subgraph alignment method according to an embodiment of this application;

[0031] Figure 5 This is a schematic diagram of a biomimetic ruminant VSLAM system according to an embodiment of this application;

[0032] Figure 6 This is a structural block diagram of an optional image processing apparatus according to an embodiment of this application;

[0033] Figure 7 This is a schematic diagram of the structure of an optional electronic device according to an embodiment of this application. Detailed Implementation

[0034] In the following description, specific details such as particular system architectures and techniques are set forth for illustrative purposes and not for limitation, in order to provide a thorough understanding of the embodiments of this application. However, those skilled in the art will understand that this application may also be implemented in other embodiments without these specific details. In other instances, detailed descriptions of well-known systems, apparatuses, circuits, and methods have been omitted so as not to obscure the description of this application with unnecessary detail.

[0035] To illustrate the technical solution described in this application, specific embodiments are provided below.

[0036] It should be understood that, when used in this specification and the appended claims, the term "comprising" indicates the presence of the described features, integrals, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components and / or collections thereof.

[0037] It should also be understood that the terminology used in this specification is for the purpose of describing particular embodiments only and is not intended to limit the scope of the application. As used in this specification and the appended claims, the singular forms “a,” “an,” and “the” are intended to include the plural forms unless the context clearly indicates otherwise.

[0038] It should also be further understood that the term “and / or” as used in this application specification and the appended claims means any combination of one or more of the associated listed items and all possible combinations, and includes such combinations.

[0039] As used in this specification and the appended claims, the term "if" may be interpreted, depending on the context, as "when," "once," "in response to determination," or "in response to detection." Similarly, the phrases "if determined" or "if [described condition or event] is detected" may be interpreted, depending on the context, as "once determined," "in response to determination," "once [described condition or event] is detected," or "in response to detection of [described condition or event]."

[0040] Furthermore, in the description of this application, the terms "first," "second," "third," etc., are used only to distinguish descriptions and should not be construed as indicating or implying relative importance.

[0041] According to one aspect of the embodiments of this application, an image processing method is provided. Optionally, in this embodiment, the above-described image processing method can be applied to, for example... Figure 1 The hardware environment shown consists of terminal device 102 and server 104. For example... Figure 1 As shown, terminal device 102 is connected to server 104 via a network and can be used to provide services (such as application services) to terminal devices or clients installed on terminal devices. It can set up a database on the server or independently of the server to provide data storage services to server 104.

[0042] The aforementioned network may include, but is not limited to, at least one of the following: wired network, wireless network. The aforementioned wired network may include, but is not limited to, at least one of the following: wide area network, metropolitan area network, local area network. The aforementioned wireless network may include, but is not limited to, at least one of the following: Wi-Fi (Wireless Fidelity), Bluetooth. The terminal device 102 may be, but is not limited to, devices such as smartphones, smart computers, and smart tablets.

[0043] The image processing method of this embodiment can be executed by server 104, by terminal device 102, or by both server 104 and terminal device 102. Taking the execution of the image processing method of this embodiment by terminal device 102 as an example... Figure 2 This is a schematic flowchart of an optional image processing method according to an embodiment of this application, such as... Figure 2 As shown, the process of this method may include the following steps:

[0044] Step S202: Perform first image processing on the target image frame to obtain a first sub-image, wherein the first image processing is real-time processing according to the image acquisition time corresponding to the target image frame.

[0045] The image processing method in this embodiment can be applied to scenarios that process real-time image sequences. The real-time image sequence can be an image sequence captured by a mobile robot using an onboard sensor (e.g., an image sensor) or a real-time image sequence input by a user. In this embodiment, the type of real-time image sequence is not limited.

[0046] Optionally, the target image frame can be a target image frame obtained after filtering from a real-time image sequence. The real-time image sequence includes multiple image frames, which are arranged according to their acquisition time. Similarly, the image frames included in the target image frame are also arranged according to their acquisition time. For example, when image 1 (an example of a target image frame), image 2 (an example of a target image frame), and image 3 (an example of a target image frame) are filtered from the real-time image frame, and the acquisition time of image 1 is 12:55 on [date], the acquisition time of image 2 is 12:56 on [date], and the acquisition time of image 3 is 12:57 on [date], the arrangement order of images 1, 2, and 3 in the target image frame is image 1-image 2-image 3.

[0047] Optionally, the process of performing the first image processing on the target image frame to obtain the first sub-image can be: performing the first image processing on two or more adjacent image frames in the target image frame to obtain the first sub-image. Alternatively, the process of performing the first image processing on adjacent image frames in the target image frame to obtain the first sub-image can be: merging identical image features based on the image features included in the adjacent image frames in the target image frame to obtain the first sub-image. For example, when adjacent image frames in the target image frame both include the image feature of a utility pole, and the preceding image frame includes the upper part of the utility pole while the following image frame includes the middle part of the utility pole, these two image frames can be merged to obtain a sub-image that includes both the upper and middle parts of the utility pole.

[0048] Optionally, the first image processing described above is performed in real time according to the image acquisition time corresponding to the target image frame. That is, in the process of performing the first image processing on the target image frame to obtain the first sub-image, the first image processing is only performed between image 1 and image 2, and between image 2 and image 3, and is not performed between image 1 and image 3.

[0049] Step S204: Perform second image processing on the target image frame to obtain a second sub-image. The second image processing is non-real-time image processing performed on all image frames in the target image frame in a random order.

[0050] Since the information content of a single frame image is limited and there is no scale constraint during the sub-image alignment process, it can provide fewer sub-image alignment constraints. If the first sub-image formed after the first image processing is directly used as the target fusion sub-image, the success rate of the target fusion sub-image in the process of global sub-image alignment may be low.

[0051] Therefore, a second image processing can be performed on the target image frame at a target time to obtain a second sub-image. The target time can be the time before the first image processing is performed on the target image frame to obtain the first sub-image, or the time after the first image processing is performed on the target image frame to obtain the first sub-image, or the same time as the first image processing is performed on the target image frame to obtain the first sub-image. This embodiment does not limit this.

[0052] For example, the backend can perform subgraph modeling in parallel with the foreground, and use the constructed subgraphs as basic units for rumination back to the foreground, providing richer constraint information for the alignment of multiple subgraphs (i.e., the first subgraph) on the foreground. The foreground thread is equivalent to the digestive and chewing process of a ruminant's true stomach, serving as the main thread of the VSLAM (Visual Simultaneous Localization and Mapping) system, responsible for processing tens of frames of real-time images per second and providing real-time localization results of the image within the current subgraph. The backend thread is equivalent to the food storage organ of rumination, serving as an auxiliary thread to construct subgraph units of the stored images and send these subgraph units (i.e., the second subgraph) to the foreground. The foreground utilizes the subgraph information supplemented by the backend, constructing data associations for keyframes through scene recognition and feature point matching within each subgraph. Using the constraints formed between the foreground and backend subgraphs, multi-subgraph alignment and global optimization can be achieved through bundle optimization.

[0053] Optionally, the process of performing the second image processing on the target image frame to obtain the second sub-image can be: merging identical image features based on the image features included in any two or more image frames in the target image frame to obtain the second sub-image. This process of merging identical image features based on the image features included in any two or more image frames in the target image frame to obtain the second sub-image is similar to the process of merging identical image features based on the image features included in adjacent image frames in the target image frame to obtain the first sub-image, and will not be elaborated further in this embodiment.

[0054] Optionally, the above-mentioned second image processing is non-real-time image processing performed on all image frames in the target image frame in a random order. That is, in the process of performing the second image processing on the target image frame to obtain the second sub-image, the second image processing can be performed not only between image 1 and image 2, and between image 2 and image 3, but also between image 1 and image 3. In other words, the second image processing can be performed on all image frames in the target image frame in a random order. If the target image frame includes four images: image 1, image 2, image 3, and image 4, then the second image processing must be performed on image 1 and image 2, image 1 and image 3, image 1 and image 4, image 2 and image 3, image 2 and image 4, and image 3 and image 4.

[0055] Step S206: Generate a target fusion sub-image based on the visual overlap area between the first sub-image and the second sub-image, wherein the target fusion sub-image includes the visual overlap area.

[0056] In this embodiment, after obtaining the first sub-image and the second sub-image, a target fusion sub-image can be generated based on the visual overlap area between the first sub-image and the second sub-image.

[0057] Optionally, before generating the target fused sub-image based on the visual overlap area between the first and second sub-images, the visual overlap area between the first and second sub-images can be determined. The process of determining the visual overlap area between the first and second sub-images can be as follows: first, find the same feature information between the first and second sub-images, and then determine the area corresponding to the same feature information as the visual overlap area.

[0058] Optionally, the process of generating a target fusion sub-image based on the visual overlap area of ​​the first sub-image and the second sub-image can be as follows: first, calculate the transformation relationship between the first sub-image and the second sub-image based on the position of the visual overlap area in the first sub-image and the position of the visual overlap area in the second sub-image; then, based on the above transformation relationship, transfer all features contained in the first sub-image to the second sub-image (and transfer all features contained in the second sub-image to the first sub-image) to obtain the target fusion sub-image.

[0059] Optionally, the target fusion sub-image mentioned above includes not only all feature information of the first sub-image but also all feature information of the second sub-image. For example, when the first sub-image includes feature information A1, A2, and A3, and the second sub-image includes feature information A3, A4, and A5, the region corresponding to feature information A3 is the aforementioned visual overlap region, and the generated target fusion sub-image includes feature information A1, A2, A3, A4, and A5.

[0060] Through steps S202 to S206, unlike existing time-series image processing methods, a biomimetic rumination mechanism is proposed. This mechanism performs secondary filtering of keyframe images in the background and constructs a background subgraph (i.e., a second subgraph) non-real-time, providing supplementary information for subgraph alignment to the real-time foreground thread. Due to the establishment of this parallel mechanism, the system's real-time performance is guaranteed. Meanwhile, the background can provide sufficient processing time for images, acquiring information that is unavailable under real-time requirements, further enriching the environment model, improving subgraph alignment in multi-subgraph VSLAM, and effectively increasing the success rate of global subgraph alignment and fusion, thereby improving the accuracy of global localization and modeling.

[0061] In one exemplary embodiment, before performing the first image processing on the target image frame, the method further includes: acquiring a real-time image sequence, the real-time image sequence including multiple image frames; and selecting the target image frame from the multiple image frames.

[0062] In this embodiment, before performing the first image processing on the target image frame, the target image frame can be selected from the real-time image sequence. Optionally, the process of selecting the target image frame from the real-time image sequence can be: first, obtain the real-time image sequence, which includes multiple image frames, and then select the target image frame from the multiple image frames.

[0063] Optionally, the process of obtaining the real-time image sequence described above can be: using an image acquisition component to acquire images of the area to be acquired in real time to obtain a real-time image sequence; the image acquisition component can be a camera or a thermal imager; or, acquiring a real-time image sequence input by the user.

[0064] Optionally, the process of selecting the target image frame from multiple image frames described above can be: identifying the image features of each image frame in the multiple image frames, and selecting the target image frame from the multiple image frames based on the identified image features; alternatively, it can be obtaining one image frame from the multiple image frames after a preset time period to obtain the target image frame; or it can be obtaining one image frame from the multiple image frames after a preset number of times to obtain the target image frame. This embodiment does not limit this. For example, when the preset time is 1 second, an image frame can be obtained every 1 second and used as the target image frame. Or, when the preset number is 4, an image frame can be obtained every 4 image frames and used as the target image frame.

[0065] In this embodiment, the target image frame is selected from the real-time image sequence, and then the first image processing is performed on the target image frame, instead of directly performing the first image processing on all image frames included in the real-time image sequence. This reduces the number of image frames that need to be processed and improves the efficiency of generating the first sub-image.

[0066] In one exemplary embodiment, selecting a target image frame from multiple image frames includes: calculating the image feature difference between every two adjacent image frames in the multiple image frames based on the image features of each image frame in the multiple image frames; and selecting the target image frame from the real-time image sequence based on the image feature difference.

[0067] Optionally, in this embodiment, the process of calculating the image feature differences between any two adjacent image frames based on the image features of each image frame in the multiple image frames can be as follows: a bag-of-words global descriptor for the image is established using image feature points, and then the overall differences between the images are calculated using the global descriptor. These image feature points are used to identify image features in the image frames. For example, when a telephone pole exists in an image frame, the location of the telephone pole can be determined as an image feature point.

[0068] After calculating the image feature differences between any two adjacent image frames in a series of image frames, target image frames can be selected from the real-time image sequence based on these differences. Optionally, the process of selecting target image frames from the real-time image sequence based on image feature differences can be as follows: if the image feature difference is greater than or equal to a feature difference threshold, the image frame is determined as the target image frame. For example, when the image feature difference between image 3 and image 4 is 3, and the feature difference threshold is 2, image 4 can be determined as the target image frame; that is, the image is selected as a keyframe only when the image feature difference is sufficiently large.

[0069] In this embodiment, target image frames are selected from the real-time image sequence based on the differences in image features between adjacent image frames. This improves the accuracy of the identified target image frames, as well as the accuracy of the generated first and second sub-images. Consequently, it increases the success rate of global alignment and fusion of sub-images, thereby improving the accuracy of global positioning and modeling of the system.

[0070] In one exemplary embodiment, the plurality of image frames includes M image frames, where M is a positive integer. Based on the image features of each image frame in the plurality of image frames, the image feature difference between every two adjacent image frames in the plurality of image frames is calculated, including: determining the first image feature of the i-th image frame and the second image feature of the (i+1)-th image frame, where i is a positive integer greater than or equal to 1 and less than or equal to M; calculating the image feature difference between the first image feature and the second image feature; and determining the image feature difference as the image feature difference between the i-th image frame and the (i+1)-th image frame.

[0071] In this embodiment, when the multiple image frames include M image frames, the first image feature of the i-th image frame and the second image feature of the (i+1)-th image frame can be determined, where i is a positive integer greater than or equal to 1 and less than or equal to M. Optionally, determining the first image feature of the i-th image frame and the second image feature of the (i+1)-th image frame can be achieved by: establishing a bag-of-words global descriptor for the i-th image frame and the (i+1)-th image frame using image feature points, and determining the global descriptor for the i-th image frame and the (i+1)-th image frame. The bag-of-words can refer to a pre-established bag-of-words model that includes multiple terms and their corresponding descriptors, and can describe any image through its constituent terms. The global descriptor refers to a feature vector that can describe the global features of an image.

[0072] After determining the global descriptor of the i-th image frame and the global descriptor of the (i+1)-th image frame, the overall difference between the i-th image frame and the (i+1)-th image frame can be calculated based on the global descriptors of the two images, and this overall difference can be determined as the image feature difference between the i-th image frame and the (i+1)-th image frame.

[0073] In this embodiment, target image frames are selected from a real-time image sequence based on the differences in image features between adjacent image frames, which can improve the accuracy of the identified target image frames.

[0074] In an exemplary embodiment, performing a first image processing step on a target image frame to obtain a first sub-image includes: identifying a first feature point of a first target object in the first image frame; identifying a second feature point of the first target object in a second image frame, wherein the first image frame and the second image frame are adjacent image frames in the target image frame; determining a first map point corresponding to the first target object based on the first feature point and the second feature point; and fusing the first image frame and the second image frame into a first sub-image based on the first map point.

[0075] In this embodiment, a first feature point of a first target object in a first image frame can be identified. The target image frame includes the first image frame, and the first target object can be an object included in the first image frame and a second image frame. The process of identifying the first feature point of the first target object in the first image frame can be: performing image recognition on the first image frame to determine whether a first target object exists in the first image frame; if a first target object exists in the first image frame, determining the feature point corresponding to the first target object. For example, if a utility pole is included in the first image frame, the utility pole can be identified as the first target object, and the feature point of the utility pole can be determined.

[0076] After identifying the first feature point of the first target object in the first image frame, the second feature point of the first target object in the second image frame can be identified. The first and second image frames are adjacent image frames in the target image frame. That is, when the image frame sequence in the target image frame is image 1-image 2-image 3, the third and fourth image frames can only be image 1 and image 2, and image 2 and image 3, but not image 1 and image 3 (image 1 and image 3 are not adjacent).

[0077] Optionally, the process of identifying the second feature point of the first target object in the second image frame is similar to the process of identifying the first feature point of the first target object in the first image frame, and this embodiment does not limit it.

[0078] Optionally, the process of determining the first map point corresponding to the first target object based on the first feature point and the second feature point can be as follows: establishing a dual-view relationship between the first image frame and the second image frame based on the first feature point and the second feature point, thereby determining the first map point corresponding to the first target object. For example, feature points between multiple keyframes can be matched to establish a dual-view relationship of the image based on the matched feature points, thereby calculating the spatial map points of the image planar feature points.

[0079] After determining the first map point, the first image frame and the second image frame can be merged into a first sub-image based on the first map point. Optionally, the process of merging the first image frame and the second image frame into a first sub-image can be as follows: based on the first map point, determine the transformation relationship between the first image frame and the second image frame, and then use the above transformation relationship to transform the first feature points included in the first image frame into the second image frame to obtain the first sub-image. For example, since the sub-image is composed of keyframes and map points calculated from the corresponding keyframes, a new sub-image can be established during the system initialization phase; a new sub-image can also be established when the system cannot establish feature point matching between keyframes. Furthermore, after establishing a new sub-image, all keyframes and map points are expressed in the local coordinate system of this sub-image.

[0080] It should be noted that there may be multiple target objects in a single keyframe, which in turn may result in multiple feature points. Therefore, there may be multiple map points in a single keyframe.

[0081] In this embodiment, map points corresponding to the same object are generated based on feature points of the same object included in two adjacent image frames, and then a sub-map is generated based on the generated map points, which can improve the accuracy of the generated sub-map.

[0082] In one exemplary embodiment, performing a second image processing step on a target image frame to obtain a second sub-image includes: identifying a third feature point of a second target object in a third image frame; identifying a fourth feature point of the second target object in a fourth image frame, wherein the third image frame and the fourth image frame are any two image frames in the target image frame; determining a second map point corresponding to the second target object based on the third feature point and the fourth feature point; and fusing the third image frame and the fourth image frame into a second sub-image based on the second map point.

[0083] Optionally, the process of identifying the third feature point of the second target object in the third image frame and the fourth feature point of the second target object in the fourth image frame is similar to the process of identifying the first feature point of the first target object in the first image frame, and will not be described again in this embodiment.

[0084] Optionally, the second target object may be an object included in the third and fourth image frames. The second target object and the first target object may be the same object or different objects. This embodiment does not limit this.

[0085] Optionally, the third and fourth image frames can be any two image frames in the target image frame. That is, when the image frame sequence in the target image frame is image 1-image 2-image 3, the third and fourth image frames can be image 1 and image 2, image 2 and image 3, or image 1 and image 3.

[0086] For example, by utilizing a non-real-time subgraph in the background (i.e., the second subgraph), some map point constructions that the real-time subgraph in the foreground (i.e., the first subgraph) cannot achieve are implemented. Because the background is non-real-time, it can spend more time calculating feature point matches between all images, thereby building more map points. Thus, the subgraph built in the background can supplement the information lacking in the foreground subgraph, establishing data associations (i.e., feature point matching) that the foreground subgraph cannot establish, thereby utilizing these surplus data associations to achieve effective subgraph alignment and fusion. Furthermore, since the foreground subgraph (i.e., the first subgraph) is built in real-time, its feature point matching is performed according to a time series, making it sequential and local. The background, being non-real-time, can traverse all keyframes to find more data matches, thus being non-sequential, global, and possessing more subgraph information.

[0087] Optionally, the process of determining the second map point corresponding to the second target object based on the third and fourth feature points is similar to the process of determining the first map point corresponding to the first target object based on the first and second feature points, and will not be described again in this embodiment. Similarly, the process of fusing the third and fourth image frames into a second sub-image based on the second map point is similar to the process of fusing the first and second image frames into a first sub-image based on the first map point, and will not be described again in this embodiment.

[0088] In this embodiment, map points corresponding to the same object are generated based on feature points of the same object included in two adjacent image frames. Then, a sub-map is generated based on the generated map points, which can improve the comprehensiveness of the generated sub-map and thus improve the accuracy of the generated target fusion sub-map.

[0089] In an exemplary embodiment, generating a target fusion sub-image based on the visual overlap area between a first sub-image and a second sub-image includes: calculating the visual overlap area between the first sub-image and the second sub-image; determining a target transformation relationship between a first coordinate system and a second coordinate system based on the visual overlap area, wherein the first coordinate system is the coordinate system corresponding to the first sub-image and the second coordinate system is the coordinate system corresponding to the second sub-image; and generating the target fusion sub-image based on the target transformation relationship, the first sub-image, and the second sub-image.

[0090] In this embodiment, since each subgraph is an independent coordinate system, subgraph alignment and fusion are required to establish a globally unified coordinate system for the environment representation.

[0091] Optionally, the visual overlap region between the first and second sub-images can be calculated first, and then the target fused sub-image can be generated based on this visual overlap region. The process of calculating the visual overlap region between the first and second sub-images can be as follows: use feature point matching to search for common regions between the sub-images to obtain the visual overlap region.

[0092] Optionally, after determining the visual overlap region, the target transformation relationship between the first coordinate system and the second coordinate system can be determined based on the visual overlap region. The first coordinate system corresponds to the first sub-image, and the second coordinate system corresponds to the second sub-image. The process of determining the target transformation relationship between the first and second coordinate systems based on the visual overlap region can be as follows: first, determine the coordinates of the visual overlap region in the first coordinate system and the coordinates of the visual overlap region in the second coordinate system; then, determine the target transformation relationship between the first and second coordinate systems based on these two coordinates. For example, after determining the common region between the first and second sub-images, the transformation relationship between the coordinate systems of the two sub-images can be calculated using this common region.

[0093] After determining the target transformation relationship, a target fusion sub-map can be generated based on the target transformation relationship, the first sub-map, and the second sub-map. Optionally, the first and second sub-maps can be transformed to the same coordinate system based on the target transformation relationship to obtain the target fusion sub-map. For example, this transformation relationship can be used to transform keyframes and map points located in two different sub-map coordinate systems to the same coordinate system, thereby achieving sub-map alignment and fusion.

[0094] In this embodiment, based on the visual overlap area between the first and second sub-images, the transformation relationship between the coordinate systems of the first and second sub-images is determined. Finally, based on the above coordinate transformation relationship, the keyframes and map points of the first sub-image are transformed to the coordinate system of the second sub-image to obtain the target fused sub-image. This allows the target fused sub-image to fully retain the features of both the first and second sub-images, thereby effectively improving the success rate of global alignment and fusion of sub-images, and thus improving the accuracy of global positioning and modeling of the system.

[0095] In an exemplary embodiment, calculating the visual overlap region between the first sub-image and the second sub-image includes: searching for a fifth feature point included in the first sub-image and searching for a sixth feature point included in the second sub-image; and determining the regions corresponding to the same feature points among the fifth and sixth feature points as the visual overlap region.

[0096] In this embodiment, optionally, during the calculation of the visual overlap region between the first sub-image and the second sub-image, the fifth feature point included in the first sub-image and the sixth feature point included in the second sub-image can be searched first. It should be noted that the processes of searching for the fifth feature point included in the first sub-image and searching for the sixth feature point included in the second sub-image can be performed simultaneously or not simultaneously. For example, the first sub-image can be searched first to determine its included feature points, and then the second sub-image can be searched to determine its included feature points.

[0097] Optionally, after determining the fifth and sixth feature points, the regions corresponding to the same feature points among the fifth and sixth feature points can be defined as visually overlapping regions. For example, when feature points B1, B2, and B3 exist in the first sub-image, and feature points B3, B4, and B5 exist in the second sub-image, the region where feature point B3 is located can be defined as a visually overlapping region.

[0098] In this embodiment, the region corresponding to the common feature points among the feature points included in the first sub-image and the second sub-image is identified as the visual overlap region, which can improve the accuracy of the identified visual overlap region and thus improve the accuracy of the generated target fusion sub-image.

[0099] The image processing method in this application embodiment will be explained below with reference to optional examples. In this optional example, the first sub-image is the foreground sub-image, and the second sub-image is the background sub-image.

[0100] Mobile robots in related technologies can use onboard sensors to perceive the surrounding environment in real time, plan paths, and move safely to a designated location to complete specific tasks. There are three problems to solve when a mobile robot moves from its current location to a destination in an unknown environment: (1) environmental perception, i.e., environmental modeling; (2) determining its position relative to the environment, i.e., localization; and (3) determining the method of reaching the target point from the current location, i.e., motion planning. However, environmental modeling and localization in related technologies still exhibit insufficient robustness and cannot operate continuously, making it difficult to meet the practical application requirements of robot navigation.

[0101] In related technologies, in order to solve the above problems, the method of constructing multiple subgraphs to maintain system operation is generally adopted. However, this method has a low success rate of global alignment of subgraphs, which can easily lead to failure of global system localization and modeling.

[0102] This optional example provides a biomimetic ruminant visual SLAM system method, applicable to visual navigation and environmental modeling and localization of mobile robots. It can solve the data association problem required for multi-subgraph VSLAM systems, as well as the problem of how to reasonably utilize images to achieve effective image matching for data association used for subgraph alignment.

[0103] Based on the characteristic that multi-subgraph alignment does not have temporal requirements, and drawing inspiration from the digestive process of ruminants, this optional example proposes an image rumination mechanism. Based on relevant biomimetic definitions, a front-end and back-end parallel tracking modeling architecture is constructed. Referring to the digestive process of ruminants in nature and the VSLAM workflow, the relevant biomimetic definitions of rumination digestion and the VSLAM process are established as follows:

[0104] Combination Figure 3 As shown, the image processing method in this optional example may include the following steps:

[0105] Step S302: Use the real-time image as the initial food input for ruminant digestion.

[0106] The input real-time image data stream is divided into a front-end data stream and a back-end data stream. The front-end data stream is sent to the real-time visual tracking and modeling thread to achieve real-time image tracking.

[0107] Step S304: Filter through the real-time images to determine the keyframe images.

[0108] The above image allocation is regarded as drainage and diversion, which is also the VSLAM keyframe selection.

[0109] Step S306: Based on the keyframes, generate a foreground sub-image in real time in the foreground and a background sub-image in non-real time in the background.

[0110] The construction of subgraphs in the background is like food storage and pre-digestion, which is also the VSLAM parallel tracking graph construction. Multiple subgraphs are built in the foreground to maintain system operation, and the saved images are used in the background to supplement the construction of subgraphs. Otherwise, the images are directly discarded to control the amount of system data.

[0111] Step S308: Merge the front-end and back-end subgraphs to obtain a merged subgraph.

[0112] The aforementioned foreground-background subgraph fusion is equivalent to food reflux and true stomach digestion, which is also a global optimization. The subgraph constructed in the background is sent back to the foreground real-time thread for unified management; this process is equivalent to the re-chewing process in the digestive mechanism. The returned subgraph is then fused with the foreground tracking model, which is equivalent to the true stomach digestion in the rumination process. Based on the returned supplementary subgraph, visual overlap areas between it and existing foreground subgraphs are identified. Alignment constraints between independent foreground subgraphs are then constructed using the supplementary subgraphs, achieving a globally consistent representation among all independent subgraphs.

[0113] Combination Figure 4 As shown, this optional embodiment also provides a biomimetic ruminant foreground / background subgraph alignment method. The upper half of the figure represents the foreground thread, schematically containing two independent subgraphs of different colors; the lower half represents the background thread, containing one subgraph. The background subgraph has a portion of its map overlapping with the foreground subgraph, represented by small dots of the same color as the foreground thread subgraph. Subgraph alignment is achieved by utilizing this overlapping portion of the map.

[0114] For the background data stream, supplementary subgraphs are constructed using the saved images. These subgraphs, built in the background, are then sent back to the real-time foreground thread for unified management; this process is equivalent to the re-chewing process in a digestive mechanism. The returned subgraphs are then integrated with the foreground tracking modeling, representing the true digestion process. Based on the returned supplementary subgraphs, data associations are established using image descriptors of map points to identify visual overlap areas between the background and existing foreground subgraphs. Furthermore, alignment constraints between independent foreground subgraphs are constructed using the supplementary subgraphs for optimization, achieving a globally consistent representation among all independent subgraphs.

[0115] Combination Figure 5As shown, this optional embodiment also provides a biomimetic ruminant VSLAM system. The system includes modules for keyframe selection, real-time foreground tracking and mapping, multi-subgraph alignment and optimization, and background non-real-time tracking and mapping. The keyframe selection module in the foreground thread splits the image stream, distributing it to both the foreground and background threads. Driven by foreground image data, the foreground thread performs real-time visual tracking and localization; the background thread performs non-real-time visual reconstruction and supplementary subgraph construction based on the background image data stream. After the background thread completes subgraph construction, it utilizes the supplementary subgraph information to build data associations between keyframes through scene recognition and feature point matching within each subgraph. Using the constraints between the foreground and background subgraphs, multi-subgraph alignment and global optimization can be achieved through bundle optimization in the background non-real-time tracking and mapping stage. The "multi-subgraph alignment and optimization" module optimizes and globally aligns each subgraph, resulting in a final globally consistent environment representation.

[0116] It should be understood that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0117] Corresponding to the image processing method described in the above embodiments, Figure 6 A structural block diagram of an image processing apparatus provided in an embodiment of this application is shown. For ease of explanation, only the parts related to the embodiments of this application are shown.

[0118] According to another aspect of the embodiments of this application, an image processing apparatus for implementing the above-described image processing method is also provided. Figure 6 This is a structural block diagram of an optional image processing apparatus according to an embodiment of this application, such as... Figure 6 As shown, the device may include:

[0119] The first processing unit 602 is used to perform first image processing on the target image frame to obtain a first sub-image, wherein the first image processing is real-time processing according to the image acquisition time corresponding to the target image frame;

[0120] The second processing unit 604 is connected to the first processing unit 602 and is used to perform second image processing on the target image frame to obtain a second sub-image. The second image processing is non-real-time image processing performed on all image frames in the target image frame in a random order.

[0121] The generation unit 606, connected to the second processing unit 604, is used to generate a target fusion sub-image based on the visual overlap area between the first sub-image and the second sub-image, wherein the target fusion sub-image includes the visual overlap area.

[0122] It should be noted that the first processing unit 602 in this embodiment can be used to execute the above step S202, the second processing unit 604 in this embodiment can be used to execute the above step S204, and the generation unit 606 in this embodiment can be used to execute the above step S206.

[0123] Through the above modules, a first sub-image is obtained by performing first image processing on the target image frame, wherein the first image processing is real-time processing according to the image acquisition time corresponding to the target image frame; a second image processing is performed on the target image frame to obtain a second sub-image, wherein the second image processing is non-real-time image processing performed on all image frames in the target image frame in a random order; a target fusion sub-image is generated based on the visual overlap area between the first sub-image and the second sub-image, wherein the target fusion sub-image includes the visual overlap area, which effectively improves the success rate of global sub-image alignment, thereby improving the accuracy of global positioning and modeling of the system.

[0124] In one exemplary embodiment, the above-described apparatus further includes:

[0125] The acquisition unit is used to acquire a real-time image sequence, which includes multiple image frames, before performing the first image processing on the target image frame;

[0126] The filtering unit is used to select the target image frame from multiple image frames.

[0127] In one exemplary embodiment, the filtering unit includes:

[0128] The first calculation module is used to calculate the difference in image features between any two adjacent image frames in the multiple image frames based on the image features of each image frame in the multiple image frames.

[0129] The filtering module is used to filter out target image frames from a real-time image sequence based on differences in image features.

[0130] In one exemplary embodiment, the plurality of image frames includes M image frames, where M is a positive integer, and the first calculation module includes:

[0131] The first determining submodule is used to determine the first image feature of the i-th image frame and the second image feature of the (i+1)-th image frame, where i is a positive integer greater than or equal to 1 and less than or equal to M;

[0132] The calculation submodule is used to calculate the image feature differences between the first image feature and the second image feature;

[0133] The second determining submodule is used to determine the image feature difference as the image feature difference between the i-th image frame and the (i+1)-th image frame.

[0134] In one exemplary embodiment, the first processing unit includes:

[0135] The first recognition module is used to recognize the first feature points of the first target object in the first image frame;

[0136] The second recognition module is used to recognize the second feature points of the first target object in the second image frame, wherein the first image frame and the second image frame are adjacent image frames in the target image frame;

[0137] The first determining module is used to determine the first map point corresponding to the first target object based on the first feature point and the second feature point;

[0138] The first fusion module is used to fuse the first image frame and the second image frame into a first sub-image based on the first map point.

[0139] In one exemplary embodiment, the second processing unit includes:

[0140] The third recognition module is used to recognize the third feature points of the second target object in the third image frame;

[0141] The fourth recognition module is used to recognize the fourth feature point of the second target object in the fourth image frame, wherein the third image frame and the fourth image frame are any two image frames in the target image frame;

[0142] The second determining module is used to determine the second map point corresponding to the second target object based on the third feature point and the fourth feature point.

[0143] The second fusion module is used to fuse the third and fourth image frames into a second sub-image based on the second map points.

[0144] In one exemplary embodiment, the generation unit includes:

[0145] The second calculation module is used to calculate the visual overlap area between the first sub-image and the second sub-image;

[0146] The third determining module is used to determine the target transformation relationship between the first coordinate system and the second coordinate system based on the visual overlap area, wherein the first coordinate system is the coordinate system corresponding to the first sub-image, and the second coordinate system is the coordinate system corresponding to the second sub-image;

[0147] The generation module is used to generate a target fusion subgraph based on the target transformation relationship, the first subgraph, and the second subgraph.

[0148] In one exemplary embodiment, the second computing module includes:

[0149] The search submodule is used to search for the fifth feature point included in the first subgraph and the sixth feature point included in the second subgraph.

[0150] The third determination submodule is used to determine the regions corresponding to the same feature points in the fifth and sixth feature points as visual overlap regions.

[0151] It should be noted that the information interaction and execution process between the above-mentioned devices / units are based on the same concept as the method embodiments of this application. For details on their specific functions and technical effects, please refer to the method embodiments section, and they will not be repeated here.

[0152] According to another aspect of the embodiments of this application, a storage medium is also provided. Optionally, in this embodiment, the storage medium can be used to execute program code for any of the image processing methods described above in the embodiments of this application.

[0153] Optionally, in this embodiment, the storage medium may be located on at least one of the network devices in the network shown in the above embodiment.

[0154] Optionally, in this embodiment, the storage medium is configured to store program code for executing steps S202, S204, and S206.

[0155] Optionally, specific examples in this embodiment can refer to the examples described in the above embodiments, and will not be repeated in this embodiment.

[0156] Optionally, in this embodiment, the storage medium may include, but is not limited to, various media capable of storing program code, such as USB flash drives, ROMs, RAMs, portable hard drives, magnetic disks, or optical disks.

[0157] Figure 7 This is a schematic diagram of an optional electronic device according to an embodiment of this application. The electronic device may be a desktop computer, laptop, handheld computer, cloud server, or other computing device.

[0158] like Figure 7 As shown, the electronic device of this embodiment includes: a processor 11, a memory 12, and a computer program 13 stored in the memory 12 and executable on the processor 11. When the processor 11 executes the computer program 13, it implements steps S202, S204, and S206 in the above-described image processing method embodiment.

[0159] For example, the computer program 13 may be divided into one or more modules / units, which are stored in the memory 12 and executed by the processor 11 to complete this application. The one or more modules / units may be a series of computer program instruction segments capable of performing a specific function, which describe the execution process of the computer program 13 in the electronic device.

[0160] Those skilled in the art will understand that Figure 7 This is merely an example of an electronic device and does not constitute a limitation on the electronic device. It may include more or fewer components than illustrated, or combine certain components, or different components. For example, the electronic device may also include input / output devices, network access devices, buses, etc.

[0161] The processor 11 can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or any conventional processor.

[0162] The memory 12 can be an internal storage unit of the electronic device, such as a hard drive or RAM. The memory 12 can also be an external storage device of the electronic device, such as a plug-in hard drive, Smart Media Card (SMC), Secure Digital (SD) card, or Flash Card. Furthermore, the memory 12 can include both internal and external storage units. The memory 12 is used to store the computer program and other programs and data required by the electronic device. The memory 12 can also be used to temporarily store data that has been output or will be output.

[0163] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is merely an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiments can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit. Furthermore, the specific names of the functional units and modules are only for easy differentiation and are not intended to limit the scope of protection of this application. The specific working process of the units and modules in the above system can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0164] In the above embodiments, the descriptions of each embodiment have different focuses. For parts that are not described in detail or recorded in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0165] In the embodiments provided in this application, it should be understood that the disclosed devices / electronic devices and methods can be implemented in other ways. For example, the device / electronic device embodiments described above are merely illustrative. For instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the displayed or discussed mutual couplings or direct couplings or communication connections may be through some interfaces; indirect couplings or communication connections between devices or units may be electrical, mechanical, or other forms.

[0166] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0167] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0168] If an integrated module / unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the methods of the above embodiments can also be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. The computer-readable medium can include: any entity or device capable of carrying the computer program code, a recording medium, a USB flash drive, a portable hard drive, a magnetic disk, an optical disk, a computer memory, a read-only memory (ROM), a random access memory (RAM), an electrical carrier signal, a telecommunication signal, and a software distribution medium, etc.

[0169] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0170] The above-described embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application, and should all be included within the protection scope of this application.

Claims

1. An image processing method, characterized in that, include: Perform a first image processing on the target image frame to obtain a first sub-image, wherein the first image processing is real-time processing according to the image acquisition time corresponding to the target image frame; A second image processing is performed on the target image frame to obtain a second sub-image, wherein the second image processing is a non-real-time image processing performed on all image frames in the target image frame in a random order; A target fusion sub-image is generated based on the visual overlap area between the first sub-image and the second sub-image, wherein the target fusion sub-image includes the visual overlap area; The first image processing step of performing the first image processing on the target image frame to obtain the first sub-image includes: Identify the first feature point of the first target object in the first image frame; Identify a second feature point of the first target object in the second image frame, wherein the first image frame and the second image frame are adjacent image frames in the target image frame; Based on the first feature point and the second feature point, determine the first map point corresponding to the first target object; Based on the first map point, the first image frame and the second image frame are merged into the first sub-image.

2. The method according to claim 1, characterized in that, Before performing the first image processing on the target image frame, the method further includes: Acquire a real-time image sequence, which includes multiple image frames; The target image frame is selected from the plurality of image frames.

3. The method as described in claim 2, characterized in that, The step of selecting the target image frame from the plurality of image frames includes: Based on the image features of each of the plurality of image frames, calculate the difference in image features between any two adjacent image frames in the plurality of image frames; The target image frame is selected from the real-time image sequence based on the differences in image features.

4. The method as described in claim 3, characterized in that, The plurality of image frames includes M image frames, where M is a positive integer. The step of calculating the image feature difference between any two adjacent image frames based on the image features of each image frame in the plurality of image frames includes: Determine the first image feature of the i-th image frame and the second image feature of the (i+1)-th image frame, where i is a positive integer greater than or equal to 1 and less than or equal to M; Calculate the image feature difference between the first image feature and the second image feature; The image feature difference is defined as the image feature difference between the i-th image frame and the (i+1)-th image frame.

5. The method according to claim 1, characterized in that, The second image processing step, which involves performing a second image processing step on the target image frame to obtain a second sub-image, includes: Identify the third feature point of the second target object in the third image frame; Identify the fourth feature point of the second target object in the fourth image frame, wherein the third image frame and the fourth image frame are any two image frames in the target image frame; Based on the third feature point and the fourth feature point, determine the second map point corresponding to the second target object; Based on the second map point, the third image frame and the fourth image frame are merged into the second sub-image.

6. The method according to claim 1, characterized in that, The step of generating a target fusion sub-image based on the visual overlap area between the first sub-image and the second sub-image includes: Calculate the visual overlap region between the first sub-image and the second sub-image; Based on the visual overlap area, the target transformation relationship between the first coordinate system and the second coordinate system is determined, wherein the first coordinate system is the coordinate system corresponding to the first sub-image, and the second coordinate system is the coordinate system corresponding to the second sub-image; The target fusion subgraph is generated based on the target transformation relationship, the first subgraph, and the second subgraph.

7. The method according to claim 6, characterized in that, The calculation of the visual overlap region between the first sub-image and the second sub-image includes: Search for the fifth feature point included in the first subgraph and search for the sixth feature point included in the second subgraph; The regions corresponding to the same feature points in the fifth and sixth feature points are defined as the visual overlap regions.

8. An image processing apparatus, characterized in that, include: The first processing unit is configured to perform first image processing on the target image frame to obtain a first sub-image, wherein the first image processing is real-time processing according to the image acquisition time corresponding to the target image frame; The second processing unit is used to perform a second image processing on the target image frame to obtain a second sub-image, wherein the second image processing is a non-real-time image processing performed on all image frames in the target image frame in a random order; A generation unit is configured to generate a target fusion sub-image based on the visual overlap region between the first sub-image and the second sub-image, wherein the target fusion sub-image includes the visual overlap region; The first processing unit includes: The first recognition module is used to recognize the first feature points of the first target object in the first image frame; The second recognition module is used to recognize the second feature points of the first target object in the second image frame, wherein the first image frame and the second image frame are adjacent image frames in the target image frame; The first determining module is used to determine the first map point corresponding to the first target object based on the first feature point and the second feature point; The first fusion module is used to fuse the first image frame and the second image frame into a first sub-image based on the first map point.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the method as described in any one of claims 1 to 7.

10. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the method as described in any one of claims 1 to 7.

Images