Multithread-based panoramic image generation method, device and chip

Abstract

The present disclosure relates to the field of panoramic photography and synthesis technology, and specifically to a method, device and chip for generating panoramic images based on multithreading. The method uses multiple threads and shared memory to achieve efficient panoramic image synthesis. The first thread is responsible for generating and storing the original captured image and its description information in the first memory and the designated storage space. The second thread periodically detects the description information in the first memory, preprocesses the existing original captured images, generates first temporary data or preprocessed images, and after all the original captured images are processed, integrates these data into second temporary data, so as to finally combine the image data of the current panoramic task with the second temporary data to generate the final panoramic image. This method reduces the waiting time by processing the shooting part and the front-end processing part in parallel through multithreading, effectively improves the efficiency and response speed of panoramic image generation, shortens the panoramic synthesis time, and improves the user experience.

Description

Technical Field

[0001] The present disclosure relates to the technical field of panoramic photography and synthesis, and in particular to a method, device and chip for generating panoramic images based on multi-threading. Background Art

[0002] In the current technological context, panoramic photography and synthesis technology has been widely used in various shooting scenes, especially in the field of drone photography. It has become an important shooting method to achieve 360° panorama (spherical model) or wide-angle panorama (flat model) through multi-angle photo synthesis. However, traditional panoramic synthesis technology faces a series of challenges and limitations.

[0003] First, the panorama synthesis process is complex, including taking photos, matching feature points, global optimization of camera pose, and image fusion. These steps need to be performed sequentially in the current solution and are highly dependent on the precise alignment and stitching of images. In particular, the global optimization of camera pose, as a key link to ensure the quality of the panorama, usually needs to be started after all pictures are taken, which prolongs the synthesis processing time.

[0004] Secondly, as the number of images increases, especially when high-resolution images are used, the amount of computation required for panorama synthesis increases dramatically, and the processing time increases significantly. This problem is particularly prominent on platforms with limited computing performance, such as drones, where the panorama synthesis time may be as long as tens of seconds, which not only delays precious shooting opportunities, but also seriously affects the user experience.

[0005] How to shorten the synthesis time of panoramic images, improve synthesis efficiency, and thus enhance user experience is an urgent problem to be solved. Summary of the invention

[0006] In order to solve the problems in the related art, the embodiments of the present disclosure provide a panoramic image generation method, device and chip based on multi-threading.

[0007] In a first aspect, an embodiment of the present disclosure provides a method for generating a panoramic image based on multithreading, wherein the multithreading includes: a first thread and a second thread, wherein the first thread and the second thread share a first memory; the method includes:

[0008] When generating a panoramic image of a current panoramic task, generating an original captured image of the current panoramic task through the first thread, and storing description information of the original captured image in the first memory;

[0009] detecting, by the second thread based on a first preset detection period, whether the first memory contains description information of the original captured image, and if so, obtaining the original captured image according to the description information of the original captured image, performing preprocessing and feature extraction on the original captured image, and generating first temporary data, or generating a preprocessed image and first temporary data corresponding to the original captured image;

[0010] The second thread is used to detect whether all the original captured images of the current panoramic task have been processed. If so, second temporary data corresponding to the current panoramic task is generated based on the first temporary data of all the original captured images of the current panoramic task, so as to generate a panoramic image of the current panoramic task based on the image of the current panoramic task and the second temporary data, wherein the image of the current panoramic task includes: all the original captured images of the current panoramic task or preprocessed images corresponding to all the original captured images of the current panoramic task.

[0011] According to an embodiment of the present disclosure, the multithreading further includes: a third thread, the second thread and the third thread share a second memory, and the generating of the panoramic image of the current panoramic task according to the image of the current panoramic task and the second temporary data includes:

[0012] storing the second temporary data to the second memory through the second thread;

[0013] The third thread detects whether the second temporary data exists in the second memory based on a second preset detection period. If so, a panoramic image of the current panoramic task is generated based on the image of the current panoramic task and the second temporary data.

[0014] According to an embodiment of the present disclosure, the method further includes: when all the original photographed images of the current panoramic task are generated, generating a panoramic image of the next panoramic task by using the method.

[0015] According to an embodiment of the present disclosure, the method further includes: processing the abnormal situation, wherein the processing of the abnormal situation includes any one or more of the following processes:

[0016] When generating the original captured image of the current panoramic task, detecting whether the generation fails through the first thread, and if so, terminating the generation of the panoramic image of the current panoramic task;

[0017] When preprocessing and feature extraction are performed on the original captured image, detecting, by the second thread, whether the original captured image or the preprocessed image corresponding to the original captured image meets the preset quality, and if not, terminating the generation of the panoramic image of the current panoramic task;

[0018] When generating the second temporary data corresponding to the current panoramic task, detecting whether the second temporary data meets the preset requirements through the second thread, and if not, terminating the generation of the panoramic image of the current panoramic task;

[0019] Before generating a panoramic image of the current panoramic task based on the image of the current panoramic task and the second temporary data, the third thread is used to detect whether the image of the current panoramic task and the second temporary data match. If they do not match, the generation of the panoramic image of the current panoramic task is terminated.

[0020] According to an embodiment of the present disclosure, the method further includes:

[0021] Generate a second temporary data file in a specified format using the second temporary data through the second thread, and store the second temporary data file in a specified storage space;

[0022] After the system is restarted or after detecting an abnormal interruption and recovery, the second thread detects whether the designated storage space contains the second temporary data file. If so, the second temporary data is obtained from the second temporary data file and stored in the second memory.

[0023] According to an embodiment of the present disclosure, the preprocessing and feature extraction of the original captured image includes:

[0024] Performing image visual attribute processing on the original captured image to generate a processed captured image;

[0025] Determining whether the size of the original captured image exceeds a preset threshold, and if so, performing image size reduction processing on the original captured image to generate the pre-processed image corresponding to the original captured image;

[0026] Extracting image feature point information from the preprocessed image corresponding to the original captured image or the processed captured image, and using the extracted image feature point information as first temporary data of the original captured image;

[0027] The image visual attribute processing includes any combination of the following processing: adjusting image contrast, adjusting image brightness, and histogram equalization.

[0028] According to an embodiment of the present disclosure, the generating second temporary data corresponding to the current panoramic task according to the first temporary data of all original photographed images of the current panoramic task includes:

[0029] Based on the first temporary data of all the original photographed images of the current panoramic task, image feature point matching is performed by using a partial matching method to generate feature point matching data of the current panoramic task;

[0030] Based on the feature point matching data, globally optimize the image poses of all the original captured images of the current panoramic task, and generate a globally optimized rotation matrix of all the original captured images;

[0031] The feature point matching data of the current panoramic task and the rotation matrix are used as the second temporary data.

[0032] According to an embodiment of the present disclosure, the first temporary data based on all the original captured images of the current panoramic task uses a partial matching method to perform image feature point matching, including:

[0033] Acquire the image shooting parameters of the current panoramic task, wherein the image shooting parameters include: a shooting sequence number and a shooting position; each original shot image corresponds to a specified shooting position and shooting sequence number;

[0034] Generate a position matching relationship table of all the original captured images according to the image capturing parameters, wherein the position matching relationship table is used to describe the position matching relationship between every two original captured images, and the position matching relationship includes: position matching and position mismatching;

[0035] Based on the first temporary data of the position-matched original photographed image indicated by the position matching relationship table, image feature point matching is performed on the position-matched original photographed image.

[0036] According to an embodiment of the present disclosure, the method of globally optimizing the image poses of all original captured images of the current panoramic task based on the feature point matching data and generating a globally optimized rotation matrix of all original captured images includes:

[0037] According to the initial rotation angle controlled by the shooting positioning device, an initial rotation matrix of the image to be optimized relative to the optimized anchor image is obtained; wherein the optimized anchor image is one of the original captured images selected from all the original captured images of the current panoramic task according to a preset rule, and the image to be optimized is the other original captured image except the optimized anchor image;

[0038] Based on the feature point matching data, globally optimize the initial rotation matrix to obtain a globally optimized rotation matrix;

[0039] Determine the difference between the globally optimized rotation matrix and the initial rotation matrix, and obtain an optimization result based on the difference; if the optimization result is successful, use the globally optimized rotation matrix and the unit matrix corresponding to the optimized anchor point image as the globally optimized rotation matrix of all the original captured images.

[0040] According to an embodiment of the present disclosure, generating a panoramic image of the current panoramic task according to the image of the current panoramic task and the second temporary data includes:

[0041] Acquire a camera intrinsic parameter matrix corresponding to the image of the current panoramic task; the camera intrinsic parameter matrix includes: a calibrated camera intrinsic parameter matrix or a globally optimized camera intrinsic parameter matrix;

[0042] Obtaining remapped image region quadrilaterals corresponding to the images of the current panoramic task according to the camera intrinsic parameter matrix, the globally optimized rotation matrix of all the original captured images, and the vertex coordinates of the images of the current panoramic task;

[0043] Determining, based on the image area quadrilaterals, a remapped image area corresponding to each image of the current panoramic task;

[0044] Constructing a mapping table between each pixel in the remapped image area and each image of the current panoramic task according to the camera intrinsic parameter matrix, the globally optimized rotation matrix of all the original captured images, and the coordinates of each pixel in the remapped image area;

[0045] A panoramic image of the current panoramic task is generated based on the mapping table.

[0046] According to an embodiment of the present disclosure, when the camera intrinsic parameter matrix is ​​a globally optimized camera intrinsic parameter matrix, the method further includes:

[0047] Obtaining a camera intrinsic parameter matrix to be optimized corresponding to the image of the current panoramic task;

[0048] Based on the feature point matching data, globally optimize the camera intrinsic parameter matrix to be optimized to generate the globally optimized camera intrinsic parameter matrix;

[0049] The step of using the feature point matching data of the current panoramic task and the rotation matrix as the second temporary data includes:

[0050] Using the feature point matching data of the current panoramic task, the rotation matrix and the globally optimized camera intrinsic parameter matrix as the second temporary data;

[0051] The acquiring of the camera intrinsic parameter matrix corresponding to the image of the current panoramic task includes:

[0052] A camera intrinsic parameter matrix corresponding to the image of the current panoramic task is acquired based on the second temporary data.

[0053] In a second aspect, an embodiment of the present disclosure provides a panoramic image generation device based on multi-threading, wherein the multi-threading includes: a first thread and a second thread, wherein the first thread and the second thread share a first memory; the device includes:

[0054] a first thread running module, configured to generate an original captured image of the current panoramic task through the first thread when generating a panoramic image of the current panoramic task, and store description information of the original captured image into the first memory;

[0055] The second thread running module is configured to detect whether the first memory contains description information of the original captured image based on a first preset detection period through the second thread, and if so, then: the original captured image is obtained according to the description information of the original captured image, and the original captured image is preprocessed and feature extracted to generate first temporary data, or a preprocessed image and first temporary data corresponding to the original captured image are generated; and the second thread is configured to detect whether all the original captured images of the current panoramic task have been processed, and if so, then: second temporary data corresponding to the current panoramic task is generated based on the first temporary data of all the original captured images of the current panoramic task, so as to generate a panoramic image of the current panoramic task based on the image of the current panoramic task and the second temporary data, wherein the image of the current panoramic task includes: all the original captured images of the current panoramic task or preprocessed images corresponding to all the original captured images of the current panoramic task.

[0056] In a third aspect, a chip is provided in an embodiment of the present disclosure, comprising the device described in the second aspect; or, comprising a memory and a processor; wherein the memory is used to store one or more computer instructions, wherein the one or more computer instructions are executed by the processor to implement any method described in the first aspect.

[0057] According to the technical solution provided by the embodiment of the present disclosure, efficient panoramic image synthesis is achieved by using multiple threads (first thread, second thread) and shared memory (first memory). First, the first thread is responsible for generating and storing the original captured image and its description information in the first memory and the designated storage space. Then, the second thread periodically detects the description information in the first memory, preprocesses the existing original captured image, generates the first temporary data, the first temporary data and the preprocessed image, and after all the original captured images are processed, integrates these data into the second temporary data, so as to finally combine the image data of the current panoramic task with the second temporary data to generate the final panoramic image. This method reduces the waiting time by processing the shooting part and the front-end processing part in parallel through multi-threading, effectively improves the efficiency and response speed of panoramic image generation, shortens the panoramic image synthesis time, and thus improves the user experience.

[0058] In addition, in the technical solution provided by the embodiment of the present disclosure, the multithreading also includes a third thread, and the second thread and the third thread share a second memory. The second thread stores the generated second temporary data in the second memory, and the third thread detects the second temporary data in the second memory at another preset period. Once detected, a panoramic image of the current panoramic task is generated according to the image of the current panoramic task and the second temporary data, that is, the three links of the shooting part, the front-end processing part and the back-end synthesis part are respectively put into independent threads for processing, so that the back-end synthesis part can be carried out in parallel with the shooting part and the front-end processing part, thereby further shortening the total time of panoramic image generation, further improving the efficiency and response speed of panoramic image generation, and enhancing user experience.

[0059] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0060] Other features, objectives and advantages of the present disclosure will become more apparent through the following detailed description of non-limiting embodiments in conjunction with the accompanying drawings. In the accompanying drawings:

[0061] Figure 1 A flowchart showing a method for generating a panoramic image based on multithreading according to an embodiment of the present disclosure is shown;

[0062] Figure 2 A flow chart showing an operation method of the first thread in the shooting part of the panoramic image generation task according to an embodiment of the present disclosure is shown;

[0063] Figure 3 A flow chart showing a method for generating second temporary data according to an embodiment of the present disclosure is shown;

[0064] Figure 4A schematic diagram showing shooting positions and arrangement of all original shot images of a current panoramic task according to an embodiment of the present disclosure;

[0065] Figure 5 The embodiment according to the present disclosure is shown based on Figure 4 A schematic diagram of a position matching relationship table generated by the position relationship of each picture shown;

[0066] Figure 6 A flow chart showing a method for operating a second thread in a front-end processing portion of a panoramic image generation task according to an embodiment of the present disclosure is shown;

[0067] Figure 7 A flowchart showing a method for generating a panoramic image of a current panoramic task according to an embodiment of the present disclosure is shown;

[0068] Figure 8 The embodiment according to the present disclosure is shown Figure 4 Schematic diagram of remapping area of ​​each original captured image shown;

[0069] Fig. 9 A schematic diagram showing the final effect obtained after remapping the original image in its entirety;

[0070] Fig.10 A schematic diagram showing a method for determining a boundary area when synthesizing a panoramic image according to an embodiment of the present disclosure is shown;

[0071] Fig.11 A schematic diagram showing a final effect obtained after partially remapping an original image according to an embodiment of the present disclosure;

[0072] Fig.12 A flow chart showing the operation method of the third thread in the back-end synthesis part of the panoramic image generation task according to an embodiment of the present disclosure;

[0073] Fig.13 A schematic structural diagram of a panoramic image generating device based on multi-threading according to an embodiment of the present disclosure is shown. DETAILED DESCRIPTION

[0074] Hereinafter, exemplary embodiments of the present disclosure will be described in detail with reference to the accompanying drawings so that those skilled in the art can easily implement them. In addition, for the sake of clarity, parts not related to the description of the exemplary embodiments are omitted in the accompanying drawings.

[0075] In the present disclosure, it should be understood that terms such as "include" or "have" are intended to indicate the presence of features, numbers, steps, behaviors, components, parts, or a combination thereof disclosed in the present specification, and are not intended to exclude the possibility that one or more other features, numbers, steps, behaviors, components, parts, or a combination thereof exist or are added.

[0076] It should also be noted that, in the absence of conflict, the embodiments and features in the embodiments of the present disclosure may be combined with each other. The present disclosure will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0077] As mentioned earlier, when performing panoramic image synthesis, existing panoramic image synthesis solutions have the problem of being time-consuming. When the number of pictures is large or the resolution is high, the amount of calculation increases dramatically. Especially on platforms with limited computing resources such as drones, the synthesis time may be as long as tens of seconds, which not only affects the user experience, but may also miss important shooting opportunities.

[0078] In order to shorten the panoramic image synthesis time, improve synthesis efficiency, and thus enhance user experience, the inventors of the present invention have carefully studied and considered, optimized and improved the existing panoramic image synthesis scheme from multiple dimensions, and provided a concise and efficient panoramic image generation method.

[0079] Based on the in-depth understanding of the panoramic image generation technology and the accurate grasp of the characteristics of each link, the inventor of the present invention decomposed the panoramic image generation task that was originally executed sequentially in a single thread into three independent parts: shooting, front-end processing and back-end synthesis, and introduced a multi-threaded parallel collaborative processing architecture, putting each part into a separate thread for execution, and sharing the same memory for data exchange between every two adjacent parts to achieve parallel processing. Among them, the shooting part is mainly responsible for collecting the original photos required for the panoramic image. By separating the shooting part, it can be ensured that the focus is on obtaining high-quality original photos without being disturbed by other processing links; the front-end processing part is mainly responsible for preprocessing and feature extraction of the original photos taken. Since the front-end processing part requires strong computing power and algorithm support, separating the front-end processing part can make full use of computing resources, accelerate the image preprocessing and feature extraction process, and the front-end processing part can be executed in parallel with other parts, thereby improving the overall processing efficiency; the back-end synthesis part is mainly responsible for stitching the preprocessed image or the original shot image according to the feature point matching results obtained by the front-end processing to generate the final panoramic image. Since the back-end synthesis part needs to process a large amount of image data and complex stitching algorithms, separating the back-end synthesis part can ensure that with sufficient resources, it can focus on completing the stitching and optimization tasks of the panoramic image, and the back-end synthesis part can be executed in parallel with other parts, thereby further improving the overall processing efficiency.

[0080] In summary, decomposing the panoramic image generation process into three parts: shooting, front-end processing, and back-end synthesis, and executing each part in a separate thread is a reasonable and efficient solution based on the characteristics of each part. Among them, each thread can independently allocate and use computing resources, avoiding resource competition and waiting time, thereby optimizing resource usage; and independent threads processing each part of the task can reduce system coupling and enhance system stability and scalability. If a problem occurs in a part, it will not affect the normal operation of other parts, which is convenient for troubleshooting and repair. Therefore, this decomposition method can make full use of computing resources, improve processing efficiency, and enhance system stability and scalability.

[0081] Figure 1 A flowchart of a method for generating a panoramic image based on multithreading according to an embodiment of the present disclosure is shown. The multithreading includes: a first thread and a second thread, and the first thread and the second thread share a first memory. Figure 1 As shown, the method includes the following steps S110~S130:

[0082] In step S110 , when generating a panoramic image of the current panoramic task, an original captured image of the current panoramic task is generated by the first thread, and description information of the original captured image is stored in the first memory.

[0083] According to an embodiment of the present disclosure, when generating the original captured image of the current panoramic task, the first thread detects whether the generation fails, and if so, terminates the generation of the panoramic image of the current panoramic task. In this way, if a problem is found when generating the original captured image and the generation of the panoramic image is terminated, subsequent unnecessary calculations and resource consumption can be avoided. This helps save system resources and improve overall processing efficiency.

[0084] For example, during the gimbal movement or camera shooting process, the first thread continuously monitors the operation status. If a gimbal movement abnormality (such as mechanical stall) or a camera shooting error (such as insufficient storage space) is detected, the shooting process is interrupted and the error information is recorded for subsequent processing.

[0085] The original photographed images in the present disclosure refer to the original photos taken by using a panoramic camera equipped with a wide-angle lens or a fisheye lens. The original photos at different angles can be taken by using a gimbal with a motor to control the camera, or by using a mechanical arm to control the camera movement, or even manually taking photos. As long as the photos at different angles are taken according to certain steps, it will be fine.

[0086] In the present disclosure, the first thread is a separate thread responsible for the process of the shooting part. Figure 2The flowchart of the operation method of the first thread in the shooting part of the panoramic image generation task according to the embodiment of the present disclosure is shown. Take the shooting positioning device (such as: gimbal) controlling the camera to obtain the original shooting image of the current panoramic task as an example. Figure 2 As shown, the following steps S111 to S116 are included:

[0087] In step S111, user instructions are received and processed, including: continuously monitoring the instruction input from the user or the system through the first thread; when an instruction to shoot a panoramic image is received, parsing the instruction to obtain image shooting information of the current panoramic task; and initializing a panoramic shooting task object according to the parsed image shooting information.

[0088] The image shooting information includes the shooting angle range (such as 180° horizontally), shooting interval, image resolution, and other possible shooting parameters; the panoramic shooting task object contains all necessary shooting information and status tracking.

[0089] In step S112, the shooting and positioning device is controlled to move the panoramic camera and take photos according to the image shooting information to generate an original shot image of the current panoramic task.

[0090] In addition, after the original captured image of the current panoramic task is generated, it can be stored in a designated storage space so that the second thread can obtain it therefrom.

[0091] Specifically, the number of shots corresponding to all the original shots of the current panoramic task and all the specific angles of the shots are calculated according to the image shooting information; for example, for a horizontal 180° panorama, if one shot is taken every 10°, 19 shooting points (including the start and end angles) need to be calculated, so that 19 original shots are finally generated. For each calculated angle, a control instruction is sent to the shooting positioning device to move it to the specified angle. After it moves into position, the panoramic camera is triggered to take a photo. After the shooting is completed, the original shot image is obtained from the camera and stored in the specified storage space so that the second thread can correctly obtain the corresponding original shot image. The specified storage space is usually a temporary folder or a memory buffer.

[0092] In step S113, it is detected whether the generation of the original photographed image fails, if so, the generation of the panoramic image of the current panoramic task is terminated. If not, the following step S114 is executed.

[0093] In step S114, description information of the original captured image is generated and stored in the first memory.

[0094] In the present disclosure, each time an original captured image of the current panoramic task is generated, its description information can be stored in the first memory for acquisition and processing by the second thread, so that the first thread and the second thread can be processed in parallel, thereby improving synthesis efficiency.

[0095] The description information of the original captured image includes, but is not limited to, the photo file name, the shooting angle, the photo file path, etc. Such information will be used for subsequent image processing and panorama generation.

[0096] The memory (including the first memory and the second memory) disclosed in the present invention refers to a storage area in a system memory (such as a random access memory RAM) where data can be read and written. It can be non-volatile or volatile.

[0097] When storing the description information, it can be stored in the form of message queue data, which usually provides a synchronization mechanism to ensure the correct transmission of data. It can also be stored in other data forms, such as linked lists.

[0098] In step S115, it is detected whether all the original captured images of the current panoramic task have been captured. If not, the above step S112 is executed. If so, the following step S116 is executed.

[0099] In step S116, the task is completed and the status is updated.

[0100] In a specific implementation, when all original images of the current panoramic task are captured and all images and description information are correctly stored, the first thread updates the state of the panoramic shooting task object to "shooting completed". In addition, the first thread may also notify the system or user that the shooting task has been completed, and provide any necessary feedback or log information.

[0101] In addition, in a specific embodiment of the present disclosure, each part of shooting, front-end processing, and back-end synthesis can each run in a separate thread. In this way, the processing of each part is relatively independent. This multi-threaded processing method allows that when all the original shot images of the current panoramic task are generated, that is, while processing the subsequent part of the current panoramic task (front-end processing or back-end synthesis), the system does not need to destroy or re-create the first thread, and can process the next panoramic task in parallel through steps S110~130 to generate a panoramic image of the next panoramic task, thereby realizing seamless switching and resource reuse between different panoramic tasks.

[0102] In step S120, the second thread detects whether the first memory contains description information of the original captured image based on a first preset detection period. If so, the original captured image is obtained according to the description information of the original captured image, and the original captured image is preprocessed and feature extracted to generate first temporary data, or a preprocessed image and first temporary data corresponding to the original captured image are generated.

[0103] The implementation details of step S120 are described below:

[0104] In this step S120, feature points are extracted mainly based on the original captured image generated in step S110 for feature point matching in the subsequent step S130. The first temporary data refers to feature point information corresponding to the specified original captured image.

[0105] According to an embodiment of the present disclosure, the preprocessing and feature extraction of the original captured image includes:

[0106] Performing image visual attribute processing on the original captured image to generate a processed captured image; determining whether the size of the original captured image exceeds a preset threshold, and if so, reducing the image size of the original captured image to generate a preprocessed image corresponding to the original captured image; extracting image feature point information from the preprocessed image corresponding to the original captured image or the processed captured image, and using the extracted image feature point information as first temporary data of the original captured image.

[0107] The image visual attribute processing includes but is not limited to any combination of the following processing: adjusting image contrast, adjusting image brightness, and histogram equalization.

[0108] In the present disclosure, before extracting features from the original captured image, the original captured image is processed in advance, such as adjusting image contrast, brightness, and histogram equalization, so as to enhance the stability of feature point extraction. For images with larger sizes, image size reduction is performed, and feature extraction is performed based on the reduced image, and the reduced image is used in the back-end synthesis, which can reduce the time consumption of feature point extraction on the one hand, and accelerate the algorithms such as back-end seam generation on the other hand, thereby shortening the synthesis time of the panoramic image.

[0109] When extracting feature points, some existing algorithms can be used, such as SIFT (Scale-Invariant Feature Transform) algorithm, SURF (Speeded-Up Robust Features) algorithm, and ORB (Oriented FAST and Rotated BRIEF) algorithm.

[0110] According to an embodiment of the present disclosure, when the original captured image is preprocessed and feature extracted, the second thread detects whether the original captured image or the preprocessed image corresponding to the original captured image meets the preset quality. If not, such as too few image feature points or unable to be extracted, the generation of the panoramic image of the current panoramic task is terminated. In this way, through quality detection, it can be ensured that only images that meet the preset quality standards are used for the generation of panoramic images. This helps to avoid the use of blurred, distorted or low-quality images, thereby improving the overall quality of the final panoramic image and enhancing the user experience, while avoiding unnecessary waste of computing resources and saving CPU and memory resources.

[0111] In step S130, the second thread is used to detect whether all the original captured images of the current panoramic task have been processed. If they have been processed, second temporary data corresponding to the current panoramic task is generated based on the first temporary data of all the original captured images of the current panoramic task, so as to generate a panoramic image of the current panoramic task based on the image of the current panoramic task and the second temporary data, wherein the image of the current panoramic task includes: all the original captured images of the current panoramic task or preprocessed images corresponding to all the original captured images of the current panoramic task.

[0112] The implementation details of step S130 are described below:

[0113] In this step S130, feature point matching is mainly performed based on the feature point information of all original captured images in step S120 to form feature point pairs, and the image pose is globally optimized to generate a globally optimized rotation matrix. The second temporary data mainly includes: feature point matching data and a globally optimized rotation matrix.

[0114] Figure 3 FIG. 2 shows a flow chart of a method for generating second temporary data according to an embodiment of the present disclosure. Figure 3 As shown, when generating the second temporary data corresponding to the current panoramic task according to the first temporary data of all the original photographed images of the current panoramic task, the following steps S131-133 are included:

[0115] In step S131, based on the first temporary data of all the original photographed images of the current panoramic task, image feature point matching is performed by adopting a partial matching method to generate feature point matching data of the current panoramic task.

[0116] According to an embodiment of the present disclosure, the first temporary data based on all the original captured images of the current panoramic task uses a partial matching method to perform image feature point matching, including:

[0117] The image shooting parameters of the current panoramic task are obtained, wherein the image shooting parameters include: a shooting sequence number and a shooting position; each original shot image corresponds to a specified shooting position and shooting sequence number.

[0118] A position matching relationship table of all the original captured images is generated according to the image capture parameters, wherein the position matching relationship table is used to describe the position matching relationship between every two original captured images, wherein the position matching relationship includes: position matching and position mismatching. In a specific example, "1" indicates position matching and "0" indicates position mismatching.

[0119] Based on the first temporary data of the position-matched original photographed image indicated by the position matching relationship table, image feature point matching is performed on the position-matched original photographed image.

[0120] Take the panoramic synthesis of 9 original images arranged in a nine-square grid as an example:

[0121] In this example, the gimbal is controlled to rotate in 9 different directions to take 9 photos, with the panning direction at an interval of 25° and the pitch direction at an interval of 20°. Figure 4 FIG. 1 shows a schematic diagram of the shooting positions and arrangement of all original shooting images of the current panoramic task according to an embodiment of the present disclosure. Figure 4 As shown, the current panoramic task includes 9 original captured images, namely: the 1st to the 9th images, Figure 5 The embodiment according to the present disclosure is shown based on Figure 4 The schematic diagram of the position matching relationship table generated by the position relationship of each picture is shown in FIG. Figure 5 As shown, the 7th image has no overlapping area with the 3rd image, so there is no need to match, and the values ​​of <7th row, 3rd column> and <3rd row, 7th column> in the table are 0. The 1st image has overlapping area with the 4th image, so they need to be matched, and the values ​​of <4th row, 1st column> and <1st row, 4th column> in the table are 1.

[0122] Among them, the matching relationship of some pictures in the position matching relationship table can be set according to specific needs. For example, if the first picture and the third picture have overlapping areas only in the diagonal direction, they can be matched or not. In addition, the matching relationship table can be manually generated in advance and saved in the system. In the case of a large number of pictures, it can also be automatically generated by the program. For example, if each picture only matches the pictures in the four directions of up, down, left, and right, the program automatically fills 1 in the table of the corresponding direction.

[0123] When the arrangement relationship of each picture is unknown, the traditional panoramic synthesis algorithm needs to fully match each picture during the feature point matching step, that is, the matching relationship between every two pictures needs to be checked.

[0124] In the present disclosure, a partial matching method is innovatively introduced. A position matching relationship table describing the matching relationship of all pictures is determined in advance according to the image shooting parameters (shooting sequence number and shooting position), and then feature point matching is performed based on the determined position matching relationship table. In this example, the number of picture matching times can be reduced from 72 to 24. Therefore, this method can significantly narrow the matching range of pictures, significantly reduce the number of matching times, and reduce the probability of mismatching between pictures with no overlapping areas.

[0125] In step S132, based on the feature point matching data, the image poses of all the original captured images of the current panoramic task are globally optimized to generate a globally optimized rotation matrix of all the original captured images.

[0126] According to an embodiment of the present disclosure, the method of globally optimizing the image poses of all original captured images of the current panoramic task based on the feature point matching data and generating a globally optimized rotation matrix of all original captured images includes:

[0127] According to the initial rotation angle controlled by the shooting positioning device (such as a gimbal), the initial rotation matrix of the image to be optimized relative to the optimized anchor image is obtained; wherein the optimized anchor image is one of the original images selected from all the original images of the current panoramic task according to a preset rule, and the image to be optimized is the other original images except the optimized anchor image.

[0128] Based on the feature point matching data, the initial rotation matrix is ​​globally optimized to obtain a globally optimized rotation matrix.

[0129] Determine the difference between the globally optimized rotation matrix and the initial rotation matrix, and obtain an optimization result based on the difference; if the optimization result is successful, use the globally optimized rotation matrix and the unit matrix corresponding to the optimized anchor point image as the globally optimized rotation matrix of all the original captured images.

[0130] Let's take the panoramic synthesis of 9 original images arranged in a nine-square grid as an example:

[0131] like Figure 4 As shown, for the initial rotation angle of each image, for example, the heading angle and pitch angle of the first image are both 0, the heading angle of the fifth image is 25 degrees, and the pitch angle is 20 degrees. The first image can be used as the optimized anchor image, that is, its rotation is not optimized, then: the second to ninth images are the images to be optimized, that is, their rotation needs to be optimized. In general, the optimized anchor image is generally the image arranged in the middle in the spatial relationship (in this example, the first image), or it can be any selected image.

[0132] When obtaining the initial rotation matrix of the image to be optimized relative to the optimized anchor point image, the following method can be used:

[0133] First, determine the heading angle (yaw), roll angle (roll), and pitch angle (pitch) of each image to be optimized relative to the optimized anchor image. Then: for the i-th image, its Euler angle can be expressed as (yaw_i, roll_i, pitch_i).

[0134] Then, use the conversion formula from Euler angle to rotation matrix to convert each Euler angle into the corresponding rotation matrix.

[0135] The specific conversion process can be divided into three steps:

[0136] Step 1: Rotate yaw_i angle around the z axis to get the rotation matrix .

[0137] Step 2: Rotate pitch_i angle around the y-axis to get the rotation matrix .

[0138] Step 3: Rotate roll_i angle around the x-axis to get the rotation matrix .

[0139] The final rotation matrix This can be obtained by matrix multiplication: * * .

[0140] Finally, repeat the above steps for each image (except the optimized anchor image) to obtain 8 initial rotation matrices ~ .

[0141] After obtaining the initial rotation matrix, a global optimization objective function can be defined, which aims to minimize the reprojection error of feature points between all adjacent images. For each image, the rotation matrix corresponding to the image is introduced as the optimization variable (for the optimized anchor image, its rotation matrix is ​​fixed to the unit matrix), and a nonlinear optimization algorithm (such as the Levenberg-Marquardt algorithm or the Gauss-Newton algorithm, etc.) is used to minimize the global reprojection error. During the optimization process, the rotation matrix of the optimized anchor image is kept unchanged, and only the remaining 8 images are optimized to generate the globally optimized rotation matrix. ~ .

[0142] Due to the internal parameter errors of the controller and the camera, after global optimization ~ Will be better than the initial ~ There is a slight difference. If the difference is too large, it proves that the optimization is wrong. Passed the verification and The difference in the optimization results can be used to determine whether the optimization is successful. and The Difference It can be expressed using the following formula (1):

[0143] (1)

[0144] like ,but: Should be unit matrix , by comparing The difference between the matrix and the unit matrix determines the optimization result. Converted to shaft angle, the angle modulus of the shaft angle does not exceed θ1. Converted into Euler angles, the angle modulus of the three axes of the Euler angles does not exceed θ2.

[0145] The inventors of the present disclosure have noticed in specific projects that, generally, when the error of the gimbal controller does not exceed 1°, θ1 can take a value of 10° and θ2 can take a value of 3°. If the angle exceeds this value, a stitching error visible to the naked eye may occur, which can be used to determine in advance that the optimization has failed and terminate the stitching task.

[0146] The present disclosure introduces an optimized anchor image during global optimization, which means that there is a fixed reference point during the global optimization process. This reference point remains unchanged throughout the optimization process, providing a stable reference for other images and helping to maintain the stability and consistency of the entire image stitching. At the same time, by globally optimizing the rotation matrix of the image to be optimized relative to the optimized anchor image, it can be ensured that all images are adjusted relative to a common starting point during the optimization process. This relative optimization method helps to reduce the cumulative error caused by independent optimization, thereby avoiding overall drift. In addition, a mechanism for obtaining the optimization result based on the difference between the globally optimized rotation matrix and the initial rotation matrix is ​​introduced. This difference verification mechanism provides additional quality control for the optimization process, which helps to detect and correct errors in a timely manner. As a result, the problems of overall drift and abnormal errors in global optimization are effectively avoided.

[0147] In step S133, the feature point matching data of the current panoramic task and the rotation matrix are used as the second temporary data.

[0148] According to an embodiment of the present disclosure, when generating the second temporary data corresponding to the current panoramic task, the second thread detects whether the second temporary data meets the preset requirements. If not, the generation of the panoramic image of the current panoramic task is terminated. In this way, by detecting whether the second temporary data meets the preset requirements, it can be ensured that the generation of the panoramic image is based on accurate and reliable data. If the data does not meet the requirements, it may mean that there are problems in the processing or feature extraction of the original captured image, or that the stitching and fusion of the panoramic image will not produce satisfactory results. In this case, terminating the generation of the panoramic image can avoid the generation of low-quality images, thereby improving the accuracy of the panoramic image generation.

[0149] The operation of generating the panoramic image of the current panoramic task according to the image of the current panoramic task and the second temporary data in step S130 can be directly executed in the second thread or separately executed in another thread. The present disclosure provides a third thread, that is, this operation is processed separately in the third thread, so that the three parts of the shooting part, the front-end processing part and the back-end synthesis part involved in the panoramic image generation process are all processed by a separate thread, which further shortens the total time of the panoramic image generation.

[0150] According to an embodiment of the present disclosure, the multithreading further includes: a third thread, the second thread and the third thread share a second memory, and the generating of the panoramic image of the current panoramic task according to the image of the current panoramic task and the second temporary data includes:

[0151] storing the second temporary data to the second memory through the second thread;

[0152] The third thread detects whether the second temporary data exists in the second memory based on a second preset detection period. If so, a panoramic image of the current panoramic task is generated based on the image of the current panoramic task and the second temporary data.

[0153] The following describes the running details of the second thread:

[0154] In the present disclosure, the second thread is a separate thread responsible for the process of the front-end processing part. Figure 6 FIG. 1 is a flow chart showing the operation method of the second thread in the front-end processing part of the panoramic image generation task according to an embodiment of the present disclosure. Figure 6 As shown, the following steps S121~S12b are included:

[0155] In step S121, it is detected whether the first memory contains description information of the original captured image. If so, the following step S122 is executed.

[0156] In step S122, the original captured image is acquired according to the description information of the original captured image.

[0157] In step S123, it is determined whether the size of the original captured image exceeds a preset threshold, if yes, the following step S124 is executed, if no, the following step S125 is executed.

[0158] In step S124, the original captured image is reduced in size to generate the pre-processed image corresponding to the original captured image.

[0159] In step S125, image visual attribute processing is performed on the original captured image to generate a processed captured image.

[0160] In step S126, it is detected whether the processed captured image or the pre-processed image meets the preset quality. If not, the generation of the panoramic image of the current panoramic task is terminated. If yes, the following step S127 is executed.

[0161] In step S127, image feature point information is extracted from the preprocessed image corresponding to the original captured image or the processed captured image, and the extracted image feature point information is used as first temporary data of the original captured image.

[0162] In step S128, it is detected whether all the original photographed images of the current panoramic task have been processed. If the processing is completed, the following step S129 is executed; if the processing is not completed, the execution returns to step S121.

[0163] In step S129 , second temporary data corresponding to the current panoramic task is generated according to the first temporary data of all original photographed images of the current panoramic task.

[0164] In step S12a, it is detected whether the second temporary data meets the preset requirements. If not, the generation of the panoramic image of the current panoramic task is terminated. If yes, step S12b is executed.

[0165] In step S12b, the second temporary data is stored in the second memory.

[0166] According to an embodiment of the present disclosure, after generating second temporary data corresponding to the current panoramic task based on the first temporary data of all original captured images of the current panoramic task, the second temporary data can also be used by the second thread to generate a second temporary data file in a specified format, and the second temporary data file can be stored in a designated storage space; wherein the designated storage space has a non-volatile storage characteristic, for example: it can be a designated storage area in a hard disk (HDD) or some type of persistent memory (such as NVRAM), etc.

[0167] Thus, after the system is restarted or after abnormal interruption recovery is detected, the following steps S12c and S12d are also included:

[0168] In step S12c, it is detected whether the designated storage space contains the second temporary data file. If so, step S12d is executed, i.e., obtaining the second temporary data from the second temporary data file, and then step S12b is executed, i.e., storing the second temporary data in the second memory.

[0169] The inventor of the present disclosure noticed during the project development that there are still certain risks in the panoramic synthesis process, such as abnormal situations such as power outages. If the second temporary data is only stored in the memory after the system is interrupted for various reasons, these data are likely to be lost. After the system is restarted, the synthesis process may be forced to be interrupted for the current panoramic task, resulting in the failure of the synthesis work that has been carried out. The original photos need to be re-shot and the subsequent processing needs to be carried out, which invisibly increases the synthesis time of the panoramic image and also brings unnecessary losses and troubles to users.

[0170] The present disclosure generates a second temporary data file in a specified format from the second temporary data and stores it in a specified storage space, thereby ensuring that the data used for back-end synthesis will not be lost after an abnormal system interruption (such as power outage, system crash, etc.). In this way, even if the system is restarted or a failure occurs, for the current panoramic image generation, it is only necessary to restore the data by reading these files from the specified storage space, and then restore the interrupted task, thereby shortening the panoramic image synthesis time. At the same time, storing the second temporary data as a file can make data management and debugging more convenient. For example, developers can check these files to verify the correctness of the data, or locate the problem by tracing back these files when a problem occurs.

[0171] It should be noted that: if a preprocessed image corresponding to the original captured image is generated in step S120, that is, the image size reduction processing operation is performed, then when generating the panoramic image of the current panoramic task, the preprocessed image corresponding to all the original captured images is used as the image of the current panoramic task to perform the back-end synthesis part operation, otherwise all the original captured images of the current panoramic task are used as the image of the current panoramic task to perform the back-end synthesis part operation.

[0172] The following describes the implementation details of the back-end synthesis part:

[0173] Figure 7 A flow chart of a method for generating a panoramic image of a current panoramic task according to an embodiment of the present disclosure is shown. Figure 7 As shown, when the panoramic image of the current panoramic task is generated according to the image of the current panoramic task and the second temporary data, the following steps S141-S145 are included:

[0174] In step S141, a camera intrinsic parameter matrix corresponding to the image of the current panoramic task is obtained; the camera intrinsic parameter matrix includes: a calibrated camera intrinsic parameter matrix or a globally optimized camera intrinsic parameter matrix.

[0175] When generating a panoramic image for the current panoramic task, if a preprocessed image corresponding to the original captured image is used, the camera intrinsic parameter matrix is ​​the camera intrinsic parameter matrix of the preprocessed image; if all original captured images are used, the camera intrinsic parameter matrix is ​​the camera intrinsic parameter matrix of the original captured images.

[0176] The Camera Intrinsic Matrix is ​​a matrix that describes the internal parameters of the camera, which determine how the camera maps points in the three-dimensional world to the two-dimensional image plane. These parameters usually include focal length (fx, fy), optical center (cx, cy), and possible lens distortion parameters.

[0177] The calibrated camera intrinsic parameter matrix in this disclosure refers to the camera intrinsic parameter matrix obtained through the camera calibration process. Camera calibration is a process of estimating camera intrinsic parameters using known points in the physical world (usually a chessboard or dot array, etc.). By taking images of these known points and using a specific algorithm (such as the Zhang Zhengyou calibration method), the camera intrinsic parameter matrix can be calculated.

[0178] Among them, when obtaining the camera intrinsic parameter matrix after global optimization, it can be done in the following way:

[0179] First, the camera intrinsic parameter matrix to be optimized corresponding to the image of the current panoramic task is obtained. The camera intrinsic parameter matrix to be optimized refers to the camera intrinsic parameter matrix that needs to be further adjusted or optimized. By globally optimizing these intrinsic parameters, the accuracy and quality of the processing results can be significantly improved.

[0180] Then, based on the feature point matching data, the camera intrinsic parameter matrix to be optimized is globally optimized to generate the globally optimized camera intrinsic parameter matrix. Specifically, the reprojection error function is constructed using the feature point matching data. The reprojection error is a measure of the difference between the feature point projected onto the image plane in three-dimensional space through the camera intrinsic parameters and extrinsic parameters (rotation matrix and translation vector) and the actual detected feature point position. The camera intrinsic parameter matrix to be optimized is used as the optimization variable, and the goal is to minimize the sum of the reprojection errors of all feature points. Then, an appropriate optimization algorithm is selected, such as the Levenberg-Marquardt algorithm, the gradient descent method or its variants, to solve this nonlinear least squares problem, and to set the initial value, number of iterations, convergence conditions and other parameters of the optimization algorithm. Finally, the camera intrinsic parameter matrix to be optimized and the optimization problem are input into the selected optimization algorithm. The optimization process is iterated until the convergence condition is met or the maximum number of iterations is reached, until the globally optimized camera intrinsic parameter matrix is ​​output.

[0181] After obtaining the globally optimized camera intrinsic parameter matrix, the feature point matching data of the current panoramic task, the rotation matrix and the globally optimized camera intrinsic parameter matrix can be put into the second memory as the second temporary data for use by the back-end synthesis part. In this way, when obtaining the camera intrinsic parameter matrix corresponding to the image of the current panoramic task, the camera intrinsic parameter matrix corresponding to the image of the current panoramic task can be obtained based on the second temporary data.

[0182] In step S142, the remapped image area quadrilaterals corresponding to the images of the current panoramic task are obtained according to the camera intrinsic parameter matrix, the globally optimized rotation matrix of all the original captured images and the vertex coordinates of the image of the current panoramic task.

[0183] Specifically, when remapping the pixels in the original image to the output image, it can be calculated by the following formula (2):

[0184] (2)

[0185] Wherein, K is the camera intrinsic parameter matrix, is the rotation matrix of the i-th original captured image after global optimization, the vertex coordinates of the remapped image area quadrilateral corresponding to the i-th image of the current panoramic task, The vertex coordinates of the i-th image of the current panorama task.

[0186] In step S143, the remapped image areas corresponding to the images of the current panoramic task are determined based on the image area quadrilaterals.

[0187] After determining the remapped image area, the discarded area is also indirectly determined. Taking the composite panorama of a plane model as an example, due to the increase in the viewing angle of the composite image, the four corners will produce a large deformation after remapping. When the final composite image is cropped using a rectangle, the redundant parts of the four corners will also be discarded.

[0188] In step S144, a mapping table between each pixel in the remapped image area and each image of the current panoramic task is constructed according to the camera intrinsic parameter matrix, the globally optimized rotation matrix of all the original captured images, and the coordinates of each pixel in the remapped image area.

[0189] When compositing images in the backend, the remapping step takes up a large proportion of the time, especially when using CPU calculations, this step can account for more than 50% of the total backend time.

[0190] In the final picture synthesis process, the present disclosure does not construct a mapping table between each pixel in the output frame and the original image (i.e., the coordinates of the pixel in the output frame corresponding to the pixel in the original image), but only constructs a mapping table between each pixel in the remapped image area except the discarded area and each image of the current panoramic task. The mapping relationship of each pixel is determined by the above formula (2).

[0191] In this way, the present disclosure can reduce the image remapping area by calculating the remapped image boundary in advance, that is, the mapping calculation of the discarded area is omitted, thereby shortening the time consumption of panoramic synthesis.

[0192] In step S145 , a panoramic image of the current panoramic task is generated based on the mapping table.

[0193] Specifically, the process may be performed according to the steps of initializing a panoramic image canvas, traversing a mapping table, assigning and fusing pixels, processing boundaries and gaps, and post-processing and optimizing.

[0194] Let’s take the panoramic synthesis of 9 original images arranged in a nine-square grid as an example:

[0195] Figure 8 The embodiment according to the present disclosure is shown Figure 4 The schematic diagram of the remapping area of ​​each original captured image is shown in FIG. If all the original images are remapped according to the above formula (2), the final image obtained is as follows: Fig. 9 As shown in the red box area, when using rectangular cropping, a larger area will be discarded after cropping the redundant parts on the four sides.

[0196] The present disclosure calculates the discarded area in advance during remapping, wherein the specific method of determining the boundary area is as follows:

[0197] Determine the upper boundary line, lower boundary line, left boundary line and right boundary line of the boundary area respectively. Take the arrangement of m rows and n columns of images as an example: the upper boundary line is the upper edge line of the bounding rectangle of the image in the 1st row and jth column, where j is the column where the optimized anchor image is located; the lower boundary line is the lower edge line of the bounding rectangle of the image in the mth row and jth column; the left boundary line is the left edge line of the bounding rectangle of the image in the kth row and 1st column, where k is the row where the optimized anchor image is located; the right boundary line is the right edge line of the bounding rectangle of the image in the kth row and nth column.

[0198] For example: Fig.10 FIG. 2 is a schematic diagram showing a method for determining a boundary area when synthesizing a panoramic image according to an embodiment of the present disclosure. Fig.10 As shown in the figure, if the optimized anchor image is selected as the first image, the upper, lower, left and right boundaries of the image are the upper, lower, left and right boundaries of images 2, 6, 8 and 4 respectively. The boundary area is determined by the edge of the remapped area of ​​images 2, 6, 8 and 4, that is, the values ​​of d2, d6, d8 and d4. Specifically, the values ​​of d2, d6, d8 and d4 can be determined by the bounding rectangle of each remapped area.

[0199] This optimization reduces the mapping table to Fig.11 The blue frame area of ​​the remapped area is shown, and the mapping calculation of the remaining area is omitted.

[0200] In this example, if the camera has a horizontal viewing angle of 75°, a vertical viewing angle of 60°, and each picture has a heading interval of 25° and a pitch interval of 20°, the pixel mapping calculation can be reduced by about 40%, that is, the total calculation time of the remapping transformation can be reduced by about 40%, which greatly shortens the panoramic image synthesis time, improves the synthesis efficiency, and thus improves the user experience.

[0201] According to an embodiment of the present disclosure, before generating a panoramic image of the current panoramic task based on the image of the current panoramic task and the second temporary data, the third thread is used to detect whether the image of the current panoramic task and the second temporary data match. If they do not match, the generation of the panoramic image of the current panoramic task is terminated.

[0202] When checking whether the image of the current panoramic task matches the second temporary data, the rotation matrix, the number of picture matches and the number of photos can be checked to see if they are consistent. For example, if n photos are taken, the number of rotation matrices required is equal to n, and the number of matched photos is >= n. It is also possible to check whether all data types such as matching data, rotation matrix, camera internal parameters, etc. are complete. If it is text data, it is also possible to check whether the text data contains garbled characters, that is, non-ASCII code value data.

[0203] By introducing a matching detection mechanism, the processing flow of panoramic image generation can be optimized to avoid unnecessary duplication or erroneous operations in subsequent processing, thereby improving processing efficiency and accuracy.

[0204] The following is a detailed description of the running details of the third thread:

[0205] In the present disclosure, the third thread is a separate thread responsible for the process of the backend synthesis part. Fig.12 FIG. 1 is a flowchart showing the operation method of the third thread in the back-end synthesis part of the panoramic image generation task according to an embodiment of the present disclosure. Fig.12 As shown, the following steps S151 to S154 are included:

[0206] In step S151, it is detected whether the second temporary data exists in the second memory. If it exists, the following step S152 is executed.

[0207] In step S152, the second temporary data and the image of the current panoramic task are obtained to detect whether the image of the current panoramic task and the second temporary data match. If they match, the following step S153 is executed; if they do not match, the generation of the panoramic image of the current panoramic task is terminated.

[0208] Here, the second temporary data obtained may be recovered from the second temporary data file through the second thread after the system is restarted or after an abnormal interruption is detected. In some abnormal situations, such as power outages, the data may not be saved in time and may be lost. Therefore, this detection can avoid the occurrence of synthesis failure due to the use of incorrect synthetic data.

[0209] In step S153, a panoramic image of the current panoramic task is generated according to the image of the current panoramic task and the second temporary data.

[0210] In step S154, temporary data related to the current panoramic task is deleted, such as temporary pictures, all text data, etc.

[0211] In the present disclosure, the method of splitting the three major steps of shooting, front-end processing and back-end synthesis in the generation of panoramic images into different thread processing makes full use of the time-consuming characteristics of each step and minimizes user waiting. Since the shooting process requires the pan / tilt to move and stop shooting after reaching a specific angle, the shooting task often takes a long time. The front-end processing and photo shooting can be processed in parallel to reasonably save the waiting time of the front-end processing. In addition, the front-end processing data (first temporary data) is loaded using the first memory, and the front-end processing is no longer associated with the remaining steps. Therefore, the front-end processing thread can be put into the background, which can reduce user waiting, for example, the panoramic photo of the next panoramic task can be continued. In addition, the back-end synthesis data (second temporary data) of each panoramic synthesis task is put into the task queue of the second memory, so that the number of processing tasks is unlimited, and the user can continue to shoot panoramic photos for a long time without waiting.

[0212] After evaluation, when generating panoramic images, applying the technical solution of the present disclosure can achieve a user experience on a CPU device that is similar to that of a customized device (GPU or other parallel processing unit).

[0213] In addition, in the present disclosure, in order to ensure that communication and data processing between threads are timely and do not waste system resources due to frequent detection, the specific values ​​of the first preset detection period and the second preset detection period can be determined according to the following two methods:

[0214] Method 1: First, an initial value is set for the first preset detection cycle and the second preset detection cycle respectively based on experience or preliminary tests. After that, the current system load status is evaluated by obtaining key indicators such as CPU usage, memory occupancy, IO waiting time, etc., and the working status of the first thread, the second thread and the third thread is monitored at the same time, including whether they are busy, whether there are tasks to be processed, etc. Finally, the specific values ​​of the first preset detection cycle and the second preset detection cycle are dynamically adjusted according to the current system load status and the status of each thread, specifically including: if the system load is lower than the preset lower limit threshold, it means that the system resources are sufficient. At this time, the first preset detection cycle and the second preset detection cycle can be appropriately shortened to improve the communication efficiency and data processing speed between threads. If the system load is higher than the preset upper limit threshold, it means that the system resources are tight. At this time, the first preset detection cycle and the second preset detection cycle can be extended to reduce the switching and resource competition between threads and reduce the system overhead. If the first thread generates the original captured image slowly, the first preset detection cycle can be appropriately extended to avoid the second thread from wasting resources due to frequent inspection. If the second thread processes the image slowly, and the first thread has generated a large number of original captured images, the first preset detection cycle can be appropriately shortened so that the second thread can process these images in time. Similarly, the second preset detection period is dynamically adjusted according to the state of the third thread and the data status of the second memory.

[0215] Method 2: Pre-statistic the execution parameters of all executable panoramic tasks in the shooting, front-end processing and back-end synthesis parts. For example, for the first preset detection cycle, the execution parameters of each panoramic task in the shooting part can be pre-statisticed, such as the generation rate of the original captured image (the number of images generated per second or the frame rate), the size and resolution of the original captured image, the delay or interval time during the shooting process and other historical execution parameters. Specifically, by considering the generation rate of the shooting part and the concurrency of the processing part, it can be ensured that the second thread can detect the newly generated original captured image in time. If the shooting rate is very high and the processing concurrency is strong, the first preset detection cycle should be set shorter. On the contrary, if the shooting rate is low or the processing concurrency is limited, the first preset detection cycle can be slightly longer. For the second preset detection cycle, the execution parameters of each panoramic task in the front-end processing and back-end synthesis parts can be pre-statisticed, such as the time required for preprocessing and feature extraction (average processing time or worst-case processing time), the concurrency during the processing process (i.e., how many images are processed simultaneously), the time required to generate a panoramic image according to the second temporary data, the size and resolution of the panoramic image, and the concurrency during the synthesis process. Specifically, the average processing time of the front-end processing part and the concurrent capability of the back-end synthesis part can be considered to ensure that the third thread can detect the second temporary data in time after the front-end processing is completed. If the front-end processing time is long and the concurrent capability of the back-end synthesis is limited, the second preset detection period should be set slightly longer to avoid waste of resources caused by frequent detection. On the contrary, if the front-end processing time is short and the concurrent capability of the back-end synthesis is strong, the second preset detection period can be slightly shorter.

[0216] In specific applications, a combination of the above-mentioned method 1 and method 2 may also be adopted to ensure the best effect.

[0217] According to the technical solution provided by the embodiment of the present disclosure, efficient panoramic image synthesis is achieved by using multiple threads (first thread, second thread) and shared memory (first memory). First, the first thread is responsible for generating and storing the original captured image and its description information in the first memory and the designated storage space. Then, the second thread periodically detects the description information in the first memory, preprocesses the existing original captured images, generates first temporary data or preprocessed images, and after all the original captured images are processed, integrates these data into second temporary data, so as to finally combine the image data of the current panoramic task with the second temporary data to generate the final panoramic image. This method reduces the waiting time by processing the shooting part and the front-end processing part in parallel through multi-threading, effectively improves the efficiency and response speed of panoramic image generation, shortens the panoramic image synthesis time, and thus improves the user experience.

[0218] In addition, in the technical solution provided by the embodiment of the present disclosure, the multithreading also includes a third thread, and the second thread and the third thread share a second memory. By storing the second temporary data generated in the second thread into the second memory, the third thread detects the second temporary data in the second memory with another preset period. Once detected, a panoramic image of the current panoramic task is generated according to the image of the current panoramic task and the second temporary data, that is, the three links of the shooting part, the front-end processing part and the back-end synthesis part are respectively put into independent threads for processing, so that the back-end synthesis part can be carried out in parallel with the shooting part and the front-end processing part, thereby further shortening the total time of panoramic image generation, further improving the efficiency and response speed of panoramic image generation, and enhancing user experience.

[0219] Fig.13 A schematic diagram of the structure of a panoramic image generation device based on multithreading according to an embodiment of the present disclosure is shown. The multithreading includes: a first thread and a second thread, and the first thread and the second thread share a first memory. Fig.13 As shown, the device 1300 includes: a first thread running module, which is configured to generate an original captured image of the current panoramic task through the first thread when generating a panoramic image of the current panoramic task, and store the original captured image in a first designated storage space, and store the description information of the original captured image in the first memory; a second thread running module, which is configured to detect whether the description information of the original captured image exists in the first memory based on a first preset detection period through the second thread, and if so, then: obtaining the original captured image from the first designated storage space according to the description information of the original captured image, pre-processing the original captured image, and generating Generate first temporary data, or generate a preprocessed image corresponding to the original captured image and the first temporary data; detect through the second thread whether all the original captured images of the current panoramic task have been processed, and if they have been processed, then: generate second temporary data corresponding to the current panoramic task according to the first temporary data of all the original captured images of the current panoramic task, so as to generate a panoramic image of the current panoramic task according to the image of the current panoramic task and the second temporary data, wherein the image of the current panoramic task includes: all the original captured images of the current panoramic task or the preprocessed images corresponding to all the original captured images of the current panoramic task.

[0220] According to an embodiment of the present disclosure, the multithreading further includes: a third thread, the second thread and the third thread share a second memory, and the second thread execution module is further configured to:

[0221] storing the second temporary data to the second memory through the second thread;

[0222] The device further includes: a third thread running module, which is configured to:

[0223] The third thread detects whether the second temporary data exists in the second memory based on a second preset detection period. If so, a panoramic image of the current panoramic task is generated based on the image of the current panoramic task and the second temporary data.

[0224] According to an embodiment of the present disclosure, the device further includes: a panoramic task switching module, which is configured to generate a panoramic image of a next panoramic task through the device when all original captured images of the current panoramic task are generated.

[0225] According to an embodiment of the present disclosure, the device further includes: an abnormal situation processing module, which is configured to: process the abnormal situation, and the processing of the abnormal situation includes any one or more of the following processes:

[0226] When generating the original captured image of the current panoramic task, the first thread detects whether the generation fails, and if so, terminates the generation of the panoramic image of the current panoramic task.

[0227] When preprocessing and feature extraction are performed on the original captured image, the second thread detects whether the original captured image or the preprocessed image corresponding to the original captured image meets the preset quality. If not, the generation of the panoramic image of the current panoramic task is terminated.

[0228] When generating the second temporary data corresponding to the current panoramic task, the second thread detects whether the second temporary data meets the preset requirements. If not, the generation of the panoramic image of the current panoramic task is terminated.

[0229] Before generating a panoramic image of the current panoramic task based on the image of the current panoramic task and the second temporary data, the third thread is used to detect whether the image of the current panoramic task and the second temporary data match. If they do not match, the generation of the panoramic image of the current panoramic task is terminated.

[0230] According to an embodiment of the present disclosure, the second thread execution module is further configured to:

[0231] The second thread generates a second temporary data file in a specified format from the second temporary data, and stores the second temporary data file in a specified storage space; after the system is restarted or after recovery is detected after an abnormal interruption, the second thread detects whether the specified storage space contains the second temporary data file; if so, the second temporary data is obtained from the second temporary data file, and the second temporary data is stored in the second memory.

[0232] The present disclosure also provides a chip, comprising the device as described in the above device embodiment; or, comprising a memory and a processor; wherein the memory is used to store one or more computer instructions, wherein the one or more computer instructions are executed by the processor to implement the method described in any one of the above method embodiments.

[0233] In a specific embodiment, the chip is a camera chip, and the processor is a CPU.

[0234] The present disclosure also provides a panoramic camera, comprising the device as described in the above device embodiment; or, comprising the chip as described above, or, comprising a memory and a processor; wherein the memory is used to store one or more computer instructions, wherein the one or more computer instructions are executed by the processor to implement the method described in any one of the above method embodiments.

[0235] In a specific embodiment, the processor in the panoramic camera is a CPU.

[0236] The present disclosure also provides a computer-readable storage medium, which may be a computer-readable storage medium included in the electronic device or computer system in the above embodiment; or a computer-readable storage medium that exists independently and is not assembled into a device. The computer-readable storage medium stores one or more programs, and the programs are used by one or more processors to execute the method described in the present disclosure.

[0237] The present disclosure also provides a computer program product, including a computer program, which implements any method described in the present disclosure when the computer program is executed by a processor.

[0238] The above description is only a preferred embodiment of the present disclosure and an explanation of the technical principles used. Those skilled in the art should understand that the scope of the invention involved in the present disclosure is not limited to the technical solution formed by a specific combination of the above technical features, but should also cover other technical solutions formed by any combination of the above technical features or their equivalent features without departing from the inventive concept. For example, the above features are replaced with the technical features with similar functions disclosed in the present disclosure (but not limited to) by each other.

Claims

1. A panoramic image generation method based on multithreading, characterized in that: The multithreading includes: a first thread and a second thread, the first thread and the second thread share a first memory; the method includes: When generating a panoramic image of a current panoramic task, each time an original photographed image of the current panoramic task is generated by the first thread, description information of the original photographed image is stored in the first memory; Detecting, by the second thread based on a first preset detection period, whether the first memory contains description information of the original captured image, and if so, then: acquiring the original captured image based on the description information of the original captured image, and preprocessing and extracting features from the original captured image based on the second thread to generate first temporary data of the original captured image, or generating a preprocessed image corresponding to the original captured image and the first temporary data of the original captured image; Whether all the original captured images of the current panoramic task have been processed is detected by the second thread. If the processing is completed, then: based on the second thread, second temporary data corresponding to the current panoramic task is generated according to the first temporary data of all the original captured images of the current panoramic task, so as to generate a panoramic image of the current panoramic task according to the image of the current panoramic task and the second temporary data, wherein the image of the current panoramic task includes: all the original captured images of the current panoramic task or pre-processed images corresponding to all the original captured images of the current panoramic task; Among them, when panoramic images of multiple panoramic tasks are continuously generated, after the first thread generates and stores all the original captured images of the current panoramic task, while the second thread processes the current panoramic task, the first thread continues to generate the original captured images of the next panoramic task without destroying or recreating the first thread.

2. The method according to claim 1, characterized in that: The multithreading further includes: a third thread, the second thread and the third thread share a second memory, and the generating of the panoramic image of the current panoramic task according to the image of the current panoramic task and the second temporary data includes: storing the second temporary data to the second memory through the second thread; The third thread detects whether the second temporary data exists in the second memory based on a second preset detection period. If so, a panoramic image of the current panoramic task is generated based on the image of the current panoramic task and the second temporary data.

3. The method according to claim 2, characterized in that The method further includes: processing the abnormal situation, wherein the processing of the abnormal situation includes any one or more of the following processing: When generating the original captured image of the current panoramic task, detecting whether the generation fails through the first thread, and if so, terminating the generation of the panoramic image of the current panoramic task; When preprocessing and feature extraction are performed on the original captured image, detecting, by the second thread, whether the original captured image or the preprocessed image corresponding to the original captured image meets the preset quality, and if not, terminating the generation of the panoramic image of the current panoramic task; When generating the second temporary data corresponding to the current panoramic task, detecting whether the second temporary data meets the preset requirements through the second thread, and if not, terminating the generation of the panoramic image of the current panoramic task; Before generating a panoramic image of the current panoramic task based on the image of the current panoramic task and the second temporary data, the third thread is used to detect whether the image of the current panoramic task and the second temporary data match. If they do not match, the generation of the panoramic image of the current panoramic task is terminated.

4. The method according to claim 2, characterized in that: The method further comprises: Generate a second temporary data file using the second temporary data in a specified format through the second thread, and store the second temporary data file in a specified storage space; After the system is restarted or after detecting an abnormal interruption and recovery, the second thread detects whether the designated storage space contains the second temporary data file. If so, the second temporary data is obtained from the second temporary data file and stored in the second memory.

5. The method according to claim 1, characterized in that The generating second temporary data corresponding to the current panoramic task according to the first temporary data of all the original photographed images of the current panoramic task comprises: Based on the first temporary data of all the original photographed images of the current panoramic task, image feature point matching is performed by using a partial matching method to generate feature point matching data of the current panoramic task; Based on the feature point matching data, globally optimize the image poses of all the original captured images of the current panoramic task, and generate a globally optimized rotation matrix of all the original captured images; The feature point matching data of the current panoramic task and the rotation matrix are used as the second temporary data.

6. The method according to claim 5, characterized in that The first temporary data based on all the original photographed images of the current panoramic task, using a partial matching method to perform image feature point matching, includes: Acquire the image shooting parameters of the current panoramic task, wherein the image shooting parameters include: a shooting sequence number and a shooting position; each original shot image corresponds to a specified shooting position and shooting sequence number; Generate a position matching relationship table of all the original captured images according to the image capturing parameters, wherein the position matching relationship table is used to describe the position matching relationship between every two original captured images, and the position matching relationship includes: position matching and position mismatching; Based on the first temporary data of the position-matched original photographed image indicated by the position matching relationship table, image feature point matching is performed on the position-matched original photographed image.

7. The method according to claim 5, characterized in that The method of globally optimizing the image poses of all the original captured images of the current panoramic task based on the feature point matching data and generating a globally optimized rotation matrix of all the original captured images includes: According to the initial rotation angle controlled by the shooting positioning device, an initial rotation matrix of the image to be optimized relative to the optimized anchor image is obtained; wherein the optimized anchor image is one of the original captured images selected from all the original captured images of the current panoramic task according to a preset rule, and the image to be optimized is the other original captured image except the optimized anchor image; Based on the feature point matching data, globally optimize the initial rotation matrix to obtain a globally optimized rotation matrix; Determine the difference between the globally optimized rotation matrix and the initial rotation matrix, and obtain an optimization result based on the difference; if the optimization result is successful, use the globally optimized rotation matrix and the unit matrix corresponding to the optimized anchor point image as the globally optimized rotation matrix of all the original captured images.

8. The method according to claim 7, characterized in that The step of generating the panoramic image of the current panoramic task according to the image of the current panoramic task and the second temporary data comprises: Acquire a camera intrinsic parameter matrix corresponding to the image of the current panoramic task; the camera intrinsic parameter matrix includes: a calibrated camera intrinsic parameter matrix or a globally optimized camera intrinsic parameter matrix; Obtaining remapped image region quadrilaterals corresponding to the images of the current panoramic task according to the camera intrinsic parameter matrix, the globally optimized rotation matrix of all the original captured images, and the vertex coordinates of the images of the current panoramic task; Determining, based on the image area quadrilaterals, a remapped image area corresponding to each image of the current panoramic task; Constructing a mapping table between each pixel in the remapped image area and each image of the current panoramic task according to the camera intrinsic parameter matrix, the globally optimized rotation matrix of all the original captured images, and the coordinates of each pixel in the remapped image area; A panoramic image of the current panoramic task is generated based on the mapping table.

9. The method according to claim 8, characterized in that When the camera intrinsic parameter matrix is ​​a globally optimized camera intrinsic parameter matrix, the method further includes: Obtaining a camera intrinsic parameter matrix to be optimized corresponding to the image of the current panoramic task; Based on the feature point matching data, globally optimize the camera intrinsic parameter matrix to be optimized to generate the globally optimized camera intrinsic parameter matrix; The step of using the feature point matching data of the current panoramic task and the rotation matrix as the second temporary data includes: Using the feature point matching data of the current panoramic task, the rotation matrix and the globally optimized camera intrinsic parameter matrix as the second temporary data; The acquiring of the camera intrinsic parameter matrix corresponding to the image of the current panoramic task includes: A camera intrinsic parameter matrix corresponding to the image of the current panoramic task is acquired based on the second temporary data.

10. A panoramic image generation device based on multithreading, characterized in that: The multithreading includes: a first thread and a second thread, the first thread and the second thread share a first memory; the device includes: a first thread running module configured to, when generating a panoramic image of a current panoramic task, generate an original photographed image of the current panoramic task through the first thread, and store description information of the original photographed image into the first memory; The second thread running module is configured to detect whether the first memory has description information of the original captured image based on a first preset detection period through the second thread, and if so, then: based on the second thread, the original captured image is acquired according to the description information of the original captured image, and the original captured image is preprocessed and feature extracted to generate first temporary data of the original captured image, or a preprocessed image corresponding to the original captured image and the first temporary data of the original captured image are generated; through the second thread, it is detected whether all the original captured images of the current panoramic task have been processed, and if so, then: based on the second thread, second temporary data corresponding to the current panoramic task are generated according to the first temporary data of all the original captured images of the current panoramic task, so as to generate a panoramic image of the current panoramic task according to the image of the current panoramic task and the second temporary data, wherein the image of the current panoramic task includes: all the original captured images of the current panoramic task or preprocessed images corresponding to all the original captured images of the current panoramic task; Among them, when panoramic images of multiple panoramic tasks are continuously generated, after the first thread generates and stores all the original captured images of the current panoramic task, while the second thread processes the current panoramic task, the first thread running module continues to generate the original captured images of the next panoramic task through the first thread without destroying or recreating the first thread.

11. A chip, characterized in that: Comprising the apparatus of claim 10; or comprising a memory and a processor; wherein the memory is used to store one or more computer instructions, wherein the one or more computer instructions are executed by the processor to implement the method of any one of claims 1 to 9.

Images