Multi-thread-based panoramic image generation method and apparatus, and chip

Abstract

The present disclosure relates to the technical field of panoramic photography and synthesis, and specifically relates to a multi-thread-based panoramic image generation method and apparatus, and a chip. In the method, multiple threads and a shared memory are used to implement efficient panoramic image synthesis. A first thread is responsible for generating original captured images and description information thereof, and storing same in a first memory and a specified storage space. A second thread periodically detects description information in the first memory, and performs preprocessing on existing original captured images, so as to generate first temporary data or preprocessed images, and after all the original captured images have been processed, the second thread integrates these data into second temporary data, and ultimately combines image data of the current panoramic task with the second temporary data, so as to generate a final panoramic image. In the method, a photographing part and a front-end processing part are processed in parallel by means of multiple threads, thereby reducing a waiting time, effectively improving the efficiency and response speed of panoramic image generation, shortening a panoramic synthesis time, and improving the user experience.

Description

Multi-threaded panoramic image generation method, apparatus and chip Technical Field

[0001] This disclosure relates to the field of panoramic photography and synthesis technology, specifically to a panoramic image generation method, apparatus, and chip based on multithreading. Background Technology

[0002] In the current technological context, panoramic photography and compositing technologies have been widely applied in various shooting scenarios, especially in the field of drone photography. Achieving 360° panoramas (spherical models) or wide-angle panoramas (planar models) through multi-angle photo compositing has become an important shooting method. However, traditional panoramic compositing technologies face a series of challenges and limitations.

[0003] First, the panoramic synthesis process is complex, involving multiple steps such as capturing photos, feature point matching, global camera pose optimization, and image fusion. These steps in current solutions need to be executed sequentially and are highly dependent on precise alignment and stitching between images. In particular, global camera pose optimization, a crucial step in ensuring panoramic image quality, typically can only begin after all images have been captured, leading to extended synthesis processing time.

[0004] Secondly, with the increase in the number of images, especially when using high-resolution images, the computational load for panoramic synthesis rises sharply, and the processing time increases significantly. This problem is particularly prominent on platforms with limited computing power, such as drones, where panoramic image synthesis can take tens of seconds, not only delaying valuable shooting opportunities but also seriously affecting the user experience.

[0005] How to shorten the panoramic image synthesis time, improve synthesis efficiency, and thus enhance the user experience is an urgent problem to be solved. Summary of the Invention

[0006] To address the problems in the related technologies, this disclosure provides a method, apparatus, and chip for generating panoramic images based on multithreading.

[0007] In a first aspect, this disclosure provides a multi-threaded panoramic image generation method, wherein the multi-threading includes: a first thread and a second thread, the first thread and the second thread sharing a first memory; the method includes:

[0008] When generating the panoramic image for the current panoramic task, the original captured image of the current panoramic task is generated through the first thread, and the description information of the original captured image is stored in the first memory.

[0009] The second thread detects whether the description information of the original captured image exists in the first memory based on the first preset detection period. If it exists, then: the original captured image is obtained according to the description information of the original captured image, the original captured image is preprocessed and feature extracted, and first temporary data is generated, or a preprocessed image and first temporary data corresponding to the original captured image are generated.

[0010] The second thread detects whether all the original images captured in the current panoramic task have been processed. If the processing is complete, then: a second temporary data corresponding to the current panoramic task is generated based on the first temporary data of all the original images captured in the current panoramic task, so as to generate a panoramic image of the current panoramic task based on the image of the current panoramic task and the second temporary data. The image of the current panoramic task includes: all the original images captured in the current panoramic task or the preprocessed image corresponding to all the original images captured in the current panoramic task.

[0011] According to embodiments of this disclosure, the multithreading further includes: a third thread, wherein the second thread and the third thread share a second memory, and the step of generating the panoramic image of the current panoramic task based on the image of the current panoramic task and the second temporary data includes:

[0012] The second temporary data is stored in the second memory via the second thread;

[0013] The third thread detects whether the second temporary data exists in the second memory based on the second preset detection period. If it exists, then: the panoramic image of the current panoramic task is generated based on the image of the current panoramic task and the second temporary data.

[0014] According to an embodiment of this disclosure, the method further includes: when all original captured images of the current panoramic task have been generated, generating a panoramic image for the next panoramic task using the method.

[0015] According to embodiments of this disclosure, the method further includes: handling abnormal situations, wherein handling abnormal situations includes any one or more of the following processes:

[0016] When generating the original captured image of the current panoramic task, the first thread checks whether the generation has failed. If so, the generation of the panoramic image of the current panoramic task is terminated.

[0017] When preprocessing and extracting features from the original captured image, the second thread detects whether the original captured image or the preprocessed image corresponding to the original captured image meets the preset quality. If it does not meet the preset quality, the generation of the panoramic image for the current panoramic task is terminated.

[0018] When generating the second temporary data corresponding to the current panoramic task, the second thread checks whether the second temporary data meets the preset requirements. If it does not meet the requirements, the generation of the panoramic image of the current panoramic task is terminated.

[0019] Before generating the panoramic image of the current panoramic task based on the image of the current panoramic task and the second temporary data, the third thread checks whether the image of the current panoramic task and the second temporary data match. If they do not match, the generation of the panoramic image of the current panoramic task is terminated.

[0020] According to embodiments of this disclosure, the method further includes:

[0021] The second thread generates a second temporary data file from the second temporary data in a specified format, and stores the second temporary data file in a specified storage space.

[0022] After the system restarts or after detecting and recovering from an abnormal interruption, the second thread checks whether the specified storage space contains the second temporary data file. If it does, the second temporary data is retrieved from the second temporary data file and stored in the second memory.

[0023] According to embodiments of this disclosure, the preprocessing and feature extraction of the original captured image includes:

[0024] The original captured image is subjected to image visual attribute processing to generate a processed captured image;

[0025] Determine whether the size of the original captured image exceeds a preset threshold. If so, perform image size reduction processing on the original captured image to generate the preprocessed image corresponding to the original captured image.

[0026] Image feature point information is extracted from the preprocessed image corresponding to the original captured image or the processed captured image, and the extracted image feature point information is used as the first temporary data of the original captured image;

[0027] The image visual attribute processing includes any combination of the following processing methods: adjusting image contrast, adjusting image brightness, and histogram equalization.

[0028] According to embodiments of this disclosure, generating second temporary data corresponding to the current panoramic task based on first temporary data of all original captured images of the current panoramic task includes:

[0029] Based on the first temporary data of all original images captured in the current panoramic task, image feature point matching is performed using a partial matching method to generate feature point matching data for the current panoramic task.

[0030] Based on the feature point matching data, the image pose of all original images captured in the current panoramic task is globally optimized to generate a rotation matrix of all original images captured globally optimized.

[0031] The feature point matching data of the current panoramic task and the rotation matrix are used as the second temporary data.

[0032] According to embodiments of this disclosure, the first temporary data based on all original captured images of the current panoramic task, employing a partial matching method for image feature point matching, includes:

[0033] Obtain the image capture parameters for the current panoramic task, wherein the image capture parameters include: capture sequence number and capture position; each original captured image corresponds to a specified capture position and capture sequence number;

[0034] A position matching table for all original captured images is generated based on the image capture parameters. The position matching table describes the position matching relationship between any two original captured images. The position matching relationship includes: position matching and position mismatch.

[0035] Based on the first temporary data of the original captured image with the location matching indicated by the location matching relationship table, image feature point matching is performed on the original captured image with the location matching.

[0036] According to embodiments of this disclosure, the step of globally optimizing the image pose of all original images captured in the current panoramic task based on the feature point matching data, and generating a globally optimized rotation matrix for all original images captured, includes:

[0037] Based on the initial rotation angle controlled by the shooting and positioning device, an initial rotation matrix is ​​obtained for the image to be optimized relative to the optimized anchor point image; wherein, the optimized anchor point image is one of the original images selected from all the original images of the current panoramic task according to a preset rule, and the image to be optimized is another original image other than the optimized anchor point image;

[0038] Based on the feature point matching data, the initial rotation matrix is ​​globally optimized to obtain the globally optimized rotation matrix;

[0039] Determine the difference between the globally optimized rotation matrix and the initial rotation matrix, and obtain the optimization result based on the difference; if the optimization result is successful, then use the globally optimized rotation matrix and the identity matrix corresponding to the optimized anchor point image as the globally optimized rotation matrix of all original captured images.

[0040] According to embodiments of this disclosure, generating the panoramic image of the current panoramic task based on the image of the current panoramic task and the second temporary data includes:

[0041] Obtain the camera intrinsic parameter matrix corresponding to the image of the current panoramic task; the camera intrinsic parameter matrix includes: a calibrated camera intrinsic parameter matrix or a globally optimized camera intrinsic parameter matrix;

[0042] Based on the camera intrinsic parameter matrix, the rotation matrix of all original captured images after global optimization, and the vertex coordinates of the current panoramic task image, the remapped quadrilaterals of each image region corresponding to each image of the current panoramic task are obtained.

[0043] Based on the quadrilaterals of each image region, the remapping image regions corresponding to each image of the current panoramic task are determined;

[0044] Based on the camera intrinsic parameter matrix, the rotation matrix of all original captured images after global optimization, and the pixel coordinates in the remapped image region, a mapping table is constructed between each pixel in the remapped image region and each image of the current panoramic task.

[0045] The panoramic image for the current panoramic task is generated based on the mapping table.

[0046] According to embodiments of this disclosure, when the camera intrinsic parameter matrix is ​​a globally optimized camera intrinsic parameter matrix, the method further includes:

[0047] Obtain the camera intrinsic parameter matrix to be optimized corresponding to the image of the current panoramic task;

[0048] Based on the feature point matching data, the camera intrinsic parameter matrix to be optimized is globally optimized to generate the globally optimized camera intrinsic parameter matrix.

[0049] The step of using the feature point matching data of the current panoramic task and the rotation matrix as the second temporary data includes:

[0050] The feature point matching data of the current panoramic task, the rotation matrix, and the globally optimized camera intrinsic parameter matrix are used as the second temporary data.

[0051] The step of obtaining the camera intrinsic parameter matrix corresponding to the image of the current panoramic task includes:

[0052] Based on the second temporary data, obtain the camera intrinsic parameter matrix corresponding to the image of the current panoramic task.

[0053] Secondly, this disclosure provides a multi-threaded panoramic image generation device, wherein the multi-threading includes: a first thread and a second thread, the first thread and the second thread sharing a first memory; the device includes:

[0054] The first thread running module is configured to generate the original captured image of the current panoramic task through the first thread when generating the panoramic image of the current panoramic task, and store the description information of the original captured image in the first memory.

[0055] The second thread execution module is configured to detect whether the description information of the original captured image exists in the first memory based on a first preset detection period. If it exists, then: the original captured image is obtained according to the description information of the original captured image; the original captured image is preprocessed and its features are extracted to generate first temporary data, or a preprocessed image and the first temporary data corresponding to the original captured image are generated; the second thread detects whether all the original captured images of the current panoramic task have been processed. If they have been processed, then: a second temporary data corresponding to the current panoramic task is generated based on the first temporary data of all the original captured images of the current panoramic task, so as to generate the panoramic image of the current panoramic task based on the image of the current panoramic task and the second temporary data. The image of the current panoramic task includes: all the original captured images of the current panoramic task or the preprocessed image corresponding to all the original captured images of the current panoramic task.

[0056] Thirdly, this disclosure provides a chip that includes the apparatus described in the second aspect; or, includes a memory and a processor; wherein the memory is used to store one or more computer instructions, wherein the one or more computer instructions are executed by the processor to implement the method described in any one aspect.

[0057] According to the technical solution provided in this disclosure, efficient panoramic image synthesis is achieved using multiple threads (a first thread and a second thread) and shared memory (a first memory). First, the first thread generates and stores the original captured images and their descriptive information in the first memory and a designated storage space. Next, the second thread periodically checks the descriptive information in the first memory, preprocesses the existing original captured images, generates first temporary data, and combines the first temporary data with the preprocessed image. After all original captured images have been processed, this data is integrated into second temporary data. Finally, the image data of the current panoramic task is combined with the second temporary data to generate the final panoramic image. This method reduces waiting time by using multiple threads to process the capturing and front-end processing parts in parallel, effectively improving the efficiency and response speed of panoramic image generation, shortening the panoramic image synthesis time, and thus enhancing the user experience.

[0058] Furthermore, in the technical solution provided in this embodiment, the multithreading also includes a third thread, and the second thread and the third thread share a second memory. The second thread stores the generated second temporary data into the second memory, and the third thread detects the second temporary data in the second memory at another preset period. Once detected, the panoramic image of the current panoramic task is generated based on the image of the current panoramic task and the second temporary data. That is, the three stages of shooting, front-end processing and back-end compositing are processed in separate threads, so that the back-end compositing can be performed in parallel with the shooting and front-end processing, thereby further shortening the total time of panoramic image generation, further improving the efficiency and response speed of panoramic image generation, and enhancing the user experience.

[0059] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this disclosure. Attached Figure Description

[0060] Other features, objects, and advantages of this disclosure will become more apparent from the following detailed description of non-limiting embodiments, taken in conjunction with the accompanying drawings. In the drawings:

[0061] Figure 1 shows a flowchart of a multi-threaded panoramic image generation method according to an embodiment of the present disclosure;

[0062] Figure 2 shows a flowchart of the operation method of the first thread in the shooting part of the panoramic image generation task according to an embodiment of the present disclosure;

[0063] Figure 3 shows a flowchart of a method for generating second temporary data according to an embodiment of the present disclosure;

[0064] Figure 4 shows a schematic diagram of the shooting positions and arrangement of all original images captured in the current panoramic task according to an embodiment of the present disclosure;

[0065] Figure 5 shows a schematic diagram of a position matching table generated based on the positional relationships of the images shown in Figure 4 according to an embodiment of the present disclosure;

[0066] Figure 6 shows a flowchart of the operation method of the second thread in the front-end processing part of the panoramic image generation task according to an embodiment of the present disclosure;

[0067] Figure 7 shows a flowchart of a method for generating a panoramic image of the current panoramic task according to an embodiment of the present disclosure;

[0068] Figure 8 shows a schematic diagram of the remapping regions of each original captured image shown in Figure 4 according to an embodiment of the present disclosure;

[0069] Figure 9 shows a schematic diagram of the final effect obtained after completely remapping the original image;

[0070] Figure 10 illustrates a method for determining the boundary region when synthesizing panoramic images according to an embodiment of the present disclosure;

[0071] Figure 11 shows a schematic diagram of the final effect obtained by remapping a portion of the original image according to an embodiment of the present disclosure;

[0072] Figure 12 shows a flowchart of the operation method of the third thread in the back-end compositing part of the panoramic image generation task according to an embodiment of the present disclosure;

[0073] Figure 13 shows a schematic diagram of a multi-threaded panoramic image generation apparatus according to an embodiment of the present disclosure. Detailed Implementation

[0074] In the following, exemplary embodiments of the present disclosure will be described in detail with reference to the accompanying drawings to enable those skilled in the art to readily implement them. Furthermore, for clarity, portions unrelated to the description of exemplary embodiments have been omitted from the drawings.

[0075] In this disclosure, it should be understood that terms such as “comprising” or “having” are intended to indicate the presence of features, figures, steps, behaviors, components, parts or combinations thereof disclosed in this specification, and are not intended to exclude the possibility of the presence or addition of one or more other features, figures, steps, behaviors, components, parts or combinations thereof.

[0076] It should also be noted that, unless otherwise specified, the embodiments and features described in this disclosure can be combined with each other. This disclosure will now be described in detail with reference to the accompanying drawings and embodiments.

[0077] As mentioned earlier, existing panoramic image synthesis solutions suffer from long processing times. When there are a large number of images or the resolution is high, the computational load increases dramatically. Especially on platforms with limited computing resources, such as drones, the synthesis time can be as long as tens of seconds, which not only affects the user experience but may also cause important shooting opportunities to be missed.

[0078] In order to shorten the panoramic image synthesis time, improve the synthesis efficiency, and thus enhance the user experience, the inventors of this disclosure have carefully studied and considered the existing panoramic image synthesis schemes from multiple dimensions, and have optimized and improved them, providing a simple and efficient panoramic image generation method.

[0079] Based on a deep understanding of panoramic image generation technology and an accurate grasp of the characteristics of each stage, the inventors of this disclosure decompose the originally single-threaded sequential panoramic image generation task into three independent parts: shooting, front-end processing, and back-end compositing. They also introduced a multi-threaded parallel collaborative processing architecture, in which each part is executed in a separate thread, and adjacent parts share the same memory for data exchange, thereby achieving parallel processing. The imaging section is primarily responsible for acquiring the raw photos needed for panoramic imaging. Separating the imaging section ensures that it can focus on obtaining high-quality raw photos without interference from other processing stages. The front-end processing section is mainly responsible for preprocessing and feature extraction of the captured raw photos. Since the front-end processing section requires strong computing power and algorithm support, separating it allows for full utilization of computing resources, accelerating the image preprocessing and feature extraction processes. Furthermore, the front-end processing section can be executed in parallel with other sections, thereby improving overall processing efficiency. The back-end compositing section is mainly responsible for stitching the preprocessed images or the original captured images based on the feature point matching results obtained from the front-end processing to generate the final panoramic image. Because the back-end compositing section needs to process a large amount of image data and complex stitching algorithms, separating it ensures that, with sufficient resources, it can focus on completing the panoramic image stitching and optimization tasks. Furthermore, the back-end compositing section can be executed in parallel with other sections, further improving overall processing efficiency.

[0080] In summary, decomposing the panoramic image generation process into three parts—capture, front-end processing, and back-end compositing—and executing each part in a separate thread is a reasonable and efficient solution based on the characteristics of each part. Each thread can independently allocate and use computing resources, avoiding resource contention and waiting time, thus optimizing resource utilization. Furthermore, independent thread processing of each task reduces system coupling, enhancing system stability and scalability. If a problem occurs in one part, it will not affect the normal operation of other parts, facilitating troubleshooting and repair. Therefore, this decomposition method can fully utilize computing resources, improve processing efficiency, and enhance system stability and scalability.

[0081] Figure 1 shows a flowchart of a multi-threaded panoramic image generation method according to an embodiment of the present disclosure. The multi-threading includes a first thread and a second thread, the first thread and the second thread sharing a first memory. As shown in Figure 1, the method includes the following steps S110 to S130:

[0082] In step S110, when generating the panoramic image of the current panoramic task, the original captured image of the current panoramic task is generated through the first thread, and the description information of the original captured image is stored in the first memory.

[0083] According to embodiments of this disclosure, when generating the original captured image for the current panoramic task, the first thread detects whether generation has failed. If so, the generation of the panoramic image for the current panoramic task is terminated. This way, if a problem is detected and panoramic image generation is terminated during the generation of the original captured image, unnecessary subsequent computation and resource consumption can be avoided. This helps save system resources and improve overall processing efficiency.

[0084] For example, during gimbal movement or camera shooting, the first thread continuously monitors the operation status. If an abnormality in gimbal movement (such as mechanical stall) or a camera shooting error (such as insufficient storage space) is detected, the shooting process is interrupted, and the error information is recorded for subsequent processing.

[0085] The original images disclosed herein refer to original photographs taken using a panoramic camera equipped with a wide-angle or fisheye lens. Original photographs from different angles can be taken using a gimbal with a motor, a robotic arm to control camera movement, or even manually. The key is to follow certain steps to take photographs from different angles.

[0086] In this disclosure, the first thread is a separate thread responsible for the shooting process. Figure 2 shows a flowchart of the operation method of the first thread in the shooting part of the panoramic image generation task according to an embodiment of this disclosure. Taking the shooting positioning device (e.g., gimbal) controlling the camera to acquire the original shooting image of the current panoramic task as an example, as shown in Figure 2, the following steps S111 to S116 are included:

[0087] In step S111, receiving and processing user instructions includes: continuously listening for instruction input from the user or system through a first thread; when an instruction to capture a panoramic image is received, parsing the instruction to obtain the image capture information of the current panoramic task; and initializing a panoramic capture task object based on the parsed image capture information.

[0088] The image capture information includes the shooting angle range (e.g., horizontal 180°), shooting interval, image resolution, and other possible shooting parameters; the panoramic shooting task object contains all the necessary shooting information and status tracking.

[0089] In step S112, the shooting and positioning device is controlled to move the panoramic camera and take photos according to the image shooting information, generating the original shooting image of the current panoramic task.

[0090] Additionally, after generating the original images for the current panoramic task, they can be stored in a designated storage space for the second thread to retrieve.

[0091] Specifically, based on the image capture information, the number of shots and all specific angles corresponding to all original captured images of the current panoramic task are calculated. For example, for a horizontal 180° panorama, if one shot is taken every 10°, 19 shooting points (including the start and end angles) need to be calculated, thus generating 19 original captured images. For each calculated angle, a control command is sent to the shooting and positioning device to move it to the specified angle. After it moves into position, the panoramic camera is triggered to take a picture. After the shooting is completed, the original captured image is obtained from the camera and stored in a specified storage space so that the second thread can correctly obtain the corresponding original captured image. The specified storage space is usually a temporary folder or a memory buffer.

[0092] In step S113, it is detected whether the generation of the original captured image has failed. If so, the generation of the panoramic image for the current panoramic task is terminated. If not, step S114 is executed.

[0093] In step S114, descriptive information of the original captured image is generated and stored in the first memory.

[0094] In this disclosure, each time an original image of the current panoramic task is generated, its description information can be stored in the first memory for the second thread to acquire and process. In this way, the first thread and the second thread can process in parallel, which improves the synthesis efficiency.

[0095] The descriptive information of the original captured image includes, but is not limited to, the photo file name, shooting angle, and photo file path. This information will be used for subsequent image processing and panoramic image generation.

[0096] The memory (including the first memory and the second memory) disclosed herein refers to a storage area in system memory (such as random access memory, RAM) that can be read and written to. It can be non-volatile or volatile.

[0097] When storing descriptive information, it can be based on message queues, which typically provide synchronization mechanisms to ensure correct data delivery. Other data formats, such as linked lists, can also be used.

[0098] In step S115, it is detected whether all the original images of the current panoramic task have been captured. If not, step S112 above is executed; if so, step S116 below is executed.

[0099] In step S116, the task is completed and the status is updated.

[0100] In one specific implementation, once all original images for the current panoramic task have been captured and all image and descriptive information has been correctly stored, the first thread updates the status of the panoramic capture task object to "capture completed". Furthermore, the first thread can also notify the system or user that the capture task is complete, and provide any necessary feedback or log information.

[0101] In addition, in a specific embodiment of this disclosure, each part of the shooting, front-end processing, and back-end compositing can run in a separate thread. In this way, the processing of each part is relatively independent. This multi-threaded processing method allows the system to process the next panoramic task in parallel through steps S110 to S130 without destroying or recreating the first thread, while processing the subsequent parts of the current panoramic task (front-end processing or back-end compositing), and generate the panoramic image of the next panoramic task. This achieves seamless switching and resource reuse between different panoramic tasks.

[0102] In step S120, the second thread detects whether the description information of the original captured image exists in the first memory based on the first preset detection period. If it exists, then: the original captured image is obtained according to the description information of the original captured image, the original captured image is preprocessed and feature extracted, and first temporary data is generated, or a preprocessed image and first temporary data corresponding to the original captured image are generated.

[0103] The implementation details of step S120 are described below:

[0104] In step S120, feature points are extracted based on the original captured image generated in step S110, which is then used for feature point matching in the subsequent step S130. The first temporary data refers to the feature point information corresponding to the specified original captured image.

[0105] According to embodiments of this disclosure, the preprocessing and feature extraction of the original captured image includes:

[0106] The original captured image is subjected to image visual attribute processing to generate a processed captured image; it is determined whether the size of the original captured image exceeds a preset threshold. If so, the image size of the original captured image is reduced to generate a preprocessed image corresponding to the original captured image; image feature point information is extracted from the preprocessed image corresponding to the original captured image or the processed captured image, and the extracted image feature point information is used as the first temporary data of the original captured image.

[0107] The image visual attribute processing includes, but is not limited to, any combination of the following processing methods: adjusting image contrast, adjusting image brightness, and histogram equalization.

[0108] In this disclosure, before feature extraction from the original captured image, pre-processing such as adjusting image contrast, brightness, and histogram equalization is performed on the original captured image to enhance the stability of feature point extraction. For large images, image size reduction is performed, and feature extraction is based on the reduced image. The reduced image is also used in the back-end synthesis process. This reduces the time required for feature point extraction and accelerates back-end algorithms such as seam generation, thereby shortening the panoramic image synthesis time.

[0109] When extracting feature points, existing algorithms can be used, such as SIFT (Scale-Invariant Feature Transform), SURF (Speeded-Up Robust Features), and ORB (Oriented FAST and Rotated BRIEF).

[0110] According to embodiments of this disclosure, during preprocessing and feature extraction of the original captured image, a second thread detects whether the original captured image or the preprocessed image corresponding to the original captured image meets a preset quality standard. If it does not meet the standard, such as insufficient image feature points or inability to extract them, the generation of the panoramic image for the current panoramic task is terminated. In this way, quality detection ensures that only images meeting the preset quality standard are used for panoramic image generation. This helps avoid using blurry, distorted, or low-quality images, thereby improving the overall quality of the final panoramic image, enhancing the user experience, and avoiding unnecessary waste of computational resources, saving CPU and memory resources.

[0111] In step S130, the second thread detects whether all the original images captured in the current panoramic task have been processed. If the processing is complete, then: a second temporary data corresponding to the current panoramic task is generated based on the first temporary data of all the original images captured in the current panoramic task, so as to generate a panoramic image of the current panoramic task based on the image of the current panoramic task and the second temporary data. The image of the current panoramic task includes: all the original images captured in the current panoramic task or the preprocessed image corresponding to all the original images captured in the current panoramic task.

[0112] The implementation details of step S130 are described below:

[0113] In step S130, feature point matching is performed based on the feature point information of all original captured images from step S120 to form feature point pairs, and the image pose is globally optimized to generate a globally optimized rotation matrix. The second temporary data mainly includes feature point matching data and the globally optimized rotation matrix.

[0114] Figure 3 shows a flowchart of a method for generating second temporary data according to an embodiment of the present disclosure. As shown in Figure 3, when generating second temporary data corresponding to the current panoramic task based on first temporary data of all original captured images of the current panoramic task, the method includes the following steps S131-133:

[0115] In step S131, based on the first temporary data of all original captured images of the current panoramic task, image feature point matching is performed using a partial matching method to generate feature point matching data for the current panoramic task.

[0116] According to embodiments of this disclosure, the first temporary data based on all original captured images of the current panoramic task, employing a partial matching method for image feature point matching, includes:

[0117] Obtain the image capture parameters for the current panoramic task, wherein the image capture parameters include: capture sequence number and capture position; each original captured image corresponds to a specified capture position and capture sequence number.

[0118] A position matching table is generated based on the image capture parameters to represent the positional matching relationships between all the original captured images. This table describes the positional matching relationships between any two original captured images, including both positional matches and positional mismatches. In a specific example, "1" indicates a positional match, and "0" indicates a positional mismatch.

[0119] Based on the first temporary data of the original captured image with the location matching indicated by the location matching relationship table, image feature point matching is performed on the original captured image with the location matching.

[0120] Taking a panoramic composite of 9 original images arranged in a 3x3 grid as an example:

[0121] In this example, the gimbal is controlled to rotate in 9 different directions to take 9 photos. Taking a rotation of 25° intervals in the heading direction and 20° intervals in the pitch direction as an example, Figure 4 shows a schematic diagram of the shooting positions and arrangement of all the original captured images of the current panoramic task according to an embodiment of the present disclosure. As shown in Figure 4, the current panoramic task includes 9 original captured images, namely: image 1 to image 9. Figure 5 shows a schematic diagram of the position matching relationship table generated based on the positional relationship of the images shown in Figure 4 according to an embodiment of the present disclosure. As shown in Figure 5, image 7 and image 3 have no overlapping area, so no matching is required, and the values ​​of <row 7, column 3> and <row 3, column 7> in the table are 0. Image 1 and image 4 have overlapping area, so matching is required, and the values ​​of <row 4, column 1> and <row 1, column 4> in the table are 1.

[0122] The matching relationships in the location matching table can be customized according to specific needs. For example, if the first and third images only overlap diagonally, they can be matched or not. Additionally, the matching table can be pre-generated manually and saved in the system. When there are many images, it can also be automatically generated programmatically. For instance, if each image only matches images in the four directions (up, down, left, and right), the program will automatically fill in a 1 in the corresponding direction's table.

[0123] Traditional panoramic synthesis algorithms, when the arrangement of images is unknown, require a full match of each image during the feature point matching step; that is, the matching relationship needs to be checked between every two images.

[0124] This disclosure innovatively introduces a partial matching method. Based on image capture parameters (capture sequence number and capture location), a positional matching table describing all image matching relationships is pre-determined. Feature point matching is then performed based on this table. In this example, the number of image matches can be reduced from 72 to 24. Therefore, this method can significantly narrow the matching range of images, substantially reduce the number of matches, and lower the probability of false matches between images with non-overlapping regions.

[0125] In step S132, based on the feature point matching data, the image pose of all original captured images of the current panoramic task is globally optimized to generate a globally optimized rotation matrix of all original captured images.

[0126] According to embodiments of this disclosure, the step of globally optimizing the image pose of all original images captured in the current panoramic task based on the feature point matching data, and generating a globally optimized rotation matrix for all original images captured, includes:

[0127] Based on the initial rotation angle controlled by the shooting and positioning device (e.g., gimbal), the initial rotation matrix of the image to be optimized relative to the optimized anchor point image is obtained; wherein, the optimized anchor point image is one of the original images selected from all the original images of the current panoramic task according to a preset rule, and the image to be optimized is another original image other than the optimized anchor point image.

[0128] Based on the feature point matching data, the initial rotation matrix is ​​globally optimized to obtain the globally optimized rotation matrix.

[0129] Determine the difference between the globally optimized rotation matrix and the initial rotation matrix, and obtain the optimization result based on the difference; if the optimization result is successful, then use the globally optimized rotation matrix and the identity matrix corresponding to the optimized anchor point image as the globally optimized rotation matrix of all original captured images.

[0130] Let's take a panoramic composite of 9 original images arranged in a 3x3 grid as an example:

[0131] As shown in Figure 4, for each image's initial rotation angle, for example, the yaw and pitch angles of the first image are both 0, and the yaw angle of the fifth image is 25 degrees and the pitch angle is 20 degrees. The first image can be used as the optimization anchor image, meaning its rotation is not optimized. Therefore, images 2 through 9 are the images to be optimized, meaning their rotation needs to be optimized. Generally, the optimization anchor image is the image that is spatially positioned in the middle (the first image in this example), but it can also be any arbitrarily chosen image.

[0132] When obtaining the initial rotation matrix of the image to be optimized relative to the optimization anchor image, it can be done as follows:

[0133] First, determine the yaw, roll, and pitch angles of each image to be optimized relative to the optimization anchor image. Then, for the i-th image, its Euler angles can be represented as (yaw_i, roll_i, pitch_i).

[0134] Then, using the conversion formula from Euler angles to rotation matrices, each Euler angle is converted into its corresponding rotation matrix.

[0135] The specific conversion process can be divided into three steps:

[0136] Step 1: Rotate the axis around the z-axis by an angle yaw_i to obtain the rotation matrix R. z_i .

[0137] Step 2: Rotate pitch_i around the y-axis by an angle to obtain the rotation matrix R. y_i .

[0138] Step 3: Rotate the x-axis by an angle roll_i to obtain the rotation matrix R. x_i .

[0139] The final rotation matrix R' i R' can be obtained through matrix multiplication. i =R z_i *R y_i *R x_i .

[0140] Finally, repeat the above steps for each image (except for the optimized anchor point image) to obtain 8 initial rotation matrices R'2 to R'9.

[0141] After obtaining the initial rotation matrix, a global optimization objective function can be defined to minimize the reprojection error of feature points between all adjacent images. For each image, the rotation matrix corresponding to that image is introduced as an optimization variable (for the optimized anchor image, its rotation matrix is ​​fixed as an identity matrix), and a nonlinear optimization algorithm (such as the Levenberg-Marquardt algorithm or the Gauss-Newton algorithm) is used to minimize the global reprojection error. During the optimization process, the rotation matrix of the optimized anchor image remains unchanged, and only the remaining 8 images are optimized to generate the globally optimized rotation matrices R2 to R9.

[0142] Due to controller and camera intrinsic parameter errors, the globally optimized R2 to R9 will differ slightly from the initial R'2 to R'9. If the difference is too large, it indicates an optimization error. This can be corrected by verifying R'... n With R n The difference in R' can be used to determine whether the optimization was successful. n With R n Difference R n-diff R' can be expressed using the following formula (1): n *R n =R n-diff (1)

[0143] If R' n =R n Then: R n-diff It should be a unit array. By comparing R n-diff The difference between R and the identity matrix determines the optimization result. For example: R n-diff Convert to axis angle, where the magnitude of the axis angle does not exceed θ1. R can also be... n-diff Converted to Euler angles, the angular magnitudes of all three axes of the Euler angles do not exceed θ2.

[0144] In a specific project, the inventors of this disclosure noted that: generally, when the gimbal controller error does not exceed 1°, θ1 can be taken as 10° and θ2 can be taken as 3°. If the angle exceeds this, a visible splicing error can be generated. This can be used to judge the optimization failure in advance and end the splicing task.

[0145] This disclosure introduces an optimized anchor point image during global optimization, meaning there is a fixed reference point throughout the optimization process. This reference point remains constant, providing a stable reference for other images and helping to maintain the stability and consistency of the entire image stitching. Simultaneously, by globally optimizing the rotation matrix of the image to be optimized relative to the optimized anchor point image, it ensures that all images are adjusted relative to a common starting point during optimization. This relative optimization method helps reduce the cumulative error caused by independent optimization, thus avoiding overall drift. Furthermore, a mechanism is introduced to obtain the optimization result based on the difference between the globally optimized rotation matrix and the initial rotation matrix. This difference verification mechanism provides additional quality control for the optimization process, helping to promptly detect and correct errors. Therefore, the problems of overall drift and abnormal errors that can occur in global optimization are effectively avoided.

[0146] In step S133, the feature point matching data of the current panoramic task and the rotation matrix are used as the second temporary data.

[0147] According to embodiments of this disclosure, when generating second temporary data corresponding to the current panoramic task, the second thread checks whether the second temporary data meets preset requirements. If it does not, the generation of the panoramic image for the current panoramic task is terminated. Thus, by checking whether the second temporary data meets the preset requirements, it can be ensured that the generation of the panoramic image is based on accurate and reliable data. If the data does not meet the requirements, it may mean that there are problems with the processing or feature extraction of the original captured image, or that the stitching and fusion of the panoramic images will not yield satisfactory results. In this case, terminating the generation of the panoramic image can avoid generating low-quality images, thereby improving the accuracy of panoramic image generation.

[0148] The operation of generating the panoramic image of the current panoramic task based on the image of the current panoramic task and the second temporary data involved in step S130 above can be executed directly in the second thread or in a separate thread. This disclosure provides a third thread, that is, this operation is processed separately in a third thread. In this way, the three parts involved in the panoramic image generation process—the shooting part, the front-end processing part, and the back-end compositing part—are all processed through separate threads, which further shortens the total time for panoramic image generation.

[0149] According to embodiments of this disclosure, the multithreading further includes: a third thread, wherein the second thread and the third thread share a second memory, and the step of generating the panoramic image of the current panoramic task based on the image of the current panoramic task and the second temporary data includes:

[0150] The second temporary data is stored in the second memory via the second thread;

[0151] The third thread detects whether the second temporary data exists in the second memory based on the second preset detection period. If it exists, then: the panoramic image of the current panoramic task is generated based on the image of the current panoramic task and the second temporary data.

[0152] The following describes the execution details of the second thread:

[0153] In this disclosure, the second thread, as a separate thread, is responsible for the flow of the front-end processing section. Figure 6 shows a flowchart of the operation method of the second thread in the front-end processing section of the panoramic image generation task according to an embodiment of this disclosure. As shown in Figure 6, the steps S121 to S12b are included:

[0154] In step S121, it is detected whether the description information of the original captured image exists in the first memory. If it exists, then step S122 is executed.

[0155] In step S122, the original captured image is obtained based on the description information of the original captured image.

[0156] In step S123, it is determined whether the size of the original captured image exceeds a preset threshold. If yes, then step S124 is executed; otherwise, step S125 is executed.

[0157] In step S124, the original captured image is reduced in size to generate the preprocessed image corresponding to the original captured image.

[0158] In step S125, the original captured image is subjected to image visual attribute processing to generate a processed captured image.

[0159] In step S126, it is detected whether the processed captured image or the preprocessed image meets the preset quality. If not, the generation of the panoramic image for the current panoramic task is terminated. If yes, the following step S127 is executed.

[0160] In step S127, image feature point information is extracted from the preprocessed image corresponding to the original captured image or the processed captured image, and the extracted image feature point information is used as the first temporary data of the original captured image.

[0161] In step S128, it is detected whether all the original captured images for the current panoramic task have been processed. If the processing is complete, proceed to step S129; if not, return to step S121.

[0162] In step S129, second temporary data corresponding to the current panoramic task is generated based on the first temporary data of all original captured images of the current panoramic task.

[0163] In step S12a, it is detected whether the second temporary data meets the preset requirements. If it does not meet the requirements, the generation of the panoramic image of the current panoramic task is terminated. If it meets the requirements, step S12b is executed.

[0164] In step S12b, the second temporary data is stored in the second memory.

[0165] According to embodiments of this disclosure, after generating second temporary data corresponding to the current panoramic task based on first temporary data of all original captured images of the current panoramic task, the second thread can further generate a second temporary data file in a specified format from the second temporary data and store the second temporary data file in a specified storage space; wherein, the specified storage space has non-volatile storage characteristics, such as: it can be a specified storage area in a hard disk (HDD) or certain types of persistent memory (such as NVRAM).

[0166] Thus, after the system restarts or recovers from an abnormal interruption, the following steps S12c and S12d are also included:

[0167] In step S12c, it is detected whether the specified storage space contains the second temporary data file. If it does, step S12d is executed, that is, the second temporary data is obtained from the second temporary data file, and then step S12b is executed, that is, the second temporary data is stored in the second memory.

[0168] The inventors of this publication noticed certain risks during the development of the project, such as power outages and other abnormal situations, in the process of panoramic image synthesis. If the system is interrupted for any reason, and the second temporary data is only stored in memory, this data is likely to be lost. After the system restarts, the synthesis process for the current panoramic task may be forced to stop, causing the already completed synthesis work to become invalid. This necessitates re-capturing the original photos and performing subsequent processing, which increases the panoramic image synthesis time and causes unnecessary losses and inconvenience to the user.

[0169] This disclosure ensures that data used for backend synthesis is not lost after abnormal system interruptions (such as power outages or system crashes) by generating a second temporary data file from the second temporary data in a specified format and storing it in a specified storage space. Thus, even if the system restarts or malfunctions, for the current panoramic image generation, data can be recovered simply by reading these files from the specified storage space, thereby resuming the interrupted task and shortening the panoramic image synthesis time. Furthermore, storing the second temporary data as a file facilitates data management and debugging. For example, developers can check these files to verify the correctness of the data, or locate the problem by tracing back through these files when issues arise.

[0170] It should be noted that if a preprocessed image corresponding to the original captured image is generated in step S120, that is, an image size reduction operation is performed, then when generating the panoramic image of the current panoramic task, the preprocessed image corresponding to all the original captured images is used as the image of the current panoramic task for the back-end compositing operation; otherwise, all the original captured images of the current panoramic task are used as the image of the current panoramic task for the back-end compositing operation.

[0171] The following describes the implementation details of the backend compositing part:

[0172] Figure 7 shows a flowchart of a method for generating a panoramic image for a current panoramic task according to an embodiment of the present disclosure. As shown in Figure 7, when generating the panoramic image for the current panoramic task based on the image of the current panoramic task and the second temporary data, the following steps S141 to S145 are included:

[0173] In step S141, the camera intrinsic parameter matrix corresponding to the image of the current panoramic task is obtained; the camera intrinsic parameter matrix includes: the calibrated camera intrinsic parameter matrix or the globally optimized camera intrinsic parameter matrix.

[0174] When generating the panoramic image for the current panoramic task, if a preprocessed image corresponding to the original captured image is used, the camera intrinsic matrix is ​​the camera intrinsic matrix of the preprocessed image; if all original captured images are used, the camera intrinsic matrix is ​​the camera intrinsic matrix of the original captured images.

[0175] The camera intrinsic matrix is ​​a matrix that describes the camera's internal parameters, which determine how the camera maps points in the three-dimensional world onto the two-dimensional image plane. These parameters typically include focal length (fx, fy), optical center (cx, cy), and possible lens distortion parameters.

[0176] The camera intrinsic parameter matrix in this disclosure refers to the camera intrinsic parameter matrix obtained through the camera calibration process. Camera calibration is a process of estimating camera intrinsic parameters using known points in the physical world (usually a checkerboard or dot array, etc.). By capturing images of these known points and using a specific algorithm (such as the Zhang Zhengyou calibration method), the camera's intrinsic parameter matrix can be calculated.

[0177] The globally optimized camera intrinsic parameter matrix can be obtained in the following way:

[0178] First, obtain the camera intrinsic parameter matrix to be optimized corresponding to the image of the current panoramic task. The camera intrinsic parameter matrix to be optimized refers to the camera intrinsic parameter matrix that needs further adjustment or optimization. By globally optimizing these intrinsic parameters, the accuracy and quality of the processing results can be significantly improved.

[0179] Then, based on the feature point matching data, the camera intrinsic parameter matrix to be optimized is globally optimized to generate the globally optimized camera intrinsic parameter matrix. Specifically, a reprojection error function is constructed using the feature point matching data. The reprojection error measures the difference between the projected position of a feature point in 3D space onto the image plane through camera intrinsic and extrinsic parameters (rotation matrix and translation vector) and the actual detected feature point position. The camera intrinsic parameter matrix to be optimized is used as the optimization variable, and the goal is to minimize the sum of the reprojection errors of all feature points. Then, an appropriate optimization algorithm, such as the Levenberg-Marquardt algorithm, gradient descent, or its variants, is selected to solve this nonlinear least squares problem, and the initial values, number of iterations, convergence conditions, and other parameters of the optimization algorithm are set. Finally, the camera intrinsic parameter matrix to be optimized and the optimization problem are input into the selected optimization algorithm. The optimization process is iterated until the convergence condition is met or the maximum number of iterations is reached, until the globally optimized camera intrinsic parameter matrix is ​​output.

[0180] After obtaining the globally optimized camera intrinsic matrix, the feature point matching data of the current panoramic task, the rotation matrix, and the globally optimized camera intrinsic matrix can be used as the second temporary data and placed in the second memory for use by the backend synthesis part. In this way, when obtaining the camera intrinsic matrix corresponding to the image of the current panoramic task, the camera intrinsic matrix corresponding to the image of the current panoramic task can be obtained based on the second temporary data.

[0181] In step S142, quadrilaterals of each image region corresponding to each image of the current panoramic task are obtained based on the camera intrinsic parameter matrix, the rotation matrix of all original captured images after global optimization, and the vertex coordinates of the current panoramic task image.

[0182] Specifically, when remapping the pixels in the original image to the output image, the calculation can be performed using the following formula (2): P2 i =K*R i *K -1 *P1 i (2)

[0183] [Corrected according to Rule 91, 13.01.2026] Wherein, K is the camera intrinsic parameter matrix, R i P2 is the rotation matrix of the i-th original captured image after global optimization. i P1 represents the vertex coordinates of the quadrilateral of the remapped image region corresponding to the i-th image of the current panoramic task. i Let be the vertex coordinates of the i-th image of the current panoramic task.

[0184] In step S143, the remapping image regions corresponding to each image of the current panoramic task are determined based on the quadrilaterals of each image region.

[0185] After determining the remapped image region, the discard region is also indirectly determined. Taking the synthesis of a panoramic image from a planar model as an example, the discard region is determined by the fact that due to the increased viewing angle of the synthesized image, the four corners will undergo significant deformation after remapping. When the synthesized image is cropped with a rectangle, the excess parts at the four corners will be discarded.

[0186] In step S144, a mapping table is constructed between each pixel in the remapped image region and each image of the current panoramic task, based on the camera intrinsic parameter matrix, the rotation matrix of all the original captured images after global optimization, and the pixel coordinates in the remapped image region.

[0187] In image backend compositing, the remapping step is a significant time-consuming process. Especially when using CPU computation, this step can account for more than 50% of the total backend time.

[0188] In the final image synthesis process, this disclosure does not construct a mapping table between each pixel in the output frame and the original image (i.e., the coordinates of the original image pixel corresponding to the pixel in the output frame), but only constructs a mapping table between each pixel in the remapped image region other than the discarded region and each image of the current panoramic task. The mapping relationship of each pixel is determined by the above formula (2).

[0189] In this way, by pre-calculating the remapped image boundaries, this disclosure can reduce the image remapped area, that is, eliminate the mapping calculation of the discarded area, thereby shortening the time required for panoramic synthesis.

[0190] In step S145, a panoramic image for the current panoramic task is generated based on the mapping table.

[0191] Specifically, the process can be performed by following these steps: initializing the panoramic image canvas, traversing the mapping table, assigning and blending pixels, processing boundaries and gaps, and post-processing and optimization.

[0192] Taking the panoramic synthesis of 9 original images arranged in a nine-grid pattern as an example:

[0193] [Correction 13.01.2026 according to Rule 91] Figure 8 shows a schematic diagram of the remapping regions of the original captured images shown in Figure 4 according to an embodiment of the present disclosure. If the original image is remapped according to the above formula (2), the final image obtained is shown in the shaded area in Figure 9. When using rectangular cropping, after cropping off the excess parts on the four sides, a large area will be discarded.

[0194] This disclosure pre-calculates the discard region during remapping, and the specific method for determining the boundary region is as follows:

[0195] Determine the upper, lower, left, and right boundary lines of the boundary region. Taking an arrangement of m rows and n columns of images as an example: the upper boundary line is the upper edge of the bounding rectangle of the image in row 1, column j, where j is the column containing the optimized anchor image; the lower boundary line is the lower edge of the bounding rectangle of the image in row m, column j; the left boundary line is the left edge of the bounding rectangle of the image in row k, column 1, where k is the row containing the optimized anchor image; the right boundary line is the right edge of the bounding rectangle of the image in row k, column n.

[0196] For example, Figure 10 illustrates a method for determining boundary regions when synthesizing panoramic images according to an embodiment of the present disclosure. As shown in Figure 10, if the optimized anchor point image is selected as the first image, then the top, bottom, left, and right boundaries of the image are the top, bottom, left, and right boundaries of images 2, 6, 8, and 4, respectively. The boundary regions are determined by the values ​​of the remapping region edges of images 2, 6, 8, and 4, namely d2, d6, d8, and d4. Specifically, the values ​​of d2, d6, d8, and d4 can be determined using the bounding rectangle of each remapping region.

[0197] [Corrected according to Rule 91, 13.01.2026] With this optimization scheme, the mapping table can be reduced to the black box area of ​​the remapping region shown in Figure 11, thus eliminating the mapping calculation of the remaining areas.

[0198] In this example, if the camera has a horizontal field of view of 75°, a vertical field of view of 60°, and each image is spaced 25° apart in the yaw direction and 20° apart in the pitch direction, then approximately 40% of the pixel mapping calculation can be reduced. This means that the total computation time for remapping transformation can be reduced by about 40%, which greatly shortens the panoramic image synthesis time, improves synthesis efficiency, and thus enhances the user experience.

[0199] According to an embodiment of this disclosure, before generating the panoramic image of the current panoramic task based on the image of the current panoramic task and the second temporary data, the third thread detects whether the image of the current panoramic task and the second temporary data match. If they do not match, the generation of the panoramic image of the current panoramic task is terminated.

[0200] When detecting whether the image of the current panoramic task matches the second temporary data, it is possible to check whether the rotation matrix, the number of matching images, and the number of photos are consistent. For example, if n photos are taken, the number of rotation matrices needed is equal to n, and the number of matching photos is greater than or equal to n. It is also possible to check whether all data types, such as matching data, rotation matrices, and camera intrinsic parameters, are complete. If it is text data, it is also possible to check whether there are garbled characters in the text data, i.e., non-ASCII code values.

[0201] Introducing a matching detection mechanism can optimize the processing flow of panoramic image generation, avoiding unnecessary repetition or erroneous operations in subsequent processing, thereby improving processing efficiency and accuracy.

[0202] The following is a detailed description of the execution details of the third thread:

[0203] In this disclosure, the third thread, as a separate thread, is responsible for the back-end compositing process. Figure 12 shows a flowchart of the operation method of the third thread in the back-end compositing part of the panoramic image generation task according to an embodiment of this disclosure. As shown in Figure 12, the process includes the following steps S151 to S154:

[0204] In step S151, it is detected whether the second temporary data exists in the second memory. If it exists, then step S152 is executed.

[0205] In step S152, the second temporary data and the image of the current panoramic task are acquired, and it is detected whether the image of the current panoramic task and the second temporary data match. If they match, then step S153 is executed; if they do not match, then the generation of the panoramic image of the current panoramic task is terminated.

[0206] Here, the second temporary data may be recovered from the second temporary data file by a second thread after the system restarts or after an abnormal interruption is detected and recovered. In some abnormal situations, such as power outages, data may not be saved in time and may be lost. Therefore, this detection can avoid the situation where synthesis fails due to the use of incorrect synthesis data.

[0207] In step S153, a panoramic image of the current panoramic task is generated based on the image of the current panoramic task and the second temporary data.

[0208] In step S154, temporary data related to the current panoramic task is deleted. This includes, for example, temporary images and all text data.

[0209] This disclosure discloses a method that separates the three main steps of panoramic image generation—shooting, front-end processing, and back-end compositing—into different threads. This fully utilizes the time-consuming characteristics of each step and minimizes user waiting time. Since the shooting process requires gimbal movement and then stabilizing at a specific angle for taking a picture, shooting tasks often take a considerable amount of time. Parallel processing of front-end processing and photo shooting can reasonably reduce the waiting time of front-end processing. Furthermore, by using a first memory to load front-end processing data (first temporary data), front-end processing is no longer associated with other steps, allowing the front-end processing thread to be placed in the background, further reducing user waiting time. For example, it can continue shooting panoramic photos for the next panoramic task. Moreover, the back-end compositing data (second temporary data) of each panoramic compositing task is placed in a task queue in a second memory, making the number of processing tasks unlimited. Users can continuously shoot panoramic photos for extended periods without waiting.

[0210] After evaluation, it was found that when generating panoramic images, the technical solution disclosed herein can achieve a user experience on a CPU device that is similar to that on a custom device (GPU or other parallel processing unit).

[0211] In addition, in this disclosure, to ensure that inter-thread communication and data processing are timely without wasting system resources due to frequent detection, the specific values ​​of the first preset detection period and the second preset detection period can be determined according to the following two methods:

[0212] Method 1: First, set initial values ​​for the first and second preset detection cycles based on experience or preliminary testing. Then, assess the current system load by acquiring key indicators such as CPU utilization, memory usage, and I / O wait time, while simultaneously monitoring the working status of the first, second, and third threads, including whether they are busy and have pending tasks. Finally, dynamically adjust the specific values ​​of the first and second preset detection cycles based on the current system load and thread status. Specifically: If the system load is below the preset lower threshold, indicating sufficient system resources, the first and second preset detection cycles can be appropriately shortened to improve inter-thread communication efficiency and data processing speed. If the system load is above the preset upper threshold, indicating resource scarcity, the first and second preset detection cycles can be extended to reduce thread switching and resource contention, thus reducing system overhead. If the first thread generates raw images slowly, the first preset detection cycle can be appropriately extended to avoid the second thread frequently checking and wasting resources. If the second thread processes images slowly, while the first thread has already generated a large number of raw images, the first preset detection cycle can be appropriately shortened so that the second thread can process these images promptly. Similarly, the second preset detection cycle is dynamically adjusted based on the state of the third thread and the data in the second memory.

[0213] Method 2: Pre-calculate the execution parameters of all executable panoramic tasks in the shooting, front-end processing, and back-end compositing parts. For example, for the first preset detection period, the execution parameters of each panoramic task in the shooting part can be pre-calculated, such as: the generation rate of the original captured images (number of images generated per second or frame rate), the size and resolution of the original captured images, and historical execution parameters such as delays or intervals during the shooting process. Specifically, by considering the generation rate of the shooting part and the concurrency capability of the processing part, it can be ensured that the second thread can detect the newly generated original captured images in a timely manner. If the shooting rate is high and the processing concurrency capability is strong, the first preset detection period should be set shorter. Conversely, if the shooting rate is low or the processing concurrency capability is limited, the first preset detection period can be slightly longer. For the second preset detection period, the execution parameters of each panoramic task in the front-end processing and back-end compositing parts can be pre-calculated, such as: the time required for preprocessing and feature extraction (average processing time or worst-case processing time), the concurrency capability during processing (i.e., how many images are processed simultaneously), the time required to generate a panoramic image based on the second temporary data, the size and resolution of the panoramic image, and the concurrency capability during the compositing process. Specifically, the average processing time of the front-end processing and the concurrency capability of the back-end synthesis should be considered to ensure that the third thread can detect the second temporary data in a timely manner after the front-end processing is completed. If the front-end processing time is long and the back-end synthesis concurrency capability is limited, the second preset detection period should be set slightly longer to avoid resource waste caused by frequent detection. Conversely, if the front-end processing time is short and the back-end synthesis concurrency capability is strong, the second preset detection period can be slightly shorter.

[0214] In practical applications, a combination of methods 1 and 2 can be used to ensure the best results.

[0215] According to the technical solution provided in this disclosure, efficient panoramic image synthesis is achieved using multiple threads (a first thread and a second thread) and shared memory (a first memory). First, the first thread generates and stores the original captured images and their descriptive information in the first memory and a designated storage space. Next, the second thread periodically checks the descriptive information in the first memory, preprocesses the existing original captured images, generates first temporary data or preprocessed images, and after all original captured images have been processed, integrates these data into second temporary data. Finally, the image data of the current panoramic task and the second temporary data are combined to generate the final panoramic image. This method reduces waiting time by using multiple threads to process the capturing and front-end processing parts in parallel, effectively improving the efficiency and response speed of panoramic image generation, shortening the panoramic image synthesis time, and thus enhancing the user experience.

[0216] Furthermore, in the technical solution provided in this disclosure embodiment, the multithreading also includes a third thread, and the second thread and the third thread share a second memory. By storing the second temporary data generated by the second thread into the second memory, the third thread detects the second temporary data in the second memory at another preset period. Once detected, the panoramic image of the current panoramic task is generated based on the image of the current panoramic task and the second temporary data. That is, the three stages of shooting, front-end processing and back-end compositing are respectively put into independent threads for processing, so that the back-end compositing can be performed in parallel with the shooting and front-end processing, thereby further shortening the total time of panoramic image generation, further improving the efficiency and response speed of panoramic image generation, and enhancing the user experience.

[0217] Figure 13 shows a schematic diagram of a multi-threaded panoramic image generation device according to an embodiment of the present disclosure. The multi-threading includes a first thread and a second thread, which share a first memory. As shown in Figure 13, the device 1300 includes: a first thread running module, configured to generate an original captured image of the current panoramic task through the first thread when generating a panoramic image for the current panoramic task, and store the original captured image in a first designated storage space, and store descriptive information of the original captured image in the first memory; a second thread running module, configured to detect whether the descriptive information of the original captured image exists in the first memory based on a first preset detection period through the second thread; if it exists, then: obtain the original captured image from the first designated storage space according to the descriptive information of the original captured image, and preprocess the original captured image. The system generates first temporary data, or generates a preprocessed image corresponding to the original captured image and the first temporary data; the second thread detects whether all the original captured images of the current panoramic task have been processed. If the processing is complete, then: a second temporary data corresponding to the current panoramic task is generated based on the first temporary data of all the original captured images of the current panoramic task, so as to generate a panoramic image of the current panoramic task based on the image of the current panoramic task and the second temporary data, wherein the image of the current panoramic task includes: all the original captured images of the current panoramic task or the preprocessed image corresponding to all the original captured images of the current panoramic task.

[0218] According to embodiments of this disclosure, the multithreading further includes: a third thread, wherein the second thread and the third thread share a second memory, and the second thread execution module is further configured to:

[0219] The second temporary data is stored in the second memory via the second thread;

[0220] The device further includes: a third-thread execution module, configured as follows:

[0221] The third thread detects whether the second temporary data exists in the second memory based on the second preset detection period. If it exists, then: the panoramic image of the current panoramic task is generated based on the image of the current panoramic task and the second temporary data.

[0222] According to an embodiment of this disclosure, the device further includes a panoramic task switching module, configured to generate a panoramic image for the next panoramic task through the device when all original captured images of the current panoramic task have been generated.

[0223] According to embodiments of this disclosure, the apparatus further includes: an abnormal situation handling module, configured to handle abnormal situations, wherein handling the abnormal situation includes any one or more of the following processes:

[0224] When generating the original captured image for the current panoramic task, the first thread checks whether the generation has failed. If so, the generation of the panoramic image for the current panoramic task is terminated.

[0225] When preprocessing and extracting features from the original captured image, the second thread detects whether the original captured image or the preprocessed image corresponding to the original captured image meets the preset quality. If it does not meet the preset quality, the generation of the panoramic image for the current panoramic task is terminated.

[0226] When generating the second temporary data corresponding to the current panoramic task, the second thread checks whether the second temporary data meets the preset requirements. If it does not meet the requirements, the generation of the panoramic image for the current panoramic task is terminated.

[0227] Before generating the panoramic image of the current panoramic task based on the image of the current panoramic task and the second temporary data, the third thread checks whether the image of the current panoramic task and the second temporary data match. If they do not match, the generation of the panoramic image of the current panoramic task is terminated.

[0228] According to embodiments of this disclosure, the second thread running module is further configured as follows:

[0229] The second thread generates a second temporary data file from the second temporary data in a specified format and stores the second temporary data file in a specified storage space. After the system restarts or after detecting an abnormal interruption and recovering, the second thread checks whether the specified storage space contains the second temporary data file. If it does, the second temporary data is retrieved from the second temporary data file and stored in the second memory.

[0230] This disclosure also provides a chip including the apparatus described in the above apparatus embodiments; or, including a memory and a processor; wherein the memory is used to store one or more computer instructions, wherein the one or more computer instructions are executed by the processor to implement the method described in any of the above method embodiments.

[0231] In one specific embodiment, the chip is a camera chip, and the processor is a CPU.

[0232] This disclosure also provides a panoramic camera, including the apparatus described in the above device embodiments; or, including the chip as described above; or, including a memory and a processor; wherein the memory is used to store one or more computer instructions, wherein the one or more computer instructions are executed by the processor to implement the method described in any of the above method embodiments.

[0233] In one specific embodiment, the processor in the panoramic camera is a CPU.

[0234] This disclosure also provides a computer-readable storage medium, which may be a computer-readable storage medium included in the electronic device or computer system described in the above embodiments; or it may be a standalone computer-readable storage medium not assembled into a device. The computer-readable storage medium stores one or more programs, which are used by one or more processors to perform the methods described in this disclosure.

[0235] This disclosure also provides a computer program product, including a computer program that, when executed by a processor, implements any of the methods described in this disclosure.

[0236] The above description is merely a preferred embodiment of this disclosure and an explanation of the technical principles employed. Those skilled in the art should understand that the scope of the invention involved in this disclosure is not limited to technical solutions formed by specific combinations of the above-described technical features, but should also cover other technical solutions formed by arbitrary combinations of the above-described technical features or their equivalents without departing from the inventive concept. For example, technical solutions formed by substituting the above-described features with (but not limited to) technical features disclosed in this disclosure that have similar functions.

Claims

1. A panoramic image generation method based on multithreading, characterized in that, The multithreading includes: a first thread and a second thread, wherein the first thread and the second thread share a first memory; the method includes: When generating the panoramic image for the current panoramic task, the first thread generates one original image of the current panoramic task and stores the description information of the original image in the first memory. The second thread detects whether the description information of the original captured image exists in the first memory based on the first preset detection period. If it exists, then: the second thread obtains the original captured image based on the description information of the original captured image, and performs preprocessing and feature extraction on the original captured image to generate the first temporary data of the original captured image, or generates a preprocessed image corresponding to the original captured image and the first temporary data of the original captured image. The second thread detects whether all the original images captured in the current panoramic task have been processed. If the processing is complete, then: based on the first temporary data of all the original images captured in the current panoramic task, the second thread generates second temporary data corresponding to the current panoramic task, so as to generate the panoramic image of the current panoramic task based on the image of the current panoramic task and the second temporary data. The image of the current panoramic task includes: all the original images captured in the current panoramic task or the preprocessed image corresponding to all the original images captured in the current panoramic task. When generating panoramic images for multiple panoramic tasks consecutively, after the first thread generates and stores all the original images of the current panoramic task, while the second thread processes the current panoramic task, the first thread continues to generate the original images of the next panoramic task without destroying or recreating the first thread.

2. The method according to claim 1, characterized in that, The multithreading further includes: a third thread, wherein the second thread and the third thread share a second memory, and the step of generating the panoramic image of the current panoramic task based on the image of the current panoramic task and the second temporary data includes: The second temporary data is stored in the second memory via the second thread; The third thread detects whether the second temporary data exists in the second memory based on the second preset detection period. If it exists, then: the panoramic image of the current panoramic task is generated based on the image of the current panoramic task and the second temporary data.

3. The method according to claim 2, characterized in that, The method further includes: handling abnormal situations, wherein handling abnormal situations includes any one or more of the following: When generating the original captured image of the current panoramic task, the first thread checks whether the generation has failed. If so, the generation of the panoramic image of the current panoramic task is terminated. When preprocessing and extracting features from the original captured image, the second thread detects whether the original captured image or the preprocessed image corresponding to the original captured image meets the preset quality. If it does not meet the preset quality, the generation of the panoramic image for the current panoramic task is terminated. When generating the second temporary data corresponding to the current panoramic task, the second thread checks whether the second temporary data meets the preset requirements. If it does not meet the requirements, the generation of the panoramic image of the current panoramic task is terminated. Before generating the panoramic image of the current panoramic task based on the image of the current panoramic task and the second temporary data, the third thread checks whether the image of the current panoramic task and the second temporary data match. If they do not match, the generation of the panoramic image of the current panoramic task is terminated.

4. The method according to claim 2, characterized in that, The method further includes: The second thread generates a second temporary data file from the second temporary data in a specified format, and stores the second temporary data file in a specified storage space. After the system restarts or after detecting and recovering from an abnormal interruption, the second thread checks whether the specified storage space contains the second temporary data file. If it does, the second temporary data is retrieved from the second temporary data file and stored in the second memory.

5. The method according to claim 1, characterized in that, The step of generating second temporary data corresponding to the current panoramic task based on first temporary data of all original captured images of the current panoramic task includes: Based on the first temporary data of all original images captured in the current panoramic task, image feature point matching is performed using a partial matching method to generate feature point matching data for the current panoramic task. Based on the feature point matching data, the image pose of all original images captured in the current panoramic task is globally optimized to generate a rotation matrix of all original images captured globally optimized. The feature point matching data of the current panoramic task and the rotation matrix are used as the second temporary data.

6. The method according to claim 5, characterized in that, The first temporary data based on all original images captured in the current panoramic task uses a partial matching method to perform image feature point matching, including: Obtain the image capture parameters for the current panoramic task, wherein the image capture parameters include: capture sequence number and capture position; each original captured image corresponds to a specified capture position and capture sequence number; A position matching table for all original captured images is generated based on the image capture parameters. The position matching table describes the position matching relationship between any two original captured images. The position matching relationship includes: position matching and position mismatch. Based on the first temporary data of the original captured image with the location matching indicated by the location matching relationship table, image feature point matching is performed on the original captured image with the location matching.

7. The method according to claim 5, characterized in that, The step of globally optimizing the image pose of all original images captured in the current panoramic task based on the feature point matching data, and generating a globally optimized rotation matrix for all original images, includes: Based on the initial rotation angle controlled by the shooting and positioning device, an initial rotation matrix is ​​obtained for the image to be optimized relative to the optimized anchor point image; wherein, the optimized anchor point image is one of the original images selected from all the original images of the current panoramic task according to a preset rule, and the image to be optimized is another original image other than the optimized anchor point image; Based on the feature point matching data, the initial rotation matrix is ​​globally optimized to obtain the globally optimized rotation matrix; Determine the difference between the globally optimized rotation matrix and the initial rotation matrix, and obtain the optimization result based on the difference; if the optimization result is successful, then use the globally optimized rotation matrix and the identity matrix corresponding to the optimized anchor point image as the globally optimized rotation matrix of all original captured images.

8. The method according to claim 7, characterized in that, The step of generating the panoramic image for the current panoramic task based on the image of the current panoramic task and the second temporary data includes: Obtain the camera intrinsic parameter matrix corresponding to the image of the current panoramic task; the camera intrinsic parameter matrix includes: a calibrated camera intrinsic parameter matrix or a globally optimized camera intrinsic parameter matrix; Based on the camera intrinsic parameter matrix, the rotation matrix of all original captured images after global optimization, and the vertex coordinates of the current panoramic task image, the remapped quadrilaterals of each image region corresponding to each image of the current panoramic task are obtained. Based on the quadrilaterals of each image region, the remapping image regions corresponding to each image of the current panoramic task are determined; Based on the camera intrinsic parameter matrix, the rotation matrix of all original captured images after global optimization, and the pixel coordinates in the remapped image region, a mapping table is constructed between each pixel in the remapped image region and each image of the current panoramic task. The panoramic image for the current panoramic task is generated based on the mapping table.

9. The method according to claim 8, characterized in that, When the camera intrinsic parameter matrix is ​​a globally optimized camera intrinsic parameter matrix, the method further includes: Obtain the camera intrinsic parameter matrix to be optimized corresponding to the image of the current panoramic task; Based on the feature point matching data, the camera intrinsic parameter matrix to be optimized is globally optimized to generate the globally optimized camera intrinsic parameter matrix. The step of using the feature point matching data of the current panoramic task and the rotation matrix as the second temporary data includes: The feature point matching data of the current panoramic task, the rotation matrix, and the globally optimized camera intrinsic parameter matrix are used as the second temporary data. The step of obtaining the camera intrinsic parameter matrix corresponding to the image of the current panoramic task includes: Based on the second temporary data, obtain the camera intrinsic parameter matrix corresponding to the image of the current panoramic task.

10. A panoramic image generation device based on multithreading, characterized in that, The multithreading includes: a first thread and a second thread, wherein the first thread and the second thread share a first memory; the device includes: The first thread running module is configured to store the description information of each original image captured by the first thread into the first memory when generating the panoramic image of the current panoramic task. The second thread running module is configured to detect whether the description information of the original captured image exists in the first memory based on a first preset detection period. If it exists, then: the second thread obtains the original captured image based on the description information of the original captured image, and performs preprocessing and feature extraction on the original captured image to generate first temporary data of the original captured image, or generates a preprocessed image corresponding to the original captured image and the first temporary data of the original captured image; the second thread detects whether all the original captured images of the current panoramic task have been processed. If they have been processed, then: the second thread generates second temporary data corresponding to the current panoramic task based on the first temporary data of all the original captured images of the current panoramic task, so as to generate the panoramic image of the current panoramic task based on the image of the current panoramic task and the second temporary data, wherein the image of the current panoramic task includes: all the original captured images of the current panoramic task or the preprocessed image corresponding to all the original captured images of the current panoramic task; When generating panoramic images for multiple panoramic tasks consecutively, after the first thread generates and stores all the original images of the current panoramic task, while the second thread processes the current panoramic task, the first thread running module continues to generate the original images of the next panoramic task through the first thread without destroying or recreating the first thread.

11. A chip, characterized in that, The apparatus includes the device of claim 10; or, it includes a memory and a processor; wherein the memory is used to store one or more computer instructions, wherein the one or more computer instructions are executed by the processor to implement the method of any one of claims 1 to 9.

Images