Data acquisition method, model training method, image stitching method and related device

Abstract

The application provides a data acquisition method, a model training method, an image splicing method and related devices. The target image to be cropped can be acquired. Based on image acquisition attribute data of an image acquisition device, the offset between adjacent images to be spliced in an image sequence used for splicing the target image acquired by the image acquisition device is calculated and determined when the target image is used as a spliced image. The image acquisition attribute data indicates the number of images to be spliced corresponding to a unit size in a spliced image spliced by the image acquisition device. The target image is subjected to image-to-be-spliced cropping processing according to the offset, and an image sequence is obtained. Based on this, the application can automatically generate a large number of image sequences with overlapping parts, and automatically label the offset of the overlapping parts of adjacent images in the image sequence, which is convenient and fast.

Description

Technical Field

[0001] This application relates to the field of image stitching technology, and in particular to a data acquisition method, a model training method, an image stitching method, and related apparatus. Background Technology

[0002] Image stitching algorithms are used to combine two or more images with overlapping portions into a single panoramic image that includes information from each image. During the image stitching process, deep learning can be used to extract the overlapping portions of adjacent images, and then these overlapping portions are merged to obtain a smooth, seamless reconstructed panoramic image.

[0003] In deep learning, data acquisition and annotation are crucial steps. The data consists of image sequences with overlapping portions, and the required annotations are the offsets of the overlapping areas between adjacent images in these sequences. Improving the performance of deep learning necessitates increasing the amount of data. Therefore, how to conveniently and quickly acquire and annotate data is a technical problem that urgently needs to be solved by those skilled in the art. Summary of the Invention

[0004] Based on the above requirements, this application proposes a data acquisition method, a model training method, an image stitching method, and related apparatus. This data acquisition method can conveniently and quickly collect and label the data required for deep learning, thereby improving the effectiveness of deep learning.

[0005] The technical solution proposed in this application is as follows:

[0006] On the one hand, this application provides a data acquisition method, including:

[0007] Obtain the target image to be cropped;

[0008] Based on the image acquisition attribute data of the image acquisition device, the offset between adjacent images to be stitched in the image sequence acquired by the image acquisition device for stitching the target image is calculated when the target image is determined to be the stitched image; wherein, the image acquisition attribute data represents the number of images to be stitched per unit size in the image stitched from the images to be stitched by the image acquisition device.

[0009] The target image is cropped according to the offset to obtain the image sequence to be stitched.

[0010] On the other hand, this application also provides a model training method, including:

[0011] Training data is obtained as follows: Based on the image acquisition attribute data of the image acquisition device, the offset between adjacent images to be stitched in the image sequence acquired by the image acquisition device when the target image to be cropped is determined as the stitched image is to be obtained; wherein, the image acquisition attribute data represents the number of images to be stitched per unit size in the stitched image obtained by the image acquisition device; the target image is cropped according to the offset to obtain the image sequence to be stitched.

[0012] The image sequence is used as a training sample, and the offset between adjacent images to be stitched in the image sequence is used as a training label to train the offset detection model.

[0013] On the other hand, this application also provides an image stitching method, including:

[0014] The image sequence to be stitched is input into a pre-trained offset detection model to obtain the offset between adjacent images in the image sequence to be stitched.

[0015] The image sequence to be stitched is stitched together based on the offset between adjacent images to be stitched together in the image sequence.

[0016] The training samples for the offset detection model are a sequence of sample images, and the training labels are the offsets between adjacent sample images to be stitched in the sample image sequence. The generation process of the sample image sequence includes: calculating, based on the image acquisition attribute data of the image acquisition device, the offsets between adjacent sample images to be stitched in the sample image sequence acquired by the image acquisition device when the target image to be cropped is determined as the stitched image; wherein, the image acquisition attribute data represents the number of images to be stitched per unit size in the stitched image formed by stitching the images acquired by the image acquisition device; and cropping the target image according to the offsets to obtain the sample image sequence.

[0017] On the other hand, this application also provides a data acquisition device, including:

[0018] The acquisition module is used to acquire the target image to be cropped;

[0019] The calculation module is used to calculate, based on the image acquisition attribute data of the image acquisition device, the offset between adjacent images to be stitched in the image sequence acquired by the image acquisition device for stitching the target image when the target image is determined to be the stitched image; wherein, the image acquisition attribute data represents the number of images to be stitched per unit size in the image stitched from the images to be stitched acquired by the image acquisition device.

[0020] The cropping module is used to crop the target image according to the offset to obtain the image sequence to be stitched.

[0021] On the other hand, this application also provides a model training apparatus, comprising:

[0022] The acquisition module is used to acquire training data in the following manner: based on the image acquisition attribute data of the image acquisition device, calculate the offset between adjacent images to be stitched in the image sequence acquired by the image acquisition device when the target image to be cropped is determined as the stitched image; wherein, the image acquisition attribute data represents the number of images to be stitched per unit size in the stitched image formed by stitching the images acquired by the image acquisition device; and perform image cropping processing on the target image according to the offset to obtain the image sequence to be stitched.

[0023] The training module is used to train the offset detection model by using the image sequence as training samples and the offset between adjacent images to be stitched in the image sequence as training labels.

[0024] On the other hand, this application also provides an image stitching device, comprising:

[0025] The input module is used to input the image sequence to be stitched into a pre-trained offset detection model to obtain the offset between adjacent images to be stitched in the image sequence to be stitched.

[0026] The stitching module is used to stitch the image sequence to be stitched together according to the offset between adjacent images to be stitched in the image sequence;

[0027] The training samples for the offset detection model are a sequence of sample images, and the training labels are the offsets between adjacent sample images to be stitched in the sample image sequence. The generation process of the sample image sequence includes: calculating, based on the image acquisition attribute data of the image acquisition device, the offsets between adjacent sample images to be stitched in the sample image sequence acquired by the image acquisition device when the target image to be cropped is determined as the stitched image; wherein, the image acquisition attribute data represents the number of images to be stitched per unit size in the stitched image formed by stitching the images acquired by the image acquisition device; and cropping the target image according to the offsets to obtain the sample image sequence.

[0028] On the other hand, this application also provides an electronic device, including:

[0029] Memory and processor;

[0030] The memory is used to store programs;

[0031] The processor is configured to implement any of the above-described data acquisition methods by running a program in the memory; or, implement any of the above-described model training methods; or, implement any of the above-described image stitching methods.

[0032] On the other hand, this application also provides a storage medium, comprising: a computer program stored on the storage medium, wherein when the computer program is executed by a processor, it implements the data acquisition method described in any one of the above claims; or, implements the model training method described in any one of the above claims; or, implements the image stitching method described in any one of the above claims.

[0033] The data acquisition method proposed in this application can acquire a target image to be cropped. Based on the image acquisition attribute data of the image acquisition device, it calculates the offset between adjacent images to be stitched in the image sequence acquired by the image acquisition device when the target image is used as the stitched image. The image acquisition attribute data represents the number of images to be stitched per unit size in the stitched image obtained by the image acquisition device. The target image is then cropped according to the offset to obtain the image sequence. Based on this, this application can automatically generate a large number of image sequences with overlapping parts and automatically annotate the offset of the overlapping parts of adjacent images in the image sequence, which is convenient and fast. Attached Figure Description

[0034] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of this application. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0035] Figure 1 This is a flowchart illustrating a data acquisition method provided in an embodiment of this application;

[0036] Figure 2 This is a schematic diagram of image stitching provided in an embodiment of this application;

[0037] Figure 3 This is another schematic diagram of image stitching provided in an embodiment of this application;

[0038] Figure 4 This is a schematic diagram of the coordinate system provided in the embodiments of this application;

[0039] Figure 5 This is a schematic diagram of sample content image acquisition provided in an embodiment of this application;

[0040] Figure 6 This is another schematic diagram of sample content image acquisition provided in the embodiments of this application;

[0041] Figure 7 This is the target image provided in the embodiments of this application;

[0042] Figure 8 This is a schematic diagram of the region of interest of the target image provided in the embodiments of this application;

[0043] Figure 9 The image to be stitched is obtained by cropping according to the embodiments of this application;

[0044] Figure 10 This is a flowchart illustrating the model training method provided in the embodiments of this application;

[0045] Figure 11 This is a flowchart illustrating the image stitching method provided in an embodiment of this application;

[0046] Figure 12 This is a schematic diagram of the structure of a data acquisition device provided in an embodiment of this application;

[0047] Figure 13 This is a schematic diagram of the structure of a model training device provided in an embodiment of this application;

[0048] Figure 14 This is a schematic diagram of the structure of an image stitching device provided in an embodiment of this application;

[0049] Figure 15 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0050] Application Overview

[0051] The technical solution of this application is applicable to application scenarios involving the collection and annotation of data for model training. The model is used to extract the overlapping portions of two adjacent images during image stitching. During image stitching, the overlapping portions of two adjacent images can be merged based on the extraction results of this model to obtain a smooth and seamless panoramic image. The data used for model training includes training samples and annotations. The training samples are image sequences with overlapping portions, and the annotations are the offsets of the overlapping portions of adjacent images in these image sequences. Using the technical solution of this application, image sequences with overlapping portions can be automatically generated, and the offsets of the overlapping portions of adjacent images in the image sequences can be automatically annotated, which is convenient and fast.

[0052] Specifically, image stitching algorithms are algorithms that combine two or more images with overlapping portions into a panoramic image containing information from each image. Image stitching algorithms are of great significance in fields such as medical imaging, computer vision, satellite data, and text scanning. During the image stitching process, detection models trained using deep learning can quickly and accurately extract the overlapping portions of adjacent images, facilitating the fusion of these overlapping portions to obtain a smooth and seamless reconstructed panoramic image.

[0053] Taking text scanning as an example, the most common scanning device in this field is the scanning pen, also known as a dictionary pen or translation pen. This type of product generally consists of an AIoT chip, a camera, a speaker, a display screen, and a battery. The AIoT chip typically has several built-in modules, including a splicing module, a detection module, an Optical Character Recognition (OCR) module, a translation module, and a speech synthesis module.

[0054] When using the device, the user aligns the pen tip with the text area on a flat surface such as a book, paper, or product packaging, and presses it down. The camera then starts capturing images in real time. The user slides the pen tip along the text area, and the camera records the target image within the sliding area and sends the real-time captured image to the stitching module. The camera stops capturing images when the user lifts the pen tip.

[0055] The stitching module stitches together numerous raw images, many of which contain overlapping areas between frames, into a single stitched image that includes the content of the user's swiping scan area. Since the user's scan area may contain several lines of text or significant background areas, a detection module is needed to extract a target text line from the stitched image and output it as an image to the OCR module. The OCR module recognizes the text content in the text line image, translates it using the translation module, and then transmits both images to the display module in the upper-layer application for display on the screen. The synthesis module performs speech synthesis on the recognized and translated text content and plays it aloud through a speaker.

[0056] Among these components, the stitching module occupies a fundamental and core position. The stitching effect of the image stitching algorithm implemented in the stitching module directly determines the accuracy of the scanning result. In existing technologies, some stitching modules employ non-deep learning feature processing and image processing algorithms. These algorithms extract features from each frame of the input image sequence and perform similarity searches based on the features of consecutive frames to calculate the offset of identical content between frames. Based on this offset, the consecutive frames are fused to ultimately generate an image containing the scanned area.

[0057] While this type of algorithm has the advantages of simple features and high speed, it also has significant drawbacks. Because the rules for feature extraction are manually defined, the generalization ability of these features is generally limited. In some challenging scenes with complex backgrounds, uneven lighting, or low brightness, splicing errors may occur, requiring the addition of additional rules to reduce such errors. However, these new rules cannot adapt to all situations and are prone to a seesaw effect, meaning that while they may reduce the occurrence of such errors in some challenging scenes, they may cause errors in other scenes where conditions were originally normal.

[0058] With advancements in deep learning and AIoT chip technology, neural networks can be made increasingly smaller, while the computing power of AIoT chips is also increasing. Therefore, integrating deep learning algorithms into the stitching module of AIoT chips is entirely feasible. Using deep learning algorithms to directly extract the offsets of identical content between consecutive frames of an input image sequence is not only fast but also has strong generalization ability. With sufficient training samples, it can correctly extract offsets even in challenging scenes with complex backgrounds, uneven lighting, or low brightness, effectively avoiding stitching errors.

[0059] In deep learning, data acquisition and annotation are crucial steps. Improving the performance of deep learning requires increasing the amount of data. Currently, manual collection of training data is commonly used, but this method is not only costly but also slow. Therefore, this application proposes a data acquisition method, a model training method, an image stitching method, and related apparatus. This technical solution can automatically generate a large number of image sequences with overlapping parts and automatically annotate the offset of the overlapping parts of adjacent images in the image sequence, which is convenient and fast.

[0060] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0061] Exemplary methods

[0062] This application provides a data acquisition method, which can be executed by an electronic device. This electronic device can be any device with data and instruction processing capabilities, such as a computer, a smart terminal, or a server. See also... Figure 1 As shown, the method includes:

[0063] S101. Obtain the target image to be cropped.

[0064] The target image is generally a complete image containing all content. In this embodiment, the target image is cropped according to a certain offset to obtain multiple images to be stitched together, which are then combined to form a sequence of images to be stitched together. The aforementioned offset refers to the offset of the overlapping portion of adjacent images in the sequence. This sequence of images to be stitched together forms samples used to train the model during deep learning, and the offset represents the labeled content of these samples.

[0065] Since the main task of the detection model trained by deep learning in the image stitching process is to extract the offset of the overlapping part of the adjacent images in the input image sequence, in order to train the detection model to extract the offset of the overlapping part of the adjacent images in the input image sequence, when constructing the samples for training the detection model and cropping the target image, the adjacent images to be stitched should have overlapping parts.

[0066] The application scenario of the above detection model can be determined so that the entire image in that application scenario can be selected as the target image. The target image can be downloaded from a database or the network, or it can be obtained by capturing or scanning using a device with shooting or scanning capabilities; this embodiment does not impose any limitations.

[0067] For example, if the above detection model is applied to a text scanning scenario, the image sequence input to the model is a sequence containing text content. To train the model's ability to extract the offset of overlapping parts between adjacent images in the input image sequence, a whole image containing text content can be collected as the target image, and training samples can be extracted from the target image. For instance, whole images related to text scanning application scenarios can be selected from existing OCR training databases, or whole text images related to text scanning application scenarios can be collected from the internet. If the above detection model is specifically applied to scanning pens for primary and secondary school students, then whole images of primary and secondary school textbooks, workbooks, common books, printed papers, etc., can be selected as target images from existing OCR training databases or from the internet. Furthermore, supplementary teaching materials, textbooks, exercise books, papers, etc., can also be scanned or photographed to obtain target images.

[0068] For example, if the above detection model is used in a satellite data processing scenario, the image sequence input to the model would be a sequence containing satellite images. To train the model's ability to extract the offset of overlapping portions of adjacent images in the input image sequence, a complete image containing satellite images can be acquired as the target image, and training samples can be extracted from the target image. For instance, a complete image containing satellite images can be downloaded from a relevant satellite image resource website.

[0069] In this embodiment, after acquiring the target image, it can be cropped to retain only the region of interest containing valid image content. For example, if the detection model is used in a text scanning scenario, blank areas and image regions in the target image can be cropped to obtain a region of interest containing only text. If the detection model is used in a satellite data processing scenario, blank areas in the target image can be cropped to obtain a region of interest containing only satellite imagery.

[0070] S102. Based on the image acquisition attribute data of the image acquisition device, calculate the offset between adjacent images to be stitched in the image sequence acquired by the image acquisition device when the target image is determined as the stitched image.

[0071] The aforementioned image acquisition device is used to acquire the image sequence to be stitched during the image stitching process. In other words, the image acquisition device is used to capture images of the target, obtaining the image sequence to be stitched. The image sequence captured by the image acquisition device is then input into a trained detection model. The trained detection model extracts the offsets between adjacent images in the image sequence, and the image sequence is stitched together based on these offsets to obtain the stitched image.

[0072] It should be noted that the acquisition target mentioned above refers to the object captured by the image acquisition device, and different acquisition targets correspond to different application scenarios. For example, in a text scanning scenario, the acquisition target of the image acquisition device is a line of text; in a medical imaging scenario, the acquisition target of the image acquisition device is a lesion area on the human body, etc.

[0073] Different application scenarios require different image acquisition devices. For example, in the case of text scanning, the image acquisition device is generally a scanning pen; in the case of satellite data processing, the image acquisition device is generally a camera mounted on a satellite; and in the case of medical imaging, the image acquisition device is generally a medical image acquisition device.

[0074] The aforementioned image acquisition attribute data represents the number of images to be stitched per unit size in the image formed by stitching together the images acquired by the image acquisition device. In other words, if the size of the image formed by stitching together Y images acquired by the image acquisition device is X, then the number of images to be stitched per unit size in the image formed by stitching together the images acquired by the image acquisition device is X / Y. The aforementioned size can be area, length, width, or pixels, etc., and is not limited in this embodiment.

[0075] In certain application scenarios, if the image sequence acquired by the image acquisition device only includes one row or one column of images to be stitched together, then it can be determined that the overlapping portion of adjacent images to be stitched together in the image sequence acquired by the image acquisition device is always offset in one direction. In such cases, this embodiment only needs to calculate the image acquisition attribute data corresponding to that offset direction. That is, it calculates the number of images to be stitched together per unit size in that offset direction in the image formed by stitching together the images acquired by the image acquisition device.

[0076] Figure 2 The image shown is a sequence of images acquired by an image acquisition device that contains only one row of images to be stitched together. Figure 2 The shaded area represents the overlapping portion of adjacent images to be stitched together in the image sequence acquired by the image acquisition device. For example... Figure 2 As shown, the overlapping portions of adjacent images to be stitched in the image sequence acquired by the image acquisition device are always offset along direction Z1. If the size of the image to be stitched from the images acquired by the image acquisition device is X1 in this offset direction, and the number of images to be stitched from the image acquisition device is Y1, then the number of images to be stitched per unit size in this offset direction in the image to be stitched from the images acquired by the image acquisition device is X1 / Y1.

[0077] In some application scenarios, if the image acquisition device captures an image sequence containing multiple rows and columns of images to be stitched together, the overlapping portions of adjacent images in the sequence may be offset in multiple directions. In such cases, this embodiment only needs to calculate the image acquisition attribute data corresponding to each offset direction. That is, it calculates the number of images per unit size in each offset direction within the image formed by stitching together the images acquired by the image acquisition device.

[0078] Figure 3 The image shown illustrates a sequence of images acquired by an image acquisition device that includes multiple rows and columns of images to be stitched together. For example... Figure 3 As shown, the overlapping portions of adjacent images to be stitched together in the image sequence acquired by the image acquisition device are offset according to directions Z2 and Z3. The shaded portion H1 represents the overlapping portion offset according to direction Z2, and the shaded portion H2 represents the overlapping portion offset according to direction Z3. If the size of the image to be stitched together by the image acquisition device in direction Z2 is X2, and the number of images to be stitched together by the image acquisition device is Y2, then the number of images to be stitched together per unit size in this offset direction is X2 / Y2. If the size of the image to be stitched together by the image acquisition device in direction Z3 is X3, and the number of images to be stitched together by the image acquisition device is Y3, then the number of images to be stitched together per unit size in this offset direction is X3 / Y3.

[0079] In this embodiment, the image acquisition attribute data of the image acquisition device can be set according to actual conditions. Then, parameters such as the resolution, frame rate, and number of settings of the image acquisition device can be adjusted to ensure that the image acquisition attribute data of the image acquisition device reaches the set data. Furthermore, before the image acquisition device leaves the factory, the production personnel can conduct tests in advance. Through extensive testing and calculations, the image acquisition attribute data of the image acquisition device can be determined. This data is stored in the image acquisition device's memory or labeled in the instruction manual. This allows this embodiment to calculate and determine the offset between adjacent images to be stitched in the image sequence acquired by the image acquisition device when the target image is used as the stitched image. The specific testing process by the production personnel of the image acquisition device is not limited in this embodiment.

[0080] Based on the image acquisition attribute data of the image acquisition device, the method of acquiring images to be stitched by the image acquisition device can be simulated. By calculating and determining the offset of the overlapping part of the adjacent images to be stitched in the image sequence used to stitch the target image when the target image is used as the stitched image, the method of acquiring images to be stitched can be simulated.

[0081] First, based on the number of images per unit size in the image to be stitched from the images to be stitched by the image acquisition device, and the size of the target image, the number of images to be stitched in the image sequence acquired by the image acquisition device for stitching the target image can be determined.

[0082] Specifically, in certain application scenarios, such as Figure 2 In the illustrated embodiment, the overlapping portions of adjacent images to be stitched in the image sequence acquired by the image acquisition device are always offset in one direction. In this case, this embodiment determines the number of images to be stitched in the image sequence acquired by the image acquisition device when using the target image as the stitched image, based solely on the image acquisition attribute data corresponding to that offset direction and the size of the target image in that offset direction.

[0083] In certain application scenarios, such as Figure 3 In the illustrated embodiment, the overlapping portions of adjacent images to be stitched in the image sequence acquired by the image acquisition device are offset in multiple directions. In this case, this embodiment determines the number of images to be stitched in each offset direction of the image sequence acquired by the image acquisition device when the target image is used as the stitched image, based on the image acquisition attribute data corresponding to each offset direction and the size of the target image in each offset direction.

[0084] Then, based on the number of images to be stitched in the image sequence acquired by the image acquisition device when the target image is used as the stitched image, the offset between adjacent images to be stitched in the image sequence acquired by the image acquisition device when the target image is used as the stitched image is further determined.

[0085] If the distance between adjacent acquisition positions remains constant when the image acquisition device acquires images to be stitched together, then the offset is a constant value. For example, in a text scanning scenario, if the scanning pen scans at a uniform speed; or in a satellite data processing scenario or a medical imaging scenario, if the camera is uniformly positioned along the offset direction, and the images to be stitched are uniformly offset along the offset direction, then the offset is a constant value. In such cases, the ratio of the size of the target image along the offset direction to the number of images to be stitched along that offset direction can be calculated, and this ratio can be determined as the offset.

[0086] In typical scenarios such as satellite data processing and medical imaging, cameras are uniformly positioned along the offset direction. The above embodiments accurately simulate how image acquisition devices acquire images to be stitched in these scenarios. Furthermore, by calculating the offset of the overlapping portion of adjacent images in the image sequence used to stitch the target image, the method of obtaining the stitched image can be accurately simulated. Using the target image cropped based on this offset, the resulting stitched image can recreate the image acquired by the image acquisition device in a real-world scenario. Therefore, training the detection model using the stitched image obtained from cropping the target image based on this offset results in a more accurate offset extraction.

[0087] In text scanning scenarios, since the user holds the scanning pen, a constant scanning speed cannot be guaranteed. Therefore, to train a detection model with higher accuracy in offset extraction, the constructed training samples should more closely resemble the image to be stitched obtained during actual scanning. In practical applications, different users have different habits. For example, some users scan slowly at the beginning, gradually increasing their speed to a constant speed, and then gradually slowing down again towards the end; others scan slowly at the beginning, gradually increasing their speed until the end; and so on. This embodiment can simulate this situation by establishing speed change rules to perform variable-speed scanning simulation.

[0088] Specifically, when the target image is used as the stitched image, the image acquisition device acquires an image sequence for stitching the target image. This sequence can be divided into at least one of a uniform speed acquisition portion, an accelerated acquisition portion, and a decelerated acquisition portion. That is, the image sequence may include only a uniform speed acquisition portion, only an accelerated acquisition portion, only a decelerated acquisition portion; or it may include a uniform speed acquisition portion, an accelerated acquisition portion, an accelerated acquisition portion, a decelerated acquisition portion, or a uniform speed acquisition portion, a decelerated acquisition portion; or it may include a uniform speed acquisition portion, an accelerated acquisition portion, and a decelerated acquisition portion. This embodiment does not impose any limitations on these aspects, and the order of the two portions is also not limited in this embodiment when the image sequence includes at least two portions. Furthermore, based on actual test data, a scanning speed or scanning acceleration is set for each portion, and a certain number of images to be stitched are allocated to each portion of the above image sequence according to a set allocation method. Based on the above data, the offset corresponding to each portion can be determined using the conversion formulas for speed, time, and distance.

[0089] The image to be stitched, obtained by cropping the target image based on this offset, can recreate the image captured by the image acquisition device in the real scene. Therefore, training the detection model with the image to be stitched, obtained by cropping the target image based on this offset, will result in a higher accuracy of the trained detection model when extracting the offset.

[0090] S103. Perform image cropping on the target image according to the offset to obtain the image sequence to be stitched.

[0091] When the target image is used as the stitched image, after calculating the offset between adjacent images to be stitched in the image sequence acquired by the image acquisition device for stitching the target image, the target image can be cropped along the offset direction according to the above offset to obtain the image to be stitched, and the above images to be stitched are combined into a sequence of images to be stitched.

[0092] In the above embodiments, a target image to be cropped can be acquired. Based on the image acquisition attribute data of the image acquisition device, the offset between adjacent images to be stitched in the image sequence acquired by the image acquisition device for stitching the target image is calculated and determined when the target image is used as the stitched image. The image acquisition attribute data represents the number of images to be stitched per unit size in the stitched image obtained by stitching the images acquired by the image acquisition device. The target image is then cropped according to the offset to obtain the image sequence. Based on this, this application can automatically generate a large number of image sequences with overlapping parts and automatically annotate the offset of the overlapping parts of adjacent images in the image sequence, which is convenient and fast.

[0093] As an optional implementation, another embodiment of this application discloses that the steps of the above embodiments, based on the image acquisition attribute data of the image acquisition device, calculate the offset between adjacent images to be stitched in the image sequence acquired by the image acquisition device for stitching the target image when determining that the target image is to be used as the stitched image. Specifically, it may include the following steps:

[0094] Calculate the first product of the target image size and the image acquisition attribute data, and determine the number of images to be stitched in the image sequence acquired by the image acquisition device for stitching the target image; determine the offset based on the number of images to be stitched.

[0095] In this embodiment, the number of images to be stitched in the image sequence acquired by the image acquisition device for stitching the target image can be determined based on the size of the target image and the image acquisition attribute data.

[0096] Specifically, the size of the target image along the offset direction and the first product of the image acquisition attribute data along that offset direction can be calculated. The first product is then determined as the number S of images to be stitched in the image sequence acquired by the image acquisition device along that offset direction to obtain the target image. The calculation formula is as follows:

[0097]

[0098] In the above formula, Indicates the size of the target image along the offset direction. This represents the image acquisition attribute data in the offset direction.

[0099] Once the number of images to be stitched is obtained, the offset can be determined based on the number of images to be stitched.

[0100] In the above embodiments, by calculating the product of the size of the target image along the offset direction and the image acquisition attribute data along the offset direction, the number of images to be stitched in the image sequence acquired by the image acquisition device in the offset direction is determined, so as to determine the offset based on the number of images to be stitched in the image sequence.

[0101] As an optional implementation, another embodiment of this application discloses that the steps of the above embodiments, which determine the offset based on the number of images to be stitched, may specifically include the following steps:

[0102] Based on the speed at which the image acquisition device acquires images, the image sequence is divided into at least a first part and / or a second part; the first part includes the images to be stitched obtained when the image acquisition device acquires images at a constant speed, and the second part includes the images to be stitched obtained when the image acquisition device acquires images at a variable speed; the offset corresponding to the first part and / or the second part is determined according to the number of images to be stitched in the image sequence.

[0103] Specifically, as in the above embodiments, when the image acquisition device is in motion, the offset cannot be guaranteed to be a constant value in the actual shooting scenario. For example, in a text scanning scenario, since the user is scanning with a handheld scanning pen, it cannot be guaranteed that the user will always scan at a constant speed. In order to enable the training samples constructed in this embodiment to reproduce the images acquired by the image acquisition device in a real scene, after determining the number of images to be stitched in the image sequence used to stitch the target image, the image sequence can be divided into at least a first part and / or a second part.

[0104] The first part includes the image to be stitched obtained by the image acquisition device when it acquires the image at a constant speed, wherein the offset corresponding to the first part is a constant value; the second part includes the image to be stitched obtained by the image acquisition device when it acquires the image at a variable speed, wherein the variable speed acquisition includes uniform acceleration acquisition and / or uniform deceleration acquisition, wherein the offset corresponding to the second part is a variable value.

[0105] The specific number of images to be stitched in the first and / or second part can be determined by the shooting method of the simulated image acquisition device.

[0106] For example, in text scanning scenarios, most users scan slowly at the beginning, gradually increasing the speed to a constant rate, and then gradually slowing down again towards the end. Therefore, it can be defined that the first 1 / 3 of the images to be stitched together is the uniformly accelerated portion, the middle 1 / 3 is the uniformly accelerated portion, and the last 1 / 3 is the uniformly decelerated portion. Meanwhile, some users also scan slowly at the beginning, then gradually increasing the speed, and finally gradually slowing down again. Therefore, it can be defined that the first 1 / 2 of the images to be stitched together is the uniformly accelerated portion, and the last 1 / 2 is the uniformly decelerated portion. In this embodiment, the image sequence can be split based on multiple shooting methods to facilitate the reconstruction of images captured by the image acquisition device in a real-world scene.

[0107] The offset for the first part is a constant value, which can be determined based on the size of the target image along the offset direction and the number of images to be stitched along that offset direction. The offset for the second part is a variable value. The scanning speed or scanning acceleration can be set for the second part based on actual test data. Using the conversion formula between speed, time, and distance, the offset for each part can be determined.

[0108] In the above embodiments, the speed at which the image acquisition device acquires images can be simulated, and the image sequence is divided into at least a first part obtained when the acquisition is at a constant speed and / or a second part obtained when the acquisition is at a variable speed. With this setting, the image acquired by the image acquisition device in the real scene can be restored. Therefore, the detection model is trained using the image to be stitched obtained by cutting the target image based on the offset. The higher the accuracy of the offset extraction of the trained detection model, the better.

[0109] As an optional implementation, another embodiment of this application discloses that if the image sequence includes a first part, the steps of the above embodiment determine the offset corresponding to the first part based on the number of images to be stitched in the image sequence, which may specifically include the following steps:

[0110] Calculate the first ratio of the size of the target image along the offset direction to the number of images to be stitched in the image sequence, and determine the first ratio as the offset corresponding to the first part.

[0111] If the image sequence includes a first part, and the offset corresponding to the first part is a constant value, then the offset corresponding to the first part can be determined based on the size of the target image along the offset direction and the number of images to be stitched along that offset direction. Specifically, a first ratio can be calculated between the size of the target image along the offset direction and the number of images to be stitched along that offset direction in the image sequence, and this first ratio can be determined as the offset of the first part along the aforementioned offset direction. The calculation formula is as follows:

[0112]

[0113] In the above formula, Indicates the size of the target image along the offset direction. This indicates the number of images to be stitched together in the image sequence along this offset direction.

[0114] Furthermore, it should be noted that in the detection model used to extract offsets, the offset is generally decomposed into offsets along the X-axis and Y-axis. Therefore, a Cartesian coordinate system can be established in the target image, such as... Figure 4 As shown, a rectangular coordinate system can be established with the horizontal side of the target image as the X-axis and the vertical side as the Y-axis before extracting the region of interest. If X is... The component along the X-axis, and the component along the Y-axis. The component along the Y-axis, then the first part corresponds to the offset along the X-axis. The calculation formula is as follows:

[0115]

[0116] The first part corresponds to the edge Axis offset The calculation formula is as follows:

[0117]

[0118] In the above embodiments, the speed at which the image acquisition device acquires images can be simulated, thereby restoring the images acquired by the image acquisition device in real scenes. Therefore, the detection model is trained using the image to be stitched obtained by cutting the target image based on the offset, and the higher the accuracy of the trained detection model when extracting the offset, the better.

[0119] As an optional implementation, another embodiment of this application discloses that if the image sequence includes a second part, the steps of the above embodiment determine the offset corresponding to the second part based on the number of images to be stitched in the image sequence, which may specifically include the following steps:

[0120] The offset of the second part is determined by the acquisition speed of the image acquisition device and the number of images to be stitched in the second part.

[0121] The acquisition speed of the image acquisition device and the number of images to be stitched in the second part can be determined by simulating the acquisition method of the image acquisition device.

[0122] For example, in text scanning scenarios, most users scan slowly at the beginning, gradually increasing the speed to a constant rate, and then gradually slowing down again towards the end. Therefore, it can be stipulated that the first 1 / 3 of the image to be stitched is a uniformly accelerated portion, with the speed increasing from 0 to... The middle third of the image to be stitched together is a uniform speed portion, with a speed of... The last third of the image to be stitched together is a uniformly decelerated section, with the speed decreasing from... Reduced to 0. It can be determined that the acceleration... for:

[0123]

[0124] Uniform acceleration The value of is positive, representing the acceleration of the uniformly decelerated part. The value of is negative.

[0125] During the acceleration process, the first The image to be stitched is relative to the first... The offset of each image to be stitched along the offset direction for:

[0126]

[0127] During deceleration, the first The image to be stitched is relative to the first... The offset of each image to be stitched along the offset direction for:

[0128]

[0129] exist Figure 4 In the coordinate system shown, if for Components on the X-axis for In the Y-axis component, the second acceleration component corresponds to the offset along the X-axis. The calculation formula is as follows:

[0130]

[0131] The second part, the acceleration portion, corresponds to the offset along the Y-axis. The calculation formula is as follows:

[0132]

[0133] The second part, the deceleration section, corresponds to the offset along the X-axis. The calculation formula is as follows:

[0134]

[0135] The second part, the deceleration section, corresponds to the offset along the Y-axis. The calculation formula is as follows:

[0136]

[0137] In the above embodiments, the speed at which the image acquisition device acquires images can be simulated, thereby restoring the images acquired by the image acquisition device in real scenes. Therefore, the detection model is trained using the image to be stitched obtained by cutting the target image based on the offset, and the higher the accuracy of the trained detection model when extracting the offset, the better.

[0138] As an optional implementation, another embodiment of this application discloses that the method in the above embodiments may specifically include the following steps:

[0139] The second ratio of the number of images to be stitched generated when the image acquisition device acquires sample content to the size of the stitched image sequence generated when the image acquisition device acquires sample content is calculated, and the second ratio is determined as the image acquisition attribute data of the image acquisition device.

[0140] Image acquisition attribute data is related to the imaging characteristics of the image acquisition device itself, and can be determined based on these characteristics. In this embodiment, the imaging characteristics of the image acquisition device include: the theoretical size of the complete image when the image acquisition device acquires the sample content, and the number of images to be stitched generated when the image acquisition device captures the sample content. Since the purpose of image stitching is to reconstruct the complete image of the sample content, the size of the aforementioned complete image can be used as the size of the image formed by stitching the images to be stitched together from the images acquired by the image acquisition device. Then, a second ratio is calculated between the number of images to be stitched generated when the image acquisition device captures the sample content and the size of the aforementioned complete image to obtain the image acquisition attribute data.

[0141] Based on this, when the image sequence acquired by the image acquisition device includes only one row of images to be stitched, the ratio of the number of images to be stitched generated when the image acquisition device captures the sample content to the size of the complete image in the offset direction can be calculated to obtain image acquisition attribute data; when the image sequence acquired by the image acquisition device includes multiple rows and columns of images to be stitched, the ratio of the number of images to be stitched generated when the image acquisition device captures the sample content to the size of the complete image in each offset direction can be calculated to obtain image acquisition attribute data.

[0142] In the above embodiments, the image acquisition attribute data of the image acquisition device can be determined by simulating the image acquisition method of the image acquisition device, which is simple and convenient.

[0143] As an optional implementation, another embodiment of this application discloses that the method in the above embodiments may specifically include the following steps:

[0144] Calculate a second product of the sample content size and the resolution of the image acquisition device, and determine the size of the stitched image sequence generated by the image acquisition device when acquiring the sample content. And / or

[0145] Calculate the third product of the duration of sample content acquisition by the image acquisition device and the frame rate of the image acquisition device, and determine the number of images to be stitched when the image acquisition device acquires sample content.

[0146] In this embodiment, the size of the stitched image sequence generated by the image acquisition device when acquiring sample content can be determined based on the size of the sample content and the resolution of the image acquisition device. Specifically, a second product of the size of the sample content and the resolution of the image acquisition device can be calculated, and this second product is used to determine the size of the stitched image sequence generated by the image acquisition device when acquiring sample content. The number of images to be stitched can also be determined based on the duration of sample content acquisition and the frame rate of the image acquisition device. Specifically, a third product of the duration of sample content acquisition and the frame rate of the image acquisition device can be used to determine the number of images to be stitched.

[0147] For example, in a text scanning scenario, the image sequence captured by the scanner's camera typically only includes images such as... Figure 5 The image shown is a row of images to be stitched together. According to people's scanning habits, when a scanning pen scans a line of text, the offset direction of the overlapping part of adjacent images to be stitched together in the resulting image sequence is generally the arrangement direction of the text in the line of text. Figure 5 If the text is arranged in horizontal lines, then the offset direction is... Figure 5The length direction of the horizontal text line is Z4.

[0148] The size of the complete image that the image acquisition device would theoretically capture when acquiring the entire sample content can be determined based on the resolution of the scanner's camera and the size of the text lines. For example, if the scanner's camera resolution is M1 pixels per inch... Figure 5 If the length of the Chinese text line is O1 inches, then theoretically, when the image acquisition device captures a complete image of the sample content, the length of the complete image will be (M1×O1) pixels. The number of images to be stitched together when the image acquisition device captures the sample content can be determined based on the time taken to scan the text line and the frame rate of the scanning pen's camera. For example, if the time taken to scan the text line is T seconds and the scanning pen's camera frame rate is N frames / second, then the number of images to be stitched together when the image acquisition device captures the sample content is (T×N) frames. The calculated image acquisition attribute data is (T×N) / (M1×O1).

[0149] It should be noted that several researchers can be selected and instructed to scan the sample content at a normal speed using a scanning pen. The scanning time should be recorded, and the highest and lowest times should be discarded. The average time taken by each researcher to scan the text line should be recorded as the total time taken to scan that line. Furthermore, if the text length in the text line is in centimeters, the text length can be converted to inches using the conversion relationship 1 inch = 2.54 centimeters before multiplying by the resolution.

[0150] Another example is that, in the context of satellite data processing, the image sequence captured by a camera mounted on a satellite may include, for instance, images such as... Figure 6 The image shown is a multi-row, multi-column sequence of images to be stitched together. In this case, the offset direction of the overlapping portion of adjacent images to be stitched together in the image sequence acquired by a camera mounted on a satellite includes... Figure 4 The directions Z5 and Z6 are in the middle.

[0151] The theoretical size of the complete image of the sample content when captured by the image acquisition device can be determined based on the resolution of the camera mounted on the satellite and the size of the target. For example, if the resolution of the camera mounted on the satellite is M2 pixels / inch and the length of the sample content along direction Z5 is O2 inches, then theoretically, the complete image of the sample content when captured by the image acquisition device is (M2×O2) pixels along direction Z5. If the resolution of the camera mounted on the satellite is M2 pixels / inch and the length of the target along direction Z6 is O3 inches, then theoretically, the complete image of the sample content when captured by the image acquisition device is (M2×O3) pixels along direction Z6.

[0152] The number of images to be stitched together when the image acquisition device captures the sample content can be determined based on the number of cameras mounted on the satellite. For example, if there are F1 cameras mounted along direction Z5 and F2 cameras mounted along direction Z6, then the number of images to be stitched together along direction Z5 when the image acquisition device captures the sample content is F1, and the number of images to be stitched together along direction Z6 is F2. The image acquisition attribute data along direction Z5 is calculated as F1 / (M2×O2), and the image acquisition attribute data along direction Z6 is calculated as F1 / (M2×O3).

[0153] In other scenarios, those skilled in the art can also determine the image acquisition attribute data without expending creativity based on the content described in the above embodiments, which will not be elaborated in detail in this embodiment.

[0154] In the above embodiments, the image size after stitching the image sequence generated when the image acquisition device acquires sample content and the number of images to be stitched generated when the image acquisition device acquires sample content can be quickly determined through simple experiments, which can effectively improve the processing speed of the target image.

[0155] As an optional implementation, another embodiment of this application discloses that the steps of the above embodiments, which involve cropping the target image according to an offset to obtain a sequence of images to be stitched, may specifically include the following steps:

[0156] Starting from the initial position, the image is cropped by a set area at each offset interval to obtain the image sequence to be stitched together; the set area is the area of ​​the image acquired by the image acquisition device.

[0157] When the target image is used as the stitched image, after the image acquisition device acquires the offset between adjacent images to be stitched in the image sequence used to stitch the target image, the target image can be cropped along the offset direction starting from the starting position according to the above offset to obtain the image to be stitched, and the above images to be stitched are combined into a sequence of images to be stitched.

[0158] The cropping size is related to the size of the image captured by the image acquisition device. For example, if the image captured by the image acquisition device is W inches long and H inches wide, then when cropping the target image, the length of the resulting stitched image will also be W inches and the width will also be H inches.

[0159] In addition to cropping the target image along the offset direction according to the aforementioned offset, cropping can also be performed based on offsets along the X-axis and Y-axis. Specifically, if the starting point coordinates are (px, py), a block S1 with a width of W and a height of H centered at this point is cropped; then the starting point is shifted dx pixels along the x-axis and dy pixels along the y-axis, where dx is the offset along the X-axis and dy is the offset along the Y-axis, until it reaches (px+dx, py+dy), where another block S2 with a width of W and a height of H centered at this point is cropped; and so on, each time the cropping center point is moved by dx and dy distances, and a cropping is performed, resulting in a cropped image block Si. The final generated S1, S2...Si is the sequence of images to be stitched together that can be used for training the detection model, and the offset of each frame of the image to be stitched together relative to the previous frame is (dx, dy).

[0160] The starting position is the location where the target image is first cropped. Cropping the target image from the starting position results in the first image to be stitched in the image sequence. Subsequently, at intervals calculated using the above steps, the target image is cropped again to obtain the remaining images to be stitched. In text scanning scenarios, the starting position is typically the beginning position of the text line; in satellite data processing or medical imaging scenarios, the starting position is typically the upper right corner of the target image.

[0161] For example, in a text scanning scenario, the length of the text line is 10, the image captured by the camera of the scanning pen is 5 in length and 3 in width. If the calculated offset is 2, and the cropping starts from the starting position, then 5 images to be stitched together can be obtained to form a sequence of images to be stitched together.

[0162] In another example, in a medical imaging scenario, the length and width of the medical imaging area are 16, and the offset directions include directions Z7 and Z8. If the offset along direction Z7 is 2 and the offset along direction Z8 is 4, cropping starts from the starting position. Along direction Z7, 8 images to be stitched can be obtained, and along direction Z8, 4 images to be stitched can be obtained. After all cropping is completed, a total of 32 images to be stitched are obtained to form the image sequence to be stitched.

[0163] In the above embodiments, cropping the target image according to the offset can automatically generate a large number of image sequences with overlapping parts, and automatically mark the offset of the overlapping parts of adjacent images in the image sequence, which is convenient and fast.

[0164] As an optional implementation, another embodiment of this application discloses that if the image content in the target image is a text line, the method in the above embodiments may specifically include the following steps:

[0165] Determine the text direction of the text line as the offset direction, and determine the starting position of the text line as the starting position.

[0166] Specifically, if the image content in the target image is a text line, then the text direction of the text line is the offset direction, and the starting position of the text line is the starting position.

[0167] The text direction of a line of text can be determined by the following steps:

[0168] Specifically, the Progressive Scale Expansion (PSENet) text line detection model is used to detect text lines across the entire target image. The PSENet algorithm employs a method of gradually expanding the kernel size from small to large scales to obtain the largest and most complete shape of the text. Due to the relatively large geometric spacing between these small kernels, this method can effectively distinguish adjacent text instances and is robust to text of arbitrary shapes. This detection model can detect all text lines in the image and represent the positions of the text lines as contour points.

[0169] Calculate the minimum bounding rectangle of the text line based on the detected outline points.

[0170] Since most text is currently arranged from left to right or from top to bottom, and scanning pens only support scanning from left to right, with a few models supporting scanning from top to bottom, this embodiment only describes text directions from left to right and from top to bottom. Methods for other text directions can be obtained by those skilled in the art without inventiveness based on the content described in this embodiment.

[0171] Calculate the included angle based on the smallest bounding rectangle. Specifically, using the top-left corner of the smallest bounding rectangle as a reference point, the angle between the nearest edge of that point in a clockwise direction and the x-axis is the included angle of the smallest bounding rectangle. If this included angle is greater than 80 degrees and less than or equal to 90 degrees, the text in that line is considered vertical; otherwise, it is considered horizontal text. Figure 4 For example, point M is the top left corner, and its nearest side in the clockwise direction is L1. The angle between L1 and the X-axis is less than 80 degrees, which determines... Figure 4 The text in the text is horizontal.

[0172] Determine the scan center line of the text line. For horizontal text, the scan center line is the center line of a rectangle parallel to the adjacent edge in a clockwise direction from the top left corner, and the starting point of the scan is set as the left starting point of this center line; Figure 4For example, point M is the top-left corner, and its nearest edge in the clockwise direction is L1. Then, the center line L2 of the rectangle parallel to L1 is the scan center line, and the starting point N on the left side of L2 is the starting point of the scan. The position of N is the starting position. For vertical text lines, its scan center line is the center line of the rectangle parallel to the nearest edge in the counterclockwise direction of the top-left corner, and the starting point of the scan is set as the upper starting point of this center point.

[0173] In a specific implementation scenario, if the acquired target image is as follows: Figure 7 As shown, then we can... Figure 7 The target image shown is processed by extracting the region of interest (ROI), resulting in the target image as follows. Figure 8 As shown, the target image is cropped according to the description in the above embodiment, and the resulting partial image to be stitched together is as follows. Figure 9 As shown.

[0174] This application proposes a model training method that can be executed by an electronic device. This electronic device can be any device with data and instruction processing capabilities, such as a computer, a smart terminal, or a server. See also... Figure 10 The method includes:

[0175] S1001. Obtain training data.

[0176] Based on the image acquisition attribute data of the image acquisition device, the offset between adjacent images to be stitched in the image sequence acquired by the image acquisition device when the target image to be cropped is used as the stitched image is calculated; wherein, the image acquisition attribute data represents the number of images to be stitched per unit size in the stitched image formed by stitching the images acquired by the image acquisition device; the target image is cropped according to the offset to obtain the image sequence to be stitched.

[0177] The specific process of obtaining training data can be referred to in the relevant examples of the data acquisition methods above, and will not be repeated in this embodiment.

[0178] S1002. Using the image sequence as training samples and the offset between adjacent images to be stitched in the image sequence as training labels, the offset detection model is trained for offset detection.

[0179] During training, image sequences are input as training samples into the offset detection model to obtain the detection results output by the offset detection model. By comparing the detection results output by the offset detection model with the training labels, the loss value of the offset detection model is calculated. The parameters of the offset detection model are adjusted to reduce the loss value of the offset detection model. The above training process is repeated until the loss value of the offset detection model is less than the set value, and the training is completed.

[0180] The above embodiments can automatically generate a large number of image sequences with overlapping parts, and automatically label the offset of the overlapping parts of adjacent images in the image sequence, which is convenient and fast, and can quickly obtain a large number of labeled samples, thereby improving the training speed and training effect of the model.

[0181] This application proposes an image stitching method, which can be executed by an electronic device. This electronic device can be any device with data and instruction processing capabilities, such as a computer, smart terminal, or server. See also... Figure 11 As shown, the method includes:

[0182] S1101. Input the image sequence to be stitched into the pre-trained offset detection model to obtain the offset between adjacent images to be stitched in the image sequence.

[0183] S1102. Based on the offset between adjacent images to be stitched in the image sequence, perform stitching processing on the image sequence to be stitched.

[0184] The training samples for the offset detection model are a sequence of sample images, and the training labels are the offsets between adjacent sample images to be stitched in the sample image sequence. The process of generating the sample image sequence includes: calculating the offsets between adjacent sample images to be stitched in the sample image sequence obtained by the image acquisition device when the target image to be cropped is determined as the stitched image, based on the image acquisition attribute data of the image acquisition device; wherein, the image acquisition attribute data represents the number of images to be stitched per unit size in the stitched image obtained by the image acquisition device; and cropping the target image according to the offsets to obtain the sample image sequence.

[0185] The specific process of obtaining training data can be referred to in the relevant examples of the data acquisition methods above, and will not be repeated in this embodiment.

[0186] In this embodiment, the image sequence to be stitched, acquired by the image acquisition device, is input into a pre-trained offset detection model to obtain the offset between adjacent images in the image sequence to be stitched, as output by the offset detection model.

[0187] Then, based on the offset between adjacent images to be stitched in the image sequence, the image sequence to be stitched is stitched together. Those skilled in the art can refer to relevant existing technologies for the specific stitching process; this implementation is not limited thereto.

[0188] The above embodiments can quickly detect the offset between adjacent images to be stitched in the image sequence using a pre-trained offset detection model. It is not only fast but also has strong generalization ability. With sufficient training samples, it can also correctly extract the offset in difficult scenes with complex backgrounds, uneven lighting, or low brightness, effectively avoiding stitching errors.

[0189] Exemplary device

[0190] Corresponding to the above data acquisition method, this application also discloses a data acquisition device, see [link to relevant documentation]. Figure 12 As shown, the device includes:

[0191] The acquisition module 100 is used to acquire the target image to be cropped;

[0192] The calculation module 110 is used to calculate, based on the image acquisition attribute data of the image acquisition device, the offset between adjacent images to be stitched in the image sequence acquired by the image acquisition device for stitching the target image when the target image is determined to be the stitched image; wherein, the image acquisition attribute data represents the number of images to be stitched per unit size in the image stitched from the images to be stitched by the image acquisition device.

[0193] The cropping module 120 is used to crop the target image according to the offset to obtain the image sequence to be stitched.

[0194] As an optional implementation, another embodiment of this application discloses that the computing module 110 includes:

[0195] The calculation unit is used to calculate the first product of the size of the target image and the image acquisition attribute data, and to determine the number of images to be stitched in the image sequence acquired by the image acquisition device for stitching together the target image.

[0196] The determination unit is used to determine the offset based on the number of images to be stitched together.

[0197] As an optional implementation, another embodiment of this application discloses that when the determining unit determines the offset based on the number of images to be stitched, it is specifically used to: divide the image sequence into at least a first part and / or a second part based on the speed at which the image acquisition device acquires the images; the first part includes the images to be stitched obtained when the image acquisition device acquires the images at a constant speed, and the second part includes the images to be stitched obtained when the image acquisition device acquires the images at a variable speed; and determine the offset corresponding to the first part and / or the second part based on the number of images to be stitched in the image sequence.

[0198] As an optional implementation, another embodiment of this application discloses that, if the image sequence includes a first part, when the determining unit determines the offset corresponding to the first part based on the number of images to be stitched in the image sequence, it is specifically used for:

[0199] Calculate the first ratio of the size of the target image along the offset direction to the number of images to be stitched in the image sequence, and determine the first ratio as the offset corresponding to the first part.

[0200] As an optional implementation, another embodiment of this application discloses that, if the image sequence includes a second part, when the determining unit determines the offset corresponding to the second part based on the number of images to be stitched in the image sequence, it is specifically used for:

[0201] The offset of the second part is determined by the acquisition speed of the image acquisition device and the number of images to be stitched in the second part.

[0202] As an optional implementation, another embodiment of this application discloses that the data acquisition device further includes:

[0203] The second ratio calculation module is used to calculate the second ratio between the number of images to be stitched generated when the image acquisition device acquires sample content and the size of the image after stitching together the image sequence generated when the image acquisition device acquires sample content. The second ratio is determined as the image acquisition attribute data of the image acquisition device.

[0204] As an optional implementation, another embodiment of this application discloses that the data acquisition device further includes:

[0205] The second product calculation module is used to calculate the second product of the sample content size and the resolution of the image acquisition device, and to determine the second product as the image size after stitching together the image sequences generated when the image acquisition device acquires the sample content; and / or

[0206] The third product calculation module is used to calculate the third product of the duration of the image acquisition device acquiring sample content and the frame rate of the image acquisition device, and to determine the number of images to be stitched when the image acquisition device acquires sample content.

[0207] As an optional implementation, another embodiment of this application discloses that when the cropping module 120 performs image cropping processing on the target image according to the offset to obtain the image sequence to be stitched, it is specifically used for:

[0208] Starting from the initial position, the image is cropped by a set area at each offset interval to obtain the image sequence to be stitched together; the set area is the area of ​​the image acquired by the image acquisition device.

[0209] For details on the specific working functions of each unit of the aforementioned data acquisition device, please refer to the content of the above-described data acquisition method embodiment; these details will not be repeated here.

[0210] As an optional implementation, another embodiment of this application discloses that if the image content in the target image is a text line, the data acquisition device further includes:

[0211] The text determination module is used to determine the text direction of a text line as the offset direction and the text start position of the text line as the start position.

[0212] Corresponding to the above-described model training method, this application also discloses a model training apparatus, see [link to relevant documentation]. Figure 13 As shown, the device includes:

[0213] The acquisition module 200 is used to acquire training data in the following manner: based on the image acquisition attribute data of the image acquisition device, it calculates the offset between adjacent images to be stitched in the image sequence acquired by the image acquisition device when the target image to be cropped is determined as the stitched image; wherein, the image acquisition attribute data represents the number of images to be stitched per unit size in the stitched image formed by stitching the images acquired by the image acquisition device; and performs image cropping processing on the target image according to the offset to obtain the image sequence to be stitched.

[0214] The training module 210 is used to train the offset detection model by using the image sequence as training samples and the offset between adjacent images to be stitched in the image sequence as training labels.

[0215] For details on the specific workings of each unit in the aforementioned model training device, please refer to the above-described model training method embodiment; these details will not be repeated here.

[0216] Corresponding to the image stitching method described above, this application also discloses an image stitching device, see [link to relevant documentation]. Figure 14 As shown, the device includes:

[0217] The input module 300 is used to input the image sequence to be stitched into a pre-trained offset detection model to obtain the offset between adjacent images to be stitched in the image sequence to be stitched.

[0218] The stitching module 310 is used to stitch the image sequence to be stitched according to the offset between adjacent images to be stitched in the image sequence.

[0219] The training samples for the offset detection model are a sequence of sample images, and the training labels are the offsets between adjacent sample images to be stitched in the sample image sequence. The process of generating the sample image sequence includes: calculating the offsets between adjacent sample images to be stitched in the sample image sequence obtained by the image acquisition device when the target image to be cropped is determined as the stitched image, based on the image acquisition attribute data of the image acquisition device; wherein, the image acquisition attribute data represents the number of images to be stitched per unit size in the stitched image obtained by the image acquisition device; and cropping the target image according to the offsets to obtain the sample image sequence.

[0220] For details on the specific working functions of each unit of the aforementioned image stitching device, please refer to the above-described embodiment of the image stitching method; these details will not be repeated here.

[0221] Another embodiment of this application also provides an electronic device, see [link to relevant documentation] Figure 15 As shown, the electronic device includes: a memory 400 and a processor 410;

[0222] The memory 400 is connected to the processor 410 and is used to store programs;

[0223] The processor 410 is configured to implement the data acquisition method disclosed in any of the above embodiments, or the model training method disclosed in any of the above embodiments, or the method disclosed in any of the above embodiments by running the program stored in the memory 400.

[0224] Specifically, the aforementioned electronic device may also include: a bus, a communication interface 420, an input device 430, and an output device 440.

[0225] The processor 410, memory 400, communication interface 420, input device 430, and output device 440 are interconnected via a bus. Among them:

[0226] A bus can include a pathway for transmitting information between various components of a computer system.

[0227] The processor 410 can be a general-purpose processor, such as a general-purpose central processing unit (CPU), a microprocessor, etc., or an application-specific integrated circuit (ASIC), or one or more integrated circuits used to control the execution of the program of the present application. It can also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), an off-the-shelf programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0228] Processor 410 may include a main processor, as well as a baseband chip, modem, etc.

[0229] The memory 400 stores a program for executing the technical solution of this application, and may also store an operating system and other critical business functions. Specifically, the program may include program code, which includes computer operation instructions. More specifically, the memory 400 may include read-only memory (ROM), other types of static storage devices capable of storing static information and instructions, random access memory (RAM), other types of dynamic storage devices capable of storing information and instructions, disk storage, flash memory, etc.

[0230] Input device 430 may include a device for receiving user input data and information, such as a keyboard, mouse, camera, scanner, light pen, voice input device, touch screen, pedometer, or gravity sensor.

[0231] Output device 440 may include devices that allow information to be output to a user, such as a display screen, printer, speaker, etc.

[0232] The communication interface 420 may include a device that uses any transceiver to communicate with other devices or communication networks, such as Ethernet, Radio Access Network (RAN), Wireless Local Area Network (WLAN), etc.

[0233] The processor 410 executes the program stored in the memory 400 and calls other devices, which can be used to implement the various steps of the data acquisition method provided in the above embodiments of this application, or to implement the various steps of the model training method provided in the above embodiments of this application, or to implement the various steps of the image stitching method provided in the above embodiments of this application.

[0234] In addition to the methods and devices described above, embodiments of this application may also be computer program products, which include computer program instructions. When executed by processor 410, the computer program instructions cause processor 410 to perform various steps of the data acquisition method provided in the above embodiments, or to perform various steps of the model training method provided in the above embodiments, or to perform various steps of the image stitching method provided in the above embodiments.

[0235] Computer program products can be written in any combination of one or more programming languages ​​to perform the operations of the embodiments of this application. The programming languages ​​include object-oriented programming languages ​​such as Java and C++, as well as conventional procedural programming languages ​​such as C or similar languages. The program code can be executed entirely on the user's computing device, partially on the user's computing device, as a standalone software package, partially on the user's computing device and partially on a remote computing device, or entirely on a remote computing device or server.

[0236] Furthermore, embodiments of this application may also be computer-readable storage media storing computer program instructions. When executed by a processor, the computer program instructions cause the processor 410 to perform various steps of the data acquisition method provided in the above embodiments, or to perform various steps of the model training method provided in the above embodiments, or to perform various steps of the image stitching method provided in the above embodiments.

[0237] The computer-readable storage medium may be any combination of one or more readable media. A readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may be, for example, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of readable storage media (a non-exhaustive list) include: an electrical connection having one or more wires, a portable disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0238] For the foregoing method embodiments, in order to simplify the description, they are all described as a series of actions. However, those skilled in the art should understand that this application is not limited to the described order of actions, because according to this application, some steps can be performed in other orders or simultaneously. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily essential to this application.

[0239] It should be noted that the various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For apparatus embodiments, since they are basically similar to method embodiments, the description is relatively simple; relevant parts can be referred to the descriptions in the method embodiments.

[0240] The steps in the methods of the various embodiments of this application can be adjusted, merged, or deleted in order according to actual needs, and the technical features described in each embodiment can be replaced or combined.

[0241] The modules and sub-modules in the apparatus and terminal in the various embodiments of this application can be merged, divided, and deleted according to actual needs.

[0242] It should be understood that the disclosed terminals, devices, and methods can be implemented in other ways, given the several embodiments provided in this application. For example, the terminal embodiments described above are merely illustrative. For instance, the division of modules or sub-modules is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple sub-modules or modules may be combined or integrated into another module, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be indirect coupling or communication connection through some interfaces, devices, or modules, and may be electrical, mechanical, or other forms.

[0243] The modules or submodules described as separate components may or may not be physically separate. The components that constitute a module or submodule may or may not be physical modules or submodules; that is, they may be located in one place or distributed across multiple network modules or submodules. Some or all of the modules or submodules can be selected to achieve the purpose of this embodiment's solution, depending on actual needs.

[0244] Furthermore, the functional modules or sub-modules in the various embodiments of this application can be integrated into one processing module, or each module or sub-module can exist physically separately, or two or more modules or sub-modules can be integrated into one module. The integrated modules or sub-modules described above can be implemented in hardware or in the form of software functional modules or sub-modules.

[0245] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0246] The steps of the methods or algorithms described in conjunction with the embodiments disclosed herein can be implemented directly by hardware, a software unit executed by a processor, or a combination of both. The software unit can be located in random access memory (RAM), main memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disk, removable disk, CD-ROM, or any other form of storage medium known in the art.

[0247] Finally, it should be noted that in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0248] The above description of the disclosed embodiments enables those skilled in the art to make or use this application. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of this application. Therefore, this application is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A data acquisition method, characterized in that, include: Obtain the target image to be cropped; Based on the image acquisition attribute data of the image acquisition device, the offset between adjacent images to be stitched in the image sequence acquired by the image acquisition device for stitching the target image is calculated when the target image is determined to be the stitched image; wherein, the image acquisition attribute data represents the number of images to be stitched per unit size in the image stitched from the images to be stitched by the image acquisition device. The target image is cropped according to the offset to obtain the image sequence to be stitched. Specifically, when calculating and determining the target image as the stitched image based on the image acquisition attribute data of the image acquisition device, the offset between adjacent images to be stitched in the image sequence acquired by the image acquisition device for stitching the target image includes: Calculate the first product of the size of the target image and the image acquisition attribute data, and determine the first product as the number of images to be stitched in the image sequence acquired by the image acquisition device for stitching together the target image; The offset is determined based on the number of images to be stitched together.

2. The method according to claim 1, characterized in that, The offset is determined based on the number of images to be stitched, including: Based on the speed at which the image acquisition device acquires images, the image sequence is divided into at least a first part and / or a second part; the first part includes the image to be stitched obtained when the image acquisition device acquires images at a constant speed, and the second part includes the image to be stitched obtained when the image acquisition device acquires images at a variable speed. The offset corresponding to the first part and / or the second part is determined based on the number of images to be stitched in the image sequence.

3. The method according to claim 2, characterized in that, If the image sequence includes a first part, then the offset corresponding to the first part is determined based on the number of images to be stitched in the image sequence, including: Calculate a first ratio between the size of the target image along the offset direction and the number of images to be stitched in the image sequence, and determine the first ratio as the offset corresponding to the first part.

4. The method according to claim 2, characterized in that, If the image sequence includes a second part, then the offset corresponding to the second part is determined based on the number of images to be stitched in the image sequence, including: The offset corresponding to the second part is determined by the acquisition speed of the image acquisition device and the number of images to be stitched in the second part.

5. The method according to claim 1, characterized in that, Also includes: The number of images to be stitched generated when the image acquisition device acquires sample content is calculated, and the second ratio is calculated to the size of the image after stitching together the image sequence generated when the image acquisition device acquires the sample content. The second ratio is determined as the image acquisition attribute data of the image acquisition device.

6. The method according to claim 5, characterized in that, Also includes: Calculate a second product of the size of the sample content and the resolution of the image acquisition device, and determine the second product as the size of the image sequence generated by the image acquisition device when acquiring the sample content; and / or Calculate the third product of the duration of the image acquisition device acquiring the sample content and the frame rate of the image acquisition device, and determine the third product as the number of images to be stitched generated when the image acquisition device acquires the sample content.

7. The method according to claim 1, characterized in that, The target image is cropped according to the offset to obtain a sequence of images to be stitched, including: Starting from the initial position, the image is cropped by a predetermined area at intervals corresponding to the offset direction to obtain the image sequence to be stitched together; the predetermined area is the area of ​​the image acquired by the image acquisition device.

8. The method according to claim 7, characterized in that, If the image content in the target image is a text line, the method further includes: The text direction of the text line is determined as the offset direction, and the text starting position of the text line is determined as the starting position.

9. A model training method, characterized in that, include: Training data is obtained as follows: Based on the image acquisition attribute data of the image acquisition device, the offset between adjacent images to be stitched in the image sequence acquired by the image acquisition device when the target image to be cropped is determined as the stitched image is to be obtained; wherein, the image acquisition attribute data represents the number of images to be stitched per unit size in the stitched image obtained by the image acquisition device; the target image is cropped according to the offset to obtain the image sequence to be stitched. The image sequence is used as a training sample, and the offset between adjacent images to be stitched in the image sequence is used as a training label to train the offset detection model. Specifically, when calculating and determining the target image to be cropped as the stitched image based on the image acquisition attribute data of the image acquisition device, the offset between adjacent images to be stitched in the image sequence acquired by the image acquisition device for stitching the target image includes: Calculate the first product of the size of the target image and the image acquisition attribute data, and determine the first product as the number of images to be stitched in the image sequence acquired by the image acquisition device for stitching together the target image; The offset is determined based on the number of images to be stitched together.

10. An image stitching method, characterized in that, include: The image sequence to be stitched is input into a pre-trained offset detection model to obtain the offset between adjacent images to be stitched in the image sequence to be stitched. The image sequence to be stitched is stitched together based on the offset between adjacent images to be stitched together in the image sequence. The training samples for the offset detection model are a sequence of sample images, and the training labels are the offsets between adjacent sample images to be stitched in the sample image sequence. The generation process of the sample image sequence includes: calculating, based on the image acquisition attribute data of the image acquisition device, the offsets between adjacent sample images to be stitched in the sample image sequence acquired by the image acquisition device when the target image to be cropped is determined as the stitched image; wherein, the image acquisition attribute data represents the number of images to be stitched per unit size in the stitched image formed by stitching images acquired by the image acquisition device; and cropping the target image according to the offsets to obtain the sample image sequence. Specifically, when calculating and determining the target image to be cropped as the stitched image based on the image acquisition attribute data of the image acquisition device, the offset between adjacent sample images to be stitched in the sample image sequence acquired by the image acquisition device for stitching the target image includes: Calculate the first product of the size of the target image and the image acquisition attribute data, and determine the first product as the number of images to be stitched in the image sequence acquired by the image acquisition device for stitching together the target image; The offset is determined based on the number of images to be stitched together.

11. A data acquisition device, characterized in that, include: The acquisition module is used to acquire the target image to be cropped; The calculation module is used to calculate, based on the image acquisition attribute data of the image acquisition device, the offset between adjacent images to be stitched in the image sequence acquired by the image acquisition device for stitching the target image when the target image is determined to be the stitched image; wherein, the image acquisition attribute data represents the number of images to be stitched per unit size in the image stitched from the images to be stitched acquired by the image acquisition device. The cropping module is used to crop the target image according to the offset to obtain the image sequence to be stitched. Specifically, when calculating and determining the target image as the stitched image based on the image acquisition attribute data of the image acquisition device, the offset between adjacent images to be stitched in the image sequence acquired by the image acquisition device for stitching the target image includes: Calculate the first product of the size of the target image and the image acquisition attribute data, and determine the first product as the number of images to be stitched in the image sequence acquired by the image acquisition device for stitching together the target image; The offset is determined based on the number of images to be stitched together.

12. A model training device, characterized in that, include: The acquisition module is used to acquire training data in the following manner: based on the image acquisition attribute data of the image acquisition device, calculate the offset between adjacent images to be stitched in the image sequence acquired by the image acquisition device when the target image to be cropped is determined as the stitched image; wherein, the image acquisition attribute data represents the number of images to be stitched per unit size in the stitched image formed by stitching the images acquired by the image acquisition device; and perform image cropping processing on the target image according to the offset to obtain the image sequence to be stitched. The training module is used to use the image sequence as training samples and the offset between adjacent images to be stitched in the image sequence as training labels to train the offset detection model. Specifically, when calculating and determining the target image to be cropped as the stitched image based on the image acquisition attribute data of the image acquisition device, the offset between adjacent images to be stitched in the image sequence acquired by the image acquisition device for stitching the target image includes: Calculate the first product of the size of the target image and the image acquisition attribute data, and determine the first product as the number of images to be stitched in the image sequence acquired by the image acquisition device for stitching together the target image; The offset is determined based on the number of images to be stitched together.

13. An image stitching device, characterized in that, include: The input module is used to input the image sequence to be stitched into a pre-trained offset detection model to obtain the offset between adjacent images to be stitched in the image sequence to be stitched. The stitching module is used to stitch the image sequence to be stitched together according to the offset between adjacent images to be stitched in the image sequence; The training samples for the offset detection model are a sequence of sample images, and the training labels are the offsets between adjacent sample images to be stitched in the sample image sequence. The generation process of the sample image sequence includes: calculating, based on the image acquisition attribute data of the image acquisition device, the offsets between adjacent sample images to be stitched in the sample image sequence acquired by the image acquisition device when the target image to be cropped is determined as the stitched image; wherein, the image acquisition attribute data represents the number of images to be stitched per unit size in the stitched image formed by stitching images acquired by the image acquisition device; and cropping the target image according to the offsets to obtain the sample image sequence. Specifically, when calculating and determining the target image to be cropped as the stitched image based on the image acquisition attribute data of the image acquisition device, the offset between adjacent sample images to be stitched in the sample image sequence acquired by the image acquisition device for stitching the target image includes: Calculate the first product of the size of the target image and the image acquisition attribute data, and determine the first product as the number of images to be stitched in the image sequence acquired by the image acquisition device for stitching together the target image; The offset is determined based on the number of images to be stitched together.

14. An electronic device, characterized in that, include: Memory and processor; The memory is used to store programs; The processor is configured to implement the data acquisition method as described in any one of claims 1 to 8 by running a program in the memory; or to implement the model training method as described in claim 9; or to implement the image stitching method as described in claim 10.

15. A storage medium, characterized in that, include: The storage medium stores a computer program, which, when executed by a processor, implements the data acquisition method as described in any one of claims 1 to 8; or, implements the model training method as described in claim 9; or, implements the image stitching method as described in claim 10.

Images