An automatic jigsaw puzzle method, apparatus, server, and storage medium

Abstract

This disclosure relates to an automatic image jigsaw puzzle method, apparatus, server, and storage medium. The automatic image jigsaw puzzle method includes: acquiring a first sequence of segmented images; inputting the first sequence of segmented images into a prediction model, and outputting the position of each segmented image in the first sequence of segmented images through the prediction model, wherein the prediction model is trained based on the correspondence between the positions of segmented image samples and the positions of the segmented image samples; and jigsaw the segmented image sequence according to the positions to obtain a jigsaw puzzle image. Embodiments of this disclosure can automatically predict the position of each segmented image from shuffled image segments and automatically jigsaw puzzles them. The predicted features can more comprehensively represent the segmented images, providing support for subsequent transfer learning to other task models.

Description

Technical Field

[0001] This disclosure relates to the field of artificial intelligence technology, and in particular to an automatic jigsaw puzzle method, apparatus, server, and storage medium. Background Technology

[0002] With the development of artificial intelligence technology, pre-trained models have emerged in the image processing field. Pre-trained models utilize machine learning training methods to obtain task-independent models from large-scale data. These pre-trained models are then fine-tuned to suit specific tasks, resulting in models with multi-task capabilities. Therefore, pre-training reduces the difficulty of model training and improves its accuracy.

[0003] In related technologies, supervised training methods are used to obtain pre-trained models for the image domain. However, supervised training relies on manually labeled data, which is costly. Most supervised pre-trained models are classification and detection models. Classification and detection tasks are relatively simple, and the image features obtained cannot represent the internal features of the image. Therefore, the accuracy of other task models obtained by fine-tuning the pre-trained model is not high. Summary of the Invention

[0004] This disclosure provides an automatic jigsaw puzzle method, apparatus, server, and storage medium. It aims to at least address the problem of low accuracy in other task models obtained by fine-tuning the pre-trained model in related technologies. The technical solution of this disclosure is as follows:

[0005] According to a first aspect of the present disclosure, an automatic jigsaw puzzle method is provided, comprising:

[0006] Obtain the first block image sequence;

[0007] The first segmented image sequence is input into the prediction model, and the prediction model outputs the position of each segmented image in the first segmented image sequence, wherein the prediction model is set to be trained based on the correspondence between the segmented image samples and the positions of the segmented image samples;

[0008] Based on the stated position, the segmented image sequence is pieced together to obtain a pieced image.

[0009] In one possible implementation, the prediction model includes a self-attention mechanism network, wherein inputting the segmented image sequence into the prediction model, and outputting the position of each segmented image in the segmented image sequence by the prediction model, includes:

[0010] The features of the block images in the first block image sequence and multiple second block image sequences are obtained, wherein the second block images are obtained by randomly sorting the first block image sequence;

[0011] The features of the first segmented image sequence and the features of each of the plurality of second segmented image sequences are respectively input into a self-attention mechanism network. After passing through the self-attention mechanism network, the predicted features are output, wherein the predicted features, the features, and the segmented images have a one-to-one correspondence.

[0012] Obtain the average value of multiple predicted features corresponding to each block image in the first block image sequence;

[0013] The position of each image block is determined based on the average value.

[0014] In one possible implementation, the prediction model is configured to be trained based on the correspondence between the positions of the segmented image samples and the positions of the segmented image samples, including:

[0015] Obtain a sample set, which includes block image samples, and the block image samples are marked with positions;

[0016] Construct a prediction model, wherein training parameters are set in the prediction model;

[0017] The segmented image samples are input into the prediction model, and the prediction model outputs the predicted positions of the segmented image samples.

[0018] Based on the difference between the predicted location and the labeled location, the training parameters are iteratively adjusted until the difference meets the preset requirements, thereby obtaining the prediction model.

[0019] In one possible implementation, the prediction model includes a self-attention mechanism network, wherein inputting the segmented image samples into the prediction model and outputting the predicted positions of the segmented image samples by the prediction model includes:

[0020] Obtain the features of the segmented image samples;

[0021] The features are input into a self-attention mechanism network, and the predicted features are output through the self-attention mechanism network.

[0022] Based on the predicted features, the predicted positions of the segmented images are determined.

[0023] In one possible implementation, obtaining the features of the segmented image samples includes:

[0024] The block image samples are processed by convolutional neural networks to obtain the features of the block image samples.

[0025] In one possible implementation, obtaining the features of the segmented image samples includes:

[0026] The features of the segmented image samples are obtained by performing a linear mapping transformation on the segmented image samples using a mapping matrix.

[0027] In one possible implementation, the location is labeled in the following ways:

[0028] Obtain multiple block image samples located in the same original image and determine a preset position in the original image as the coordinate origin position;

[0029] The plurality of segmented image samples are matched with the original image respectively to obtain the distance between each segmented image sample and the preset position;

[0030] Based on the distance, the position of each segmented image sample in the original image is determined.

[0031] In one possible implementation, the location is labeled in the following ways:

[0032] Obtain multiple block image samples located in the same original image and determine a preset position in the original image as the coordinate origin position;

[0033] The plurality of segmented image samples are matched with the original image respectively to obtain the orientation of each segmented image sample relative to the preset position;

[0034] Based on the orientation, determine the relative position of each block image sample in the plurality of block image samples with respect to the preset position;

[0035] Obtain multiple relative positions corresponding to the multiple segmented image samples, and determine the labeled position of the segmented image at the preset position as the vector representation of the multiple relative positions.

[0036] In one possible implementation, the prediction model includes a self-attention mechanism network, wherein inputting the segmented image samples into the prediction model and outputting the predicted positions of the segmented image samples by the prediction model includes:

[0037] The features of block image samples in a sequence of multiple block image samples are obtained. The sequence of multiple block image samples is obtained by randomly sorting multiple block image samples located in the same original image multiple times.

[0038] The features of the multiple segmented image sample sequences are respectively input into the prediction model, and the prediction model outputs multiple predicted features of the segmented image samples.

[0039] The predicted location of the segmented image sample is determined based on the average value of the multiple predicted features.

[0040] In one possible implementation, the prediction model includes a self-attention mechanism network, wherein inputting the segmented image samples into the prediction model and outputting the predicted positions of the segmented image samples by the prediction model includes:

[0041] The features of block image samples in a sequence of multiple block image samples are obtained. The sequence of multiple block image samples is obtained by randomly sorting multiple block image samples located in the same original image multiple times.

[0042] The features of the multiple segmented image sample sequences are respectively input into the prediction model, and the prediction model outputs multiple predicted features of the segmented image samples.

[0043] Based on the multiple prediction features, multiple prediction positions of the segmented image sample are determined, and the average of the multiple prediction positions is taken as the final prediction position of the segmented image sample.

[0044] According to a second aspect of this disclosure, an automatic jigsaw puzzle device is provided, comprising:

[0045] The acquisition module is used to acquire the first block image sequence;

[0046] The prediction module is used to input the first segmented image sequence into the prediction model, and output the position of each segmented image in the first segmented image sequence through the prediction model, wherein the prediction model is set to be trained based on the correspondence between the position of the segmented image sample and the position of the segmented image sample;

[0047] The jigsaw puzzle module is used to jigsaw the segmented image sequence according to the position to obtain a jigsaw puzzle image.

[0048] According to a third aspect of this disclosure, a task model is provided, which is obtained by training using the correspondence between predictive features and task types as described in any one of the embodiments of this disclosure.

[0049] According to a fourth aspect of this disclosure, an electronic device is provided, comprising:

[0050] processor;

[0051] Memory used to store the processor's executable instructions;

[0052] The processor is configured to execute the instructions to implement the automatic jigsaw puzzle method as described in any of the embodiments of this disclosure.

[0053] According to a fifth aspect of this disclosure, a computer-readable storage medium is provided that, when instructions in the computer-readable storage medium are executed by a processor of an electronic device, enables the electronic device to perform an automatic jigsaw puzzle method as described in any embodiment of this disclosure.

[0054] According to a sixth aspect of this disclosure, a computer program product is provided, the computer program product including instructions that, when executed by a processor of an electronic device, enable the electronic device to perform an automatic jigsaw puzzle method as described in any embodiment of this disclosure.

[0055] The technical solutions provided by the embodiments of this disclosure offer at least the following beneficial effects: The embodiments of this disclosure can automatically predict the position of each segmented image in a shuffled image and automatically stitch them together. Furthermore, the prediction model of this application is set to be trained based on the correspondence between the positions of segmented image samples and the positions of those samples. The positions of the segmented images can be automatically set without manual annotation, thus providing a self-supervised model training method. The model training result can accurately predict the position of each segmented image. Compared to classification and detection models, the model's predictive features can predict the position of each segmented image, which reflects the relationship between the segmented image and its context. Therefore, it can more comprehensively represent the segmented image, providing support for subsequent transfer learning to other task models.

[0056] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this disclosure. Attached Figure Description

[0057] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure, and are not intended to unduly limit this disclosure.

[0058] Figure 1 This is a flowchart illustrating an automatic jigsaw puzzle method according to an exemplary embodiment.

[0059] Figure 2 This is a schematic diagram illustrating a location labeling method according to an exemplary embodiment.

[0060] Figure 3 This is a schematic diagram illustrating a location labeling method according to an exemplary embodiment.

[0061] Figure 4 This is a schematic diagram illustrating a prediction model training method according to an exemplary embodiment.

[0062] Figure 5 This is a block diagram illustrating an automatic jigsaw puzzle device according to an exemplary embodiment.

[0063] Figure 6 This is a block diagram illustrating an electronic device for automatic jigsaw puzzles according to an exemplary embodiment.

[0064] Figure 7 This is a block diagram illustrating an electronic device for automatic jigsaw puzzles according to an exemplary embodiment. Detailed Implementation

[0065] To enable those skilled in the art to better understand the technical solutions of this disclosure, the technical solutions in the embodiments of this disclosure will be clearly and completely described below with reference to the accompanying drawings.

[0066] It should be noted that the terms "first," "second," etc., used in the specification, claims, and accompanying drawings of this disclosure are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this disclosure described herein can be implemented in orders other than those illustrated or described herein. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this disclosure. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this disclosure as detailed in the appended claims.

[0067] It should also be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for display, data used for analysis, etc.) involved in this disclosure are all information and data authorized by the user or fully authorized by all parties.

[0068] Figure 1 This is a flowchart illustrating an automatic jigsaw puzzle method according to an exemplary embodiment. (Reference) Figure 1 As shown, the method is used in a terminal or server and includes the following steps.

[0069] Step S101: Obtain the first segmented image sequence.

[0070] In this embodiment of the disclosure, the first segmented image sequence may include segmented images obtained by cropping the original image, and a sequence of segmented images obtained by randomly sorting the segmented images. The first segmented image sequence may also include any sequence of segmented images from the same image. This disclosure does not limit the order of the segmented image sequence; it may be in the order of the position of the segmented images in the original image, or it may be in a random order.

[0071] Step S102: The first segmented image sequence is input into the prediction model, and the prediction model outputs the position of each segmented image in the first segmented image sequence. The prediction model is set to be trained based on the correspondence between the positions of the segmented image samples and the positions of the segmented image samples.

[0072] In this embodiment of the disclosure, the position of the segmented image may include the absolute position of the segmented image in the original image. In one example, referencing... Figure 2 As shown, the original image 200 is cropped into m×n image blocks. The top-left corner is used as the origin of the coordinate system. Horizontally, the coordinates increase sequentially from left to right; vertically, the coordinates increase sequentially from top to bottom. In another example, the position of each image block is a vector representation of its relative position to other image blocks. (See reference...) Figure 3 As shown, the original image 300 is cropped into 4×4 blocks. Taking the position of block image 301 as the origin of the coordinate system, if the x-coordinate of another block image is to the left of the x-coordinate of block image 301, then the x-coordinate of that other block image is set to 1; if the x-coordinate of another block image is to the right of the x-coordinate of block image 301 or in the same row, then the x-coordinate of that other block image is set to 0; if the y-coordinate of another block image is above the y-coordinate of block image 301, then the y-coordinate of that other block image is set to 1; if the y-coordinate of another block image is below the y-coordinate of block image 301 or in the same column, then the y-coordinate of that other block image is set to 0. For example, the top-left block image, with its x-coordinate to the left of block image 301 and its y-coordinate above block image 301, has a relative position coordinate of (1,1). The predicted position of block image 301 is a vector representation of its relative position to other block images. For example, the predicted position of block image 301 is marked as follows:

[0073] [(1,1)(1,1)(0,1)(0,0)(1,1)(1,1)(0,1)(0,0)(1,0)(1,0)(0,0)(0,0)(0,0)(0,0)(0,0)(0,0)(0,0)] T (1)

[0074] In this embodiment of the disclosure, the prediction model may include various artificial neural network models, such as LSTM (Long Short-Term Memory) networks, Transformer networks (Self-Attention Networks), and improved networks based on LSTM and Transformer networks. The prediction model is trained based on the correspondence between the positions of segmented image samples and the positions of those segmented image samples. This process may include: acquiring a sample set, which includes segmented image samples labeled with positions; constructing a prediction model with training parameters; inputting the segmented image samples into the prediction model, and outputting the predicted positions of the segmented image samples; iteratively adjusting the training parameters based on the difference between the predicted positions and the labeled positions until the difference meets a preset requirement, thereby obtaining the prediction model.

[0075] Step S103: Based on the position, the segmented image sequence is stitched together to obtain a stitched image.

[0076] In one example, when the position is represented by the absolute position mentioned above, the absolute position of the image blocks output by the prediction model can be sorted according to the horizontal coordinate (X-axis) and vertical coordinate (Y-axis) of each image block, thereby sorting the corresponding image blocks and obtaining the mosaic image. In another example, when the position is represented by the vector of the relative position of the other image blocks, the prediction feature dimension of each image block is (m×n, 2), where m represents the image being divided into m rows and n represents the original image being divided into n columns. The prediction feature values ​​are floating-point numbers from 0 to 1. When the prediction feature value is greater than or equal to a preset threshold, the prediction feature value is 1; when the prediction feature value is less than the preset threshold, the prediction feature value is 0. The number of (0,1) is counted as x, and the number of (1,0) is counted as y. The absolute position (x, y) of the image block is obtained from x and y. For example... Figure 3 In the image block 301, the predicted position is indicated by the above formula (1), where there are 2 (1,0) and 2 (0,1) and the absolute position of the image block 301 is indicated by (2,2).

[0077] This disclosure provides an embodiment capable of automatically predicting the position of each segment of a shuffled image and automatically stitching them together. Furthermore, the prediction model is trained based on the correspondence between the positions of segmented image samples and their corresponding positions. The positions of the segmented images can be automatically set without manual annotation, thus representing a self-supervised model training method. The model training result accurately predicts the position of each segmented image. Compared to classification and detection models, the model's predictive features can predict the position of each segmented image, reflecting the relationship between the segmented image and its context. Therefore, it can more comprehensively represent the segmented image, providing support for subsequent transfer learning to other task models.

[0078] In one possible implementation, the prediction model includes a self-attention mechanism network, wherein inputting the segmented image sequence into the prediction model, and outputting the position of each segmented image in the segmented image sequence by the prediction model, includes:

[0079] The features of the block images in the first block image sequence and multiple second block image sequences are obtained, wherein the second block images are obtained by randomly sorting the first block image sequence;

[0080] The features of the first segmented image sequence and the features of each of the plurality of second segmented image sequences are respectively input into a self-attention mechanism network. After passing through the self-attention mechanism network, the predicted features are output, wherein the predicted features, the features, and the segmented images have a one-to-one correspondence.

[0081] Obtain the average value of multiple predicted features corresponding to each block image in the first block image sequence;

[0082] The position of each image block is determined based on the average value.

[0083] In this embodiment of the disclosure, the first segmented image sequence may include a sequence of segmented images to be sorted. In one example, the first segmented image sequence is randomly sorted to obtain multiple second segmented image sequences. The step of inputting the features of the first segmented image sequence and the features of each of the multiple second segmented image sequences into a self-attention mechanism network, and outputting predicted features through the self-attention mechanism network, includes: inputting the features of the first segmented image sequence into the self-attention mechanism network to obtain the predicted features of the first segmented image sequence; and inputting the features of each of the second segmented image sequences into the self-attention mechanism network to obtain the predicted features of the second segmented image sequences. Thus, taking segmented image a as an example, if the number of second segmented image sequences is n-1, and considering that segmented image a appears once in the first segmented image sequence, segmented image a appears a total of n times, and the number of times it is input into the self-attention mechanism network is also n, so the number of predicted features obtained is n. The processing of other segmented images in the first segmented image sequence is the same as that of segmented image a. Therefore, each segmented image obtains n predicted features. In one example, the average of multiple predicted features corresponding to each block image in the first block image sequence is taken. In another example, this average value is mapped to the location space using a fully connected layer to obtain the predicted location of the block image.

[0084] In this embodiment, by randomly sorting the first block image, multiple second block image sequences are obtained. By inputting the first block image sequence and the multiple second block image sequences into a prediction model, multiple prediction features of each block image are obtained. The corresponding position is determined based on the average value of the prediction features, which can improve the accuracy of prediction and make the sorting of the block images more accurate.

[0085] Figure 4 This is a schematic diagram illustrating a prediction model training method according to an exemplary embodiment. (Reference) Figure 4 As shown, the prediction model is trained based on the correspondence between the positions of the segmented image samples and the positions of the segmented image samples, including:

[0086] Obtain a sample set, which includes block image samples, and the block image samples are marked with positions;

[0087] Construct a prediction model, wherein training parameters are set in the prediction model;

[0088] The segmented image samples are input into the prediction model, and the prediction model outputs the predicted positions of the segmented image samples.

[0089] Based on the difference between the predicted location and the labeled location, the training parameters are iteratively adjusted until the difference meets the preset requirements, thereby obtaining the prediction model.

[0090] In this embodiment, the original image 400 is cropped to obtain multiple segmented images. In this embodiment, the number and size of the segmented images are not limited when cropping the original image 400. In one example, multiple original images can be cropped to obtain a sample set. The position of each segmented image can be automatically labeled based on its position within the original image. The position can include both absolute and relative positions as described in the above embodiments.

[0091] In one possible implementation, refer to Figure 4 As shown, the prediction model includes a self-attention mechanism network. The step of inputting the segmented image samples into the prediction model and outputting the predicted positions of the segmented image samples includes: acquiring features of the segmented image samples to obtain input features; inputting the features into the self-attention mechanism network and outputting predicted features; determining the predicted positions of the segmented images based on the predicted features; and iteratively adjusting the training parameters based on the difference between the predicted positions and the labeled positions until the difference meets a preset requirement, thus obtaining the prediction model.

[0092] In this embodiment, the prediction model employs a self-attention mechanism network (Transformer network), a variant of a recurrent neural network initially used for machine translation tasks. The self-attention mechanism network consists of an encoder and a decoder, which are themselves composed of multiple stacked encoding and decoding modules. These modules are connected in different ways through basic layers such as direct connections, multi-head attention, and fully connected layers. Multi-head attention includes multiple self-attention layers. These self-attention layers automatically utilize feature information from previous and subsequent image blocks during image encoding, resulting in output features that better represent the current image block.

[0093] In one possible implementation, obtaining the features of the segmented image samples includes:

[0094] The block image samples are processed by convolutional neural networks to obtain the features of the block image samples.

[0095] In this embodiment of the disclosure, the convolutional neural network may include CNN networks, LeNet networks, AlexNet networks, VggNet networks, ResNet networks, etc. A convolutional neural network can extract a preliminary feature from an image, similar to a filter. After extensive training, the model can automatically adjust the value of the convolution kernel and then convolve it with the image to extract features from the image.

[0096] In this embodiment of the disclosure, a convolutional neural network is used to extract features from block image samples, which can accurately and efficiently obtain the features of image samples.

[0097] In one possible implementation, obtaining the features of the segmented image samples includes: obtaining the features of the segmented image samples includes:

[0098] The features of the segmented image samples are obtained by performing a linear mapping transformation on the segmented image samples using a mapping matrix.

[0099] In this embodiment of the present disclosure, the process of using a mapping matrix to perform linear mapping transformation on block image samples to obtain the features of the block image samples specifically includes: for block image samples of size, for example N×N, firstly flattening the first block image into 1×N 2 The vector is then dimensionality-reduced using a preset mapping matrix, the size of which is, for example, N. 2 ×M, to obtain the 1×M feature vector, and obtain the features of the block image sample.

[0100] In one possible implementation, the location is labeled in the following ways:

[0101] Obtain multiple block image samples located in the same original image and determine a preset position in the original image as the coordinate origin position;

[0102] The plurality of segmented image samples are matched with the original image respectively to obtain the distance between each segmented image sample and the preset position;

[0103] Based on the distance, the position of each segmented image sample in the original image is determined as the labeled position of the segmented image.

[0104] In this embodiment of the disclosure, reference is made to Figure 2The process involves acquiring multiple segmented image samples located within the same original image, for example, all of which originate from image 200. The top-left corner of the original image 200 is used as the origin, denoted as (0, 0). Since the segmented image samples originate from the original image, each segmented image sample can be matched to its corresponding original image position. To the right of the origin, segmented image 201 is 1 unit away from the origin in the horizontal direction; therefore, its coordinates are (0, 1). Segmented image 203 is n-1 units away from the origin in the horizontal direction; therefore, its coordinates are (0, n-1). Segmented image 205 is n-1 units away from the origin in the horizontal direction and m-1 units away in the vertical direction; therefore, its coordinates are (m-1, n-1).

[0105] The location annotation method provided in this disclosure can automatically annotate the locations of segmented images.

[0106] In one possible implementation, the location is labeled in the following ways:

[0107] Obtain multiple block image samples located in the same original image and determine a preset position in the original image as the coordinate origin position;

[0108] The plurality of segmented image samples are matched with the original image respectively to obtain the orientation of each segmented image sample relative to the preset position;

[0109] Based on the orientation, determine the relative position of each block image sample in the plurality of block image samples with respect to the preset position;

[0110] Obtain multiple relative positions corresponding to the multiple segmented image samples, and determine the labeled position of the segmented image at the preset position as the vector representation of the multiple relative positions.

[0111] In this embodiment of the disclosure, reference is made to Figure 3As shown, multiple segmented image samples located in the same original image are obtained. For example, all the segmented image samples come from image 300. The position of segmented image sample 301 is used as the origin of the coordinate system. Since the segmented image samples originate from the original image, each segmented image sample can be matched to its corresponding position in the original image. If the x-coordinate of another segmented image is to the left of the x-coordinate of segmented image 301, then the x-coordinate of the other segmented image is set to 1; if the x-coordinate of another segmented image is to the right of the x-coordinate of segmented image 301 or in the same row, then the x-coordinate of the other segmented image is set to 0; if the y-coordinate of another segmented image is above the y-coordinate of segmented image 301, then the y-coordinate of the other segmented image is set to 1; if the y-coordinate of another segmented image is below the y-coordinate of segmented image 301 or in the same column, then the y-coordinate of the other segmented image is set to 0. For example, the top-left corner of the image block has its horizontal coordinate located to the left of block image 301 and its vertical coordinate located above block image 301. The relative position coordinates of the top-left corner of the image block are represented as (1,1). The labeled position of block image 301 is a vector representation of the relative position of other block images. For example, the labeled position of block image sample 301 is represented as in equation (1) above.

[0112] The location annotation method provided in this disclosure can automatically annotate the locations of segmented images. The vector-based annotation method is more in line with computer processing languages, eliminating the need for further conversion of the annotated locations.

[0113] In one possible implementation, the prediction model includes a self-attention mechanism network, wherein inputting the segmented image samples into the prediction model and outputting the predicted positions of the segmented image samples by the prediction model includes:

[0114] The features of block image samples in a sequence of multiple block image samples are obtained. The sequence of multiple block image samples is obtained by randomly sorting multiple block image samples located in the same original image multiple times.

[0115] The features of the multiple segmented image sample sequences are respectively input into the prediction model, and the prediction model outputs multiple predicted features of the segmented image samples.

[0116] The predicted location of the segmented image sample is determined based on the average value of the multiple predicted features.

[0117] In this embodiment, multiple random sorting operations on the original image yield multiple block image sample sequences. The features of each block image sample sequence are input into the prediction model, which then outputs multiple predicted features for each block image sample. For example, taking block image sample m as an example, the multiple block image sample sequences contain n block image samples m. The number of times the input to the self-attention mechanism network is also n, resulting in n predicted features. The processing of other block image samples in the block image sample sequence is the same as that of block image sample m. Therefore, each block image sample yields n predicted features. In one example, the average value of the multiple predicted features corresponding to each block image sample in the block image sample sequence is taken. In another example, a fully connected layer is used to map this average value to the location space to obtain the predicted location of the block image.

[0118] In this embodiment, the corresponding position is determined based on the average value of the predicted features, which reduces the problem of large deviation in the convergence direction due to large errors in a single predicted feature, thereby promoting faster convergence of model training.

[0119] In one possible implementation, the prediction model includes a self-attention mechanism network, wherein inputting the segmented image samples into the prediction model and outputting the predicted positions of the segmented image samples by the prediction model includes:

[0120] The features of block image samples in a sequence of multiple block image samples are obtained. The sequence of multiple block image samples is obtained by randomly sorting multiple block image samples located in the same original image multiple times.

[0121] The features of the multiple segmented image sample sequences are respectively input into the prediction model, and the prediction model outputs multiple predicted features of the segmented image samples.

[0122] Based on the multiple prediction features, multiple prediction positions of the segmented image sample are determined, and the average of the multiple prediction positions is taken as the final prediction position of the segmented image sample.

[0123] In this embodiment, the feature acquisition of multiple segmented image sample sequences is achieved by randomly sorting multiple segmented image samples located in the same original image multiple times. The features of these multiple segmented image sample sequences are input into the prediction model, which then outputs multiple predicted features for each segmented image sample. This is the same as in the previous embodiment and will not be repeated here. In one example, the predicted features corresponding to each segmented image sample in the segmented image sample sequence are taken. A fully connected layer is used to map these predicted features to a location space to obtain the predicted position of the segmented image. For a segmented image sample m, multiple segmented image sample sequences correspond to multiple predicted positions, and the average of these multiple predicted positions is taken as the final predicted position of the segmented image sample.

[0124] In this embodiment of the disclosure, the final predicted position of the segmented image sample is obtained by using the average of multiple predicted positions. This reduces the problem of large deviation in the convergence direction due to large errors in a single predicted position, thereby promoting faster convergence of model training.

[0125] It should be understood that although the steps in the flowchart are shown sequentially as indicated by the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowchart may include multiple steps or stages, which are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages in other steps.

[0126] It is understood that the same / similar parts between the various embodiments of the methods described above in this specification can be referred to each other. Each embodiment focuses on the differences from other embodiments, and relevant parts can be referred to the description of other method embodiments.

[0127] Figure 5 This is a block diagram illustrating an automatic jigsaw puzzle device according to an exemplary embodiment. (Refer to...) Figure 5 The device includes:

[0128] The acquisition module 501 is used to acquire the first block image sequence;

[0129] Prediction module 502 is used to input the first segmented image sequence into the prediction model, and output the position of each segmented image in the first segmented image sequence through the prediction model, wherein the prediction model is set to be trained based on the correspondence between the position of the segmented image sample and the position of the segmented image sample;

[0130] The jigsaw puzzle module 503 is used to jigsaw the segmented image sequence according to the position to obtain a jigsaw puzzle image.

[0131] Regarding the apparatus in the above embodiments, the specific manner in which each module performs its operation has been described in detail in the embodiments related to the method, and will not be elaborated upon here.

[0132] In one possible implementation, a task model is provided, which is obtained by training using the correspondence between the predictive features and task types described in any of the embodiments of this announcement.

[0133] In one possible implementation, the task model is an object detection model, which is obtained by training using the correspondence between the predicted features and task types described in any of the embodiments of this announcement, including:

[0134] Acquire the image to be detected;

[0135] The image is input into a preset target detection model, and the target object in the image is output by the target detection model. The target detection model is set to be trained on a self-attention mechanism network and a fully convolutional network based on the correspondence between the image samples and the labels of the target objects.

[0136] In this embodiment, the object detection model is trained on a self-attention network and a fully convolutional network based on the correspondence between image samples and target object labels. Specifically, it includes: acquiring an image sample set, which comprises multiple image samples, each image sample including one or more block image samples labeled with the target object's label; constructing an image classification model, which has training parameters; in one example, the object detection model includes a fully convolutional network and a self-attention network connected in sequence; in another example, the object detection model may employ any of the network structures described in the above embodiments; inputting the block image samples into the image classification model to generate prediction results; and iteratively adjusting the training parameters based on the difference between the prediction results and the target object's label until the difference meets a preset requirement.

[0137] In this embodiment, a self-attention network is used as the component network of the image object detection model. This allows for the full extraction of contextual information from each image block, thus enabling more accurate prediction of the label of each target object in the image. Furthermore, the fully convolutional neural network can accept input images of arbitrary size without requiring scaling of the images to be classified, preserving the detailed information of the images and resulting in more accurate and reliable prediction results.

[0138] In one possible implementation, the task model is an image segmentation model, which is obtained by training using the correspondence between predicted features and task types as described in any of the embodiments of this announcement, including:

[0139] Obtain the image to be segmented;

[0140] The image is input into a preset image segmentation model, and the image segmentation model outputs one or more segmented objects in the image. The image segmentation model is trained on a self-attention network and a fully convolutional network based on the correspondence between the labels of image samples and segmented objects.

[0141] In this embodiment, the image segmentation model is trained on a self-attention network and a fully convolutional network based on the correspondence between image samples and the labels of segmented objects. Specifically, it includes: acquiring an image sample set, which comprises multiple image samples, each image sample including one or more block image samples, each block image sample labeled with a segmentation object label; constructing an image segmentation model, which has training parameters; in one example, the image segmentation model includes a fully convolutional network and a self-attention network connected in sequence; in another example, the image segmentation model may employ any of the network structures described in the above embodiments; inputting the block image samples into the image segmentation model to generate prediction results; and iteratively adjusting the training parameters based on the difference between the prediction results and the segmentation labels until the difference meets a preset requirement.

[0142] In this embodiment, a self-attention network is used as the component network of the image segmentation model. This allows for the full extraction of contextual information from each image block, thus enabling more accurate prediction of each segmented object. Furthermore, the fully convolutional neural network can accept input images of arbitrary size without requiring scaling of the images to be classified, preserving image details and resulting in more accurate and reliable predictions.

[0143] Figure 6This is a block diagram illustrating an electronic device 600 for automatic puzzle-solving according to an exemplary embodiment. For example, the electronic device 600 may be a mobile phone, computer, digital broadcasting terminal, messaging device, game console, tablet device, medical device, fitness equipment, personal digital assistant, etc.

[0144] Reference Figure 6 The electronic device 600 may include one or more of the following components: processing component 602, memory 604, power supply component 606, multimedia component 608, audio component 610, input / output (I / O) interface 612, sensor component 614, and communication component 616.

[0145] Processing component 602 typically controls the overall operation of electronic device 600, such as operations associated with display, telephone calls, data communication, camera operation, and recording operations. Processing component 602 may include one or more processors 620 to execute instructions to perform all or part of the steps of the methods described above. Furthermore, processing component 602 may include one or more modules to facilitate interaction between processing component 602 and other components. For example, processing component 602 may include a multimedia module to facilitate interaction between multimedia component 608 and processing component 602.

[0146] Memory 604 is configured to store various types of data to support the operation of electronic device 600. Examples of such data include instructions for any application or method operating on electronic device 600, contact data, phonebook data, messages, pictures, videos, etc. Memory 604 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, optical disk, or graphene storage.

[0147] Power supply component 606 provides power to various components of electronic device 600. Power supply component 606 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to electronic device 600.

[0148] Multimedia component 608 includes a screen that provides an output interface between the electronic device 600 and the user. In some embodiments, the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen may be implemented as a touchscreen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touches, swipes, and gestures on the touch panel. The touch sensors may sense not only the boundaries of the touch or swipe action but also the duration and pressure associated with the touch or swipe operation. In some embodiments, multimedia component 608 includes a front-facing camera and / or a rear-facing camera. When the electronic device 600 is in an operating mode, such as a shooting mode or a video mode, the front-facing camera and / or the rear-facing camera may receive external multimedia data. Each front-facing camera and rear-facing camera may be a fixed optical lens system or have focal length and optical zoom capabilities.

[0149] Audio component 610 is configured to output and / or input audio signals. For example, audio component 610 includes a microphone (MIC) configured to receive external audio signals when electronic device 600 is in an operating mode, such as call mode, recording mode, and voice recognition mode. The received audio signals may be further stored in memory 604 or transmitted via communication component 616. In some embodiments, audio component 610 also includes a speaker for outputting audio signals.

[0150] I / O interface 612 provides an interface between processing component 602 and peripheral interface modules, such as keyboards, click wheels, buttons, etc. These buttons may include, but are not limited to, home buttons, volume buttons, power buttons, and lock buttons.

[0151] Sensor assembly 614 includes one or more sensors for providing state assessments of various aspects of electronic device 600. For example, sensor assembly 614 can detect the on / off state of electronic device 600, the relative positioning of components such as the display and keypad of electronic device 600, changes in position of electronic device 600 or its components, the presence or absence of user contact with electronic device 600, orientation or acceleration / deceleration of device 600, and temperature changes of electronic device 600. Sensor assembly 614 may include a proximity sensor configured to detect the presence of nearby objects without any physical contact. Sensor assembly 614 may also include a light sensor, such as a CMOS or CCD image sensor, for use in imaging applications. In some embodiments, sensor assembly 614 may also include an accelerometer, gyroscope, magnetometer, pressure sensor, or temperature sensor.

[0152] Communication component 616 is configured to facilitate wired or wireless communication between electronic device 600 and other devices. Electronic device 600 can access wireless networks based on communication standards, such as WiFi, carrier networks (such as 2G, 3G, 4G, or 5G), or combinations thereof. In one exemplary embodiment, communication component 616 receives broadcast signals or broadcast-related information from an external broadcast management system via a broadcast channel. In one exemplary embodiment, communication component 616 also includes a near-field communication (NFC) module to facilitate short-range communication. For example, the NFC module may be implemented based on radio frequency identification (RFID) technology, Infrared Data Association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology, and other technologies.

[0153] In an exemplary embodiment, the electronic device 600 may be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field-programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic components to perform the methods described above.

[0154] In an exemplary embodiment, a computer-readable storage medium including instructions is also provided, such as a memory 604 including instructions, which can be executed by a processor 620 of an electronic device 600 to perform the above-described method. For example, the computer-readable storage medium may be a ROM, random access memory (RAM), CD-ROM, magnetic tape, floppy disk, and optical data storage device, etc.

[0155] In an exemplary embodiment, a computer program product is also provided, which includes instructions that can be executed by a processor 620 of an electronic device 600 to perform the above-described method.

[0156] Figure 7 This is a block diagram illustrating an electronic device 700 for automatic jigsaw puzzle assembly according to an exemplary embodiment. For example, the electronic device 700 may be a server. (Refer to...) Figure 7 The electronic device 700 includes a processing component 720, which further includes one or more processors, and memory resources represented by memory 722 for storing instructions, such as application programs, that can be executed by the processing component 720. The application programs stored in memory 722 may include one or more modules, each corresponding to a set of instructions. Furthermore, the processing component 720 is configured to execute instructions to perform the methods described above.

[0157] Electronic device 700 may further include: a power supply component 724 configured to perform power management of electronic device 700, a wired or wireless network interface 726 configured to connect electronic device 700 to a network, and an input / output (I / O) interface 727. Electronic device 700 may operate on an operating system stored in memory 722, such as Windows Server, Mac OS X, Unix, Linux, FreeBSD, or similar.

[0158] In an exemplary embodiment, a computer-readable storage medium including instructions is also provided, such as a memory 722 including instructions, which can be executed by a processor of an electronic device 700 to perform the above-described method. The storage medium may be a computer-readable storage medium, such as a ROM, random access memory (RAM), CD-ROM, magnetic tape, floppy disk, and optical data storage device.

[0159] In an exemplary embodiment, a computer program product is also provided, the computer program product including instructions that can be executed by a processor of an electronic device 700 to perform the above-described method.

[0160] It should be noted that the above-mentioned apparatus, electronic equipment, computer-readable storage medium, computer program product, etc., may also include other implementation methods according to the description of the method embodiments. For specific implementation methods, please refer to the description of the relevant method embodiments, which will not be elaborated here.

[0161] Other embodiments of this disclosure will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This disclosure is intended to cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this disclosure are indicated by the claims.

[0162] It should be understood that this disclosure is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this disclosure is limited only by the appended claims.

Claims

1. An automatic puzzle-solving method, characterized by, The method comprises: obtaining a first sequence of patch images; wherein the first sequence of patch images comprises a sequence composed of patch images from the same image; inputting the first sequence of patch images into a prediction model, and outputting the position of each first patch image in the first sequence of patch images through the prediction model, wherein the inputting the first sequence of patch images into the prediction model and outputting the position of each first patch image in the first sequence of patch images through the prediction model comprises: obtaining features of the first sequence of patch images and features of each second sequence of patch images, wherein the second patch images are obtained by randomly sorting the first sequence of patch images; inputting the features of the first sequence of patch images and the features of each second sequence of patch images into a self-attention mechanism network, and outputting prediction features through the self-attention mechanism network; obtaining the average of a plurality of prediction features corresponding to each first patch image in the first sequence of patch images; and determining the position of each first patch image according to the average; wherein the prediction model comprises a self-attention mechanism network, and the prediction model is set to be obtained by training according to the correspondence between patch image samples and the positions of the patch image samples; wherein the positions of the patch image samples are obtained in an automatic manner without manual annotation; and the position of each first patch image comprises a vector representation of the relative position of the first patch image and other patch images in the first sequence of patch images; determining the absolute position of each first patch image in the same image according to the relative position of each first patch image, and performing image assembly on the first sequence of patch images according to the absolute position to obtain an assembled image.

2. The method of claim 1, wherein, The prediction model is set to be obtained by training according to the correspondence between patch image samples and the positions of the patch image samples, which comprises: obtaining a sample set comprising patch image samples, wherein the patch image samples are labeled with positions; constructing a prediction model, wherein training parameters are set in the prediction model; inputting the patch image samples into the prediction model, and outputting the prediction positions of the patch image samples through the prediction model; iteratively adjusting the training parameters based on the difference between the prediction positions and the labeled positions until the difference meets a preset requirement, and obtaining the prediction model.

3. The method of claim 2, wherein, The prediction model comprises a self-attention mechanism network, and the inputting the patch image samples into the prediction model and outputting the prediction positions of the patch image samples through the prediction model comprises: obtaining features of the patch image samples; the obtaining features of the patch image samples comprises: performing convolution on the patch image samples by using a convolutional neural network to obtain the features of the patch image samples; inputting the features into a self-attention mechanism network, and outputting prediction features through the self-attention mechanism network; determining the prediction positions of the patch images according to the prediction features.

4. The method of claim 3, wherein, The obtaining features of the patch image samples comprises: Linearly mapping the block image sample by using a mapping matrix to obtain a feature of the block image sample.

5. The method of claim 1, wherein, The labeling manner of the position comprises: Obtaining a plurality of block image samples located in a same original image and determining a preset position in the original image as a coordinate origin position; Respectively matching the plurality of block image samples with the original image to obtain an orientation of each block image sample in the plurality of block image samples relative to the preset position; According to the orientation, determining a relative position of each block image sample in the plurality of block image samples relative to the preset position; Obtaining a plurality of relative positions corresponding to the plurality of block image samples, and determining a labeled position of a block image on the preset position as a vector representation of the plurality of relative positions.

6. An automatic puzzle device, characterized in that comprising: An acquisition module configured to acquire a first block image sequence, wherein the first block image sequence comprises a sequence of block images from a same image; A prediction module configured to input the first block image sequence into a prediction model, and output a position of each first block image in the first block image sequence via the prediction model, wherein the inputting of the first block image sequence into the prediction model and the outputting of the position of each first block image in the first block image sequence via the prediction model comprises: acquiring a feature of the first block image sequence and a feature of each second block image sequence in a plurality of second block image sequences, wherein the second block image is obtained by randomly sorting the first block image sequence; inputting the feature of the first block image sequence and the feature of each second block image sequence in the plurality of second block image sequences into a self-attention mechanism network, and outputting a prediction feature via the self-attention mechanism network; acquiring an average value of a plurality of prediction features corresponding to each first block image in the first block image sequence; and determining the position of each first block image according to the average value, wherein the prediction model comprises the self-attention mechanism network, and the prediction model is configured to be trained according to a corresponding relationship between a block image sample and a position of the block image sample, wherein the position of the block image sample is obtained in an automatic manner without manual labeling, and the position of each first block image comprises a vector representation of a relative position of the first block image to other block images in the first block image sequence; A puzzle module configured to determine an absolute position of each first block image in the same image according to the relative position of each first block image, and perform puzzle on the first block image sequence according to the absolute position to obtain a puzzled image. Comprise:

7. An electronic device, comprising: A processor; A memory for storing instructions executable by the processor; The processor is configured to execute the instructions to implement the automatic puzzle method of any one of claims 1 to 5. When the instructions in the computer-readable storage medium are executed by the processor of the electronic device, the electronic device can perform the automatic puzzle method of any one of claims 1 to 5.

8. A computer-readable storage medium, characterized in that, ​ 9. A computer program product comprising instructions therein, characterised in that, The instructions, when executed by a processor of an electronic device, enable the electronic device to perform the automatic jigsaw puzzle method of any one of claims 1-5.

Images