Method, apparatus and storage medium for semantic synchronization based on fixed image

Abstract

The application provides a fixed image-based semantic synchronization method and device and a storage medium, comprising: obtaining a preset shared picture and to-be-sent data through a sending end; encoding and expanding the preset shared picture and the to-be-sent data respectively to obtain a first vector sequence and a second vector sequence; splicing the second vector sequence after the first vector sequence to obtain a data sequence; sending the data sequence to a receiving end based on a communication channel; decoding the data sequence according to the window length of a detection window by the receiving end; in the case that the decoded picture obtained by decoding includes part of the picture content in the preset shared picture, comparing the position of the preset shared sub-picture in the decoded picture with the position in the preset shared picture to determine the moving number of bits and the moving direction of the detection window, thereby determining the synchronization position in the data sequence. The application can solve the problem that the synchronization loses stability due to possible interference and clock drift in the transmission process.

Description

Technical Field

[0001] This invention relates to the field of communication technology, and in particular to a semantic synchronization method, apparatus, and storage medium based on fixed images. Background Technology

[0002] For decades, research in communications has primarily focused on how to transmit encoded symbols more efficiently from sender to receiver. However, the proliferation of smart applications inevitably requires massive data transmissions, posing a significant challenge to the low latency and high data transmission rates of existing communication systems. This has prompted researchers to consider next-generation synchronous communication system paradigms.

[0003] Traditional synchronous communication is a symbol-based transmission method designed to ensure that the sender and receiver remain synchronized in time and frequency. In traditional synchronous communication, the sender encodes data into a series of symbols and transmits these symbols to the receiver via a clock signal; the receiver uses the same clock signal to demodulate the transmitted symbols and recover the original data.

[0004] However, in traditional synchronous communication, clock synchronization is a critical step. Both the sending and receiving ends need to use the same clock source to ensure accurate transmission and demodulation of symbols. Therefore, interference and clock drift that may occur during transmission can cause synchronization to become unstable. Summary of the Invention

[0005] In view of this, embodiments of the present invention provide a semantic synchronization method, apparatus and storage medium based on fixed images to eliminate or improve one or more defects existing in the prior art and solve the problem of synchronization instability caused by interference and clock drift that may occur during transmission.

[0006] One aspect of the present invention provides a semantic synchronization method based on a fixed image, used in a transmitting end where a communication channel is established between the transmitting end and the receiving end, the method comprising the following steps:

[0007] Retrieve preset shared images and data to be sent;

[0008] The preset shared image and the data to be sent are encoded by the encoder to obtain the first feature matrix corresponding to the preset shared image and the second feature matrix corresponding to the data to be sent.

[0009] The first feature matrix is ​​expanded into a one-dimensional vector according to the first expansion strategy to obtain the first vector sequence.

[0010] The second feature matrix is ​​expanded into a one-dimensional vector according to the second expansion strategy, resulting in the second vector sequence; the first expansion strategy is different from the second expansion strategy.

[0011] The second vector sequence is concatenated with the first vector sequence to obtain the data sequence;

[0012] The data sequence is sent to the receiving end via a communication channel. Upon receiving the data sequence, the receiving end decodes it according to the window length of the detection window using a decoder. If the decoded image includes part of the image content from a preset shared image, the position of the preset shared sub-image in the decoded image is compared with the position of the preset shared sub-image in the preset shared image to obtain the position difference. Based on the position difference, the number of bits and the direction of movement of the detection window are determined. The detection window is moved according to the number of bits and the direction of movement to determine the synchronization position in the data sequence.

[0013] Optionally, the first expansion strategy is to expand sequentially according to the order of the submatrices in the feature matrix, the row number order within each submatric, and the column number order within each submatric;

[0014] The second expansion strategy includes expanding in row number order or expanding in column number order.

[0015] Optionally, the data to be sent may include image data, voice data, text data, video data, point cloud data, or extended reality data.

[0016] Optionally, the encoder includes a semantic encoder and / or a basic encoder, wherein the semantic encoder includes a first semantic encoder and a second semantic encoder; the model structures of the first semantic encoder and the second semantic encoder are different;

[0017] The preset shared image and the data to be sent are encoded separately using an encoder, including:

[0018] The first semantic encoder is used to encode the preset shared image to obtain the first feature matrix; the second semantic encoder or the basic encoder is used to encode the data to be sent to obtain the second feature matrix.

[0019] or,

[0020] The preset shared image is encoded using a second semantic encoder to obtain a first feature matrix; the data to be sent is encoded using either the first semantic encoder or the basic encoder to obtain a second feature matrix.

[0021] Optionally, the preset shared sub-image is an image cropped from the preset shared image, including a rectangular image with the top left corner as the vertex, a rectangular image with the top right corner as the vertex, a rectangular image with the bottom left corner as the vertex, a rectangular image with the bottom right corner as the vertex, or a rectangular image of the central area in the preset shared image.

[0022] One aspect of the present invention provides a semantic synchronization method based on a fixed image for use in a receiving end, the method comprising the following steps:

[0023] The data sequence is received based on the communication channel between the sender and the receiver; the data sequence is obtained by the sender encoding the preset shared image and the data to be sent respectively, expanding them into a first vector sequence and a second vector sequence, and then concatenating them.

[0024] The decoder decodes the data sequence according to the window length of the detection window to obtain the decoded image;

[0025] The decoded images are categorized to obtain the categorization results; the categorization results are used to indicate whether the decoded images include some image content from the preset shared images.

[0026] If the classification result indicates that the decoded image includes part of the image content in the preset shared image, the position of the preset shared sub-image in the decoded image is compared with the position of the preset shared sub-image in the preset shared image to obtain the position difference;

[0027] Based on the positional differences, determine the number of steps and the direction of movement of the detection window;

[0028] The detection window is moved according to the number of bits and the direction of movement to determine the synchronization position in the data sequence.

[0029] Optionally, if the decoded image does not include some image content from the preset shared image, the method further includes:

[0030] The detection window is moved according to the preset number of moves and the preset direction of movement;

[0031] The process involves decoding the data sequence using a decoder according to the window length of the detection window to obtain a decoded image.

[0032] Optionally, the decoded images are classified to obtain classification results, including:

[0033] The decoded images are classified using a pre-defined neural network classification model to obtain the classification results;

[0034] or,

[0035] The decoded image is classified using a preset edge detection algorithm to obtain the classification result;

[0036] or,

[0037] The classification result is obtained by calculating the peak signal-to-noise ratio between the decoded image and the preset shared image;

[0038] or,

[0039] The classification result is obtained by calculating the structural similarity index between the decoded image and the preset shared image.

[0040] Another aspect of the present invention provides an apparatus for semantic synchronization based on a fixed image, the apparatus comprising: a processor and a memory, characterized in that the memory stores computer instructions, the processor is used to execute the computer instructions stored in the memory, and when the computer instructions are executed by the processor, the apparatus implements the steps of the semantic synchronization method based on the fixed image described above.

[0041] Another aspect of the present invention provides a computer-readable storage medium having a computer program stored thereon, characterized in that the program, when executed by a processor, implements the steps of the above-described semantic synchronization method based on a fixed image.

[0042] The beneficial effects of the present invention are at least as follows:

[0043] The present invention discloses a semantic synchronization method, apparatus, and storage medium based on a fixed image. The method involves: acquiring a preset shared image and data to be sent at a transmitting end; encoding and expanding the preset shared image and data to be sent to obtain a first vector sequence and a second vector sequence; concatenating the second vector sequence with the first vector sequence to obtain a data sequence; sending the data sequence to a receiving end via a communication channel; decoding the data sequence at the receiving end according to the window length of a detection window; and, if the decoded image includes part of the image content in the preset shared image, comparing the position of the preset shared sub-image in the decoded image with its position in the preset shared image to determine the number of bits and direction of movement of the detection window, thereby determining the synchronization position in the data sequence. It can solve the problem of synchronization instability caused by interference and clock drift during transmission. Based on the positional difference of the preset shared subgraph in the decoded image and the preset shared image, it determines the number of bits and the direction of movement of the detection window, thereby determining the synchronization position in the data sequence. It eliminates the need for a synchronization clock, avoids the impact of clock drift, and improves synchronization stability. When the decoded image includes part of the image content in the preset shared image, it classifies the decoded image into the preset shared image class and performs position comparison. When the decoded image does not include part of the image content in the preset shared image, it classifies the decoded image into the noise class, avoiding the impact of noise and multipath effects on the preset shared image, further improving synchronization stability. Therefore, it avoids data transmission errors or loss and improves the success rate of data transmission.

[0044] Furthermore, based on the positional differences of the preset shared subgraph in the decoded image and the preset shared image, the number of bits and the direction of movement of the detection window are determined. Then, the detection window is moved according to the number of bits and the direction of movement to determine the synchronization position in the data sequence, thereby achieving semantic synchronization. This avoids dependence on strict clock synchronization and deterministic transmission order, thus avoiding the impact of transmission delay and bandwidth heterogeneity in large-scale heterogeneous networks. This improves transmission performance and stability in large-scale heterogeneous networks, thereby meeting the requirements of high bandwidth, low latency, and high throughput for complex transmission tasks, and improving the real-time performance and efficiency of transmission tasks.

[0045] Additional advantages, objects, and features of the invention will be set forth in part in the description which follows, and will also become apparent in part to those skilled in the art upon studying the description, or may be learned by practice of the invention. The objects and other advantages of the invention can be realized and obtained by means of the structures specifically pointed out in the specification and drawings.

[0046] Those skilled in the art will understand that the objectives and advantages achievable with this invention are not limited to those specifically described above, and that the above and other objectives achievable with this invention will become clearer from the following detailed description. Attached Figure Description

[0047] The accompanying drawings, which are provided to further illustrate the invention and form part of this application, are not intended to limit the scope of the invention.

[0048] Figure 1 This is a flowchart of a semantic synchronization method based on a fixed image according to an embodiment of the present invention;

[0049] Figure 2 This is a schematic diagram of a preset shared image in a semantic synchronization method based on fixed images according to an embodiment of the present invention;

[0050] Figure 3 This refers to a preset shared image in a semantic synchronization method based on fixed images, as described in an embodiment of the present invention.

[0051] Figure 4 The image is a decoded image of the semantic synchronization method based on fixed images according to an embodiment of the present invention;

[0052] Figure 5 The image is a decoded image of the semantic synchronization method based on fixed images according to an embodiment of the present invention;

[0053] Figure 6 The image is a decoded image of the semantic synchronization method based on fixed images according to an embodiment of the present invention;

[0054] Figure 7The image is a decoded image of the semantic synchronization method based on fixed images according to an embodiment of the present invention;

[0055] Figure 8 This is a block diagram of a semantic synchronization device based on a fixed image according to an embodiment of the present invention;

[0056] Figure 9 This is a block diagram of a semantic synchronization device based on a fixed image according to an embodiment of the present invention;

[0057] Figure 10 This is a block diagram of a semantic synchronization device based on a fixed image according to an embodiment of the present invention. Detailed Implementation

[0058] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the embodiments and accompanying drawings. Here, the illustrative embodiments and descriptions of this invention are used to explain the invention, but are not intended to limit the invention.

[0059] It should also be noted that, in order to avoid obscuring the invention with unnecessary details, only the structures and / or processing steps closely related to the solution according to the invention are shown in the accompanying drawings, while other details that are not closely related to the invention are omitted.

[0060] It should be emphasized that the term "including / comprises" as used herein refers to the presence of a feature, element, step, or component, but does not exclude the presence or addition of one or more other features, elements, steps, or components.

[0061] It should also be noted that, unless otherwise specified, the term "connection" in this article can refer not only to a direct connection, but also to an indirect connection involving an intermediary.

[0062] In the following description, embodiments of the invention will be illustrated with reference to the accompanying drawings. In the drawings, the same reference numerals represent the same or similar parts, or the same or similar steps.

[0063] The semantic synchronization method based on fixed images provided in this application will be described in detail below.

[0064] like Figure 1 As shown, this application provides a semantic synchronization method based on a fixed image. This embodiment uses the method for both a sending end and a receiving end as an example. The sending end and receiving end refer to terminals that need to perform semantic synchronization based on a fixed image, including but not limited to electronic devices such as mobile phones, computers, or servers. The device types of the sending end and the receiving end may be the same or different. This embodiment does not limit the implementation methods of the sending end and the receiving end. Specifically, the semantic synchronization method based on a fixed image includes at least the following steps S101 to S112:

[0065] Step S101: The sending end obtains the preset shared image and the data to be sent.

[0066] The preset shared image refers to a pre-set shared image, including but not limited to images with resolutions of 1920×1080, 512×512, or 800×600. The preset shared image can be a square image, a rectangular image, or a circular image, etc. This embodiment does not limit the implementation method of the preset shared image.

[0067] The data to be sent includes image data, voice data, text data, video data, point cloud data, or extended reality (XR) data.

[0068] In actual implementation, the data to be sent can also be other data types, which will not be listed in this embodiment.

[0069] In this embodiment, the preset shared images and data to be sent can be pre-stored in the local storage medium of the sending end, or pre-stored in a remote server with a communication connection to the sending end, or pre-stored in a removable storage medium independent of the sending end, such as a USB flash drive or hard drive. This embodiment does not limit the storage format of the preset shared images and data to be sent.

[0070] In step S102, the sending end uses an encoder to encode the preset shared image and the data to be sent, respectively, to obtain the first feature matrix corresponding to the preset shared image and the second feature matrix corresponding to the data to be sent.

[0071] The encoder that encodes the preset shared image is different from the encoder that encodes the data to be sent, and the encoder that encodes the preset shared image is a semantic encoder (SE). Correspondingly, the model weights of the semantic encoder-decoder are also stored in the storage medium of the sending end.

[0072] In one example, the encoder includes a semantic encoder and a base encoder (BE).

[0073] At this point, the preset shared image and the data to be sent are encoded using encoders to obtain the first feature matrix corresponding to the preset shared image and the second feature matrix corresponding to the data to be sent. This includes: encoding the preset shared image using a semantic encoder to obtain the first feature matrix corresponding to the preset shared image; and encoding the data to be sent using a basic encoder to obtain the second feature matrix corresponding to the data to be sent.

[0074] In another example, the encoder includes a first semantic encoder and a second semantic encoder, and the model structure of the first semantic encoder is different from that of the second semantic encoder. Accordingly, the storage medium at the sending end also stores the model weights of the first semantic encoder and the model weights of the second semantic encoder.

[0075] At this point, the preset shared image and the data to be sent are encoded using encoders respectively to obtain a first feature matrix corresponding to the preset shared image and a second feature matrix corresponding to the data to be sent. This includes: encoding the preset shared image using a first semantic encoder to obtain a first feature matrix corresponding to the preset shared image; encoding the data to be sent using a second semantic encoder to obtain a second feature matrix corresponding to the data to be sent; or, encoding the preset shared image using a second semantic encoder to obtain a first feature matrix corresponding to the preset shared image; and encoding the data to be sent using a first semantic encoder to obtain a second feature matrix corresponding to the data to be sent.

[0076] In yet another example, the encoder includes a semantic encoder and a base encoder, wherein the semantic encoder includes a first semantic encoder and a second semantic encoder, and the model structure of the first semantic encoder is different from that of the second semantic encoder.

[0077] Specifically, the preset shared image and the data to be sent are encoded by encoders respectively, including: encoding the preset shared image using a first semantic encoder to obtain a first feature matrix; encoding the data to be sent using a second semantic encoder or a basic encoder to obtain a second feature matrix; or, encoding the preset shared image using a second semantic encoder to obtain a first feature matrix; and encoding the data to be sent using a first semantic encoder or a basic encoder to obtain a second feature matrix.

[0078] In step S103, the sending end expands the first feature matrix into a one-dimensional vector according to the first expansion strategy to obtain the first vector sequence.

[0079] The first expansion strategy involves expanding sequentially according to the order of the submatrices in the feature matrix, the row number order within each submatric, and the column number order within each submatric. Alternatively, the first expansion strategy can be an expansion strategy that follows the channel direction of the feature matrix.

[0080] For example, taking the feature matrix A as an example, the feature matrix A is represented by the following formula:

[0081] A=[[[a_111,a_112,a_113],[a_121,a_122,a_123],[a_131,a_132,a_133]],

[0082] [[a_211,a_212,a_213],[a_221,a_222,a_223],[a_231,a_232,a_233]],

[0083] [[a_311,a_312,a_313],[a_321,a_322,a_323],[a_331,a_332,a_333]]].

[0084] Following the first expansion strategy, the feature matrix A is expanded into a one-dimensional vector, resulting in a vector sequence a. The vector sequence a is represented by the following formula:

[0085] a=[a_111,a_211,a_311,a_112,a_212,a_312,a_113,a_213,a_313,a_121,a_221,

[0086] a_321,a_122,a_222,a_322,a_123,a_223,a_323,a_131,a_231,a_331,a_132,a_232,

[0087] a_332, a_133, a_233, a_333).

[0088] In step S104, the sending end expands the second feature matrix into a one-dimensional vector according to the second expansion strategy to obtain the second vector sequence.

[0089] In this embodiment, the first expansion strategy is different from the second expansion strategy; wherein, the second expansion strategy includes a strategy of expanding according to row number order or a strategy of expanding according to column number order.

[0090] In one example, the second expansion strategy includes a strategy of expanding in row number order.

[0091] For example, taking the feature matrix B as an example, the feature matrix B is expressed by the following formula:

[0092] B=[[[b_111,b_112,b_113],[b_121,b_122,b_123],[b_131,b_132,b_133]],

[0093] [[b_211,b_212,b_213],[b_221,b_222,b_223],[b_231,b_232,b_233]],

[0094] [[b_311,b_312,b_313],[b_321,b_322,b_323],[b_331,b_332,b_333]]].

[0095] Following the second expansion strategy, the feature matrix B is expanded into a one-dimensional vector, resulting in a vector sequence b. The vector sequence b is represented by the following formula:

[0096] b=[b_111,b_112,b_113,b_211,b_212,b_213,b_311,b_312,b_313,b_121,b_122,b _123,b_221,b_222,b_223,b_321,b_322,b_323,b_131,b_132,b_133,b_231,b_232,

[0097] b_233,b_331,b_332,b_333。

[0098] In another example, the second expansion strategy includes a strategy that expands in the order of example numbers.

[0099] For example, taking the feature matrix C as an example, the feature matrix C is expressed by the following formula:

[0100] C=[[[c_111,c_112,c_113],[c_121,c_122,c_123],[c_131,c_132,c_133]],

[0101] [[c_211,c_212,c_213],[c_221,c_222,c_223],[c_231,c_232,c_233]],

[0102] [[c_311,c_312,c_313],[c_321,c_322,c_323],[c_331,c_332,c_333]]].

[0103] Following the second expansion strategy, the feature matrix C is expanded into a one-dimensional vector, resulting in a vector sequence c. The vector sequence c is represented by the following equation:

[0104] c=[c_111,c_121,c_131,c_211,c_221,c_231,c_311,c_321,c_331,c_112,c_122,c _132,c_212,c_222,c_232,c_312,c_322,c_332,c_113,c_123,c_133,c_213,c_223,

[0105] c_233,c_313,c_323,c_333].

[0106] In actual implementation, the second unfolding strategy may also include other unfolding strategies, which will not be listed one by one in this embodiment.

[0107] In step S105, the sending end concatenates the second vector sequence with the first vector sequence to obtain the data sequence.

[0108] In this embodiment, the number of data to be sent is at least one, and correspondingly, the number of second vector sequences is also at least one.

[0109] When there are two or more second vector sequences, the second vector sequences are concatenated with the first vector sequences to obtain a data sequence, including: concatenating each second vector sequence with a separate first vector sequence to obtain a data subsequence; or concatenating each data subsequence sequentially to obtain a data sequence.

[0110] For example, if the first vector sequence includes the first vector sequence D, and the second vector sequence includes the second vector sequence E and the second vector sequence F, then concatenating the second vector sequence E to the first vector sequence D yields the data subsequence DE. Concatenating the second vector sequence F to the first vector sequence D yields the data subsequence DF. Finally, concatenating the data subsequence DE and the data subsequence DF yields the data sequence DEDF.

[0111] Step S106: The sending end sends the data sequence to the receiving end based on the communication channel.

[0112] After receiving the data sequence, the receiving end decodes the data sequence according to the window length of the detection window using a decoder. If the decoded image includes part of the image content in the preset shared image, the position of the preset shared sub-image in the decoded image is compared with the position of the preset shared sub-image in the preset shared image to obtain the position difference. Based on the position difference, the number of bits and the direction of movement of the detection window are determined. The detection window is moved according to the number of bits and the direction of movement to determine the synchronization position in the data sequence.

[0113] The window length of the detection window is equal to the length of the first vector sequence. For example, if the first feature matrix G is a 1×3×3×3 matrix, then after expanding the first feature matrix G into the first sequence vector g, the length of the first sequence vector g is 27. In this case, the window length of the detection window is 27.

[0114] In this embodiment, some of the image content can be the image content of any area in a preset shared image.

[0115] Preset shared sub-images refer to images cropped from preset shared images, including rectangular images with the top left corner as the vertex, rectangular images with the top right corner as the vertex, rectangular images with the bottom left corner as the vertex, rectangular images with the bottom right corner as the vertex, or rectangular images of the central area within the preset shared images.

[0116] In actual implementation, the preset shared sub-image can also be an image of other shapes, such as a circle or a triangle. This embodiment does not limit the shape of the preset shared sub-image.

[0117] In this embodiment, the number of preset shared sub-images is at least one. The shape of the preset shared image includes, but is not limited to, rectangles, squares, or circles. The preset shared image can be pre-stored in the local storage medium of the receiving end, or pre-stored in a remote server with a communication connection to the receiving end, or pre-stored in a removable storage medium independent of the receiving end, such as a USB flash drive or hard drive.

[0118] Step S107: The receiving end receives the data sequence based on the communication channel between it and the sending end.

[0119] The data sequence is obtained by the sending end encoding the preset shared image and the data to be sent, expanding them into a first vector sequence and a second vector sequence, and then concatenating them.

[0120] It should be added that the distinction between the sending end and the receiving end is only for functional purposes. In actual implementation, the same terminal can be either the sending end, i.e., the terminal that generates and sends the data sequence, or the receiving end, i.e., the terminal that determines the synchronization position in the data sequence.

[0121] In step S108, the receiving end decodes the data sequence according to the window length of the detection window using a decoder to obtain a decoded image.

[0122] In step S109, the receiving end classifies the decoded image to obtain the classification result.

[0123] The classification result is used to indicate whether the decoded image includes some image content from the preset shared image.

[0124] Specifically, the classification results include a noise class and a preset shared image class. If the decoded image does not include some of the image content from the preset shared image, the decoded image will be classified as noise; if the decoded image includes some of the image content from the preset shared image, the decoded image will be classified as shared with the preset shared image.

[0125] In one example, a decoded image can be input into a preset neural network classification model, which will then output a classification result. This preset neural network classification model is trained using training data.

[0126] If the preset neural network classification model outputs a classification result indicating that the decoded image and the preset shared image are of the same type, confirm that the decoded image includes part of the image content from the preset shared image. If the classification result indicates that the decoded image and the preset shared image are of different types, confirm that the decoded image is noise.

[0127] Optionally, the training data includes sample decoded images and the corresponding sample classification results.

[0128] Accordingly, the training process of the preset neural network classification model includes: inputting the sample decoded image into the preset neural network model to obtain the training result; inputting the training result and the sample classification result corresponding to the sample decoded image into the loss function to obtain the loss result; training the neural network model based on the loss result to reduce the difference between the training result and the corresponding sample classification result until the neural network model converges to obtain the preset neural network classification model.

[0129] The neural network model can be a Visual Geometry Group Net (VGGNet), a Residual Network (ResNet), or a Densely Connected Convolutional Network (DenseNet). This embodiment does not limit the implementation method of the neural network model.

[0130] In another example, edge detection algorithms are used to detect the edge information of the decoded image and the preset shared image. By comparing the edge information, the classification result between the decoded image and the preset shared image is obtained.

[0131] The edge detection algorithm can be Canny Edge Detection, Sobel Operator, Scharr Operator, etc. This embodiment does not limit the implementation of the edge detection algorithm.

[0132] In another example, the classification result is obtained by calculating the Peak Signal-to-Noise Ratio (PSNR) or the Structural Similarity Index (SSIM) between the decoded image and the preset shared image.

[0133] If the peak signal-to-noise ratio (PSNR) between the decoded image and the preset shared image is greater than or equal to a preset PNR, the decoded image is confirmed to include a portion of the image content in the preset shared image; if the PNR between the decoded image and the preset shared image is less than the preset PNR, the decoded image is confirmed to be noise; or, if the structural similarity index between the decoded image and the preset shared image is greater than or equal to a preset structural similarity index, the decoded image is confirmed to include a portion of the image content in the preset shared image; if the structural similarity index between the decoded image and the preset shared image is less than the preset structural similarity index, the decoded image is confirmed to be noise.

[0134] The preset peak signal-to-noise ratio can be 20 dB, 40 dB, or 50 dB, etc.; the structural similarity index is in the range of [-1, 1]. Correspondingly, the preset structural similarity index can be 0.4, 0.5, or 0.7, etc. This embodiment does not limit the preset peak signal-to-noise ratio and the preset structural similarity index.

[0135] Specifically, the decoded image is classified to obtain a classification result, including: classifying the decoded image using a preset neural network classification model to obtain a classification result; or, classifying the decoded image using a preset edge detection algorithm to obtain a classification result; or, calculating the peak signal-to-noise ratio between the decoded image and a preset shared image to obtain a classification result; or, calculating the structural similarity index between the decoded image and a preset shared image to obtain a classification result.

[0136] In step S110, if the classification result indicates that the decoded image includes part of the image content in the preset shared image, the receiving end compares the position of the preset shared sub-image in the decoded image with the position of the preset shared sub-image in the preset shared image to obtain the position difference.

[0137] If the decoded image is classified as a preset shared image, meaning that the decoded image includes part of the image content in the preset shared image, then the preset shared sub-image and the decoded image are template matched to compare the position coordinates of the preset shared sub-image in the decoded image and the position coordinates of the preset shared sub-image in the preset shared image, thus obtaining the positional difference.

[0138] The position coordinates refer to the relative position coordinates of the marker point in the preset shared sub-image within the preset shared image or decoded image. The marker point can be any point in the preset shared sub-image. If the preset shared sub-image is circular, the marker point can be the center of the preset shared sub-image; if the preset shared sub-image is square, the marker point can be the apex of the preset shared sub-image.

[0139] For example: Reference Figure 2Taking a preset shared image with a resolution of 512×512 as an example, four preset shared sub-images are cropped from this preset shared image, including sub-image A, sub-image B, sub-image C, and sub-image D. The marker point of each preset shared sub-image is the top left corner point. At this time, the position coordinates of sub-image A in the preset shared image are (128, 128), the position coordinates of sub-image B in the preset shared image are (128, 256), the position coordinates of sub-image C in the preset shared image are (256, 128), and the position coordinates of sub-image D in the preset shared image are (256, 256). Sub-images A, B, C, and D are matched with the decoded image in turn to determine the position coordinates of sub-images A, B, C, and D in the decoded image respectively. If the coordinates of sub-image A are (128, 128), the coordinates of sub-image B are (128, 256), the coordinates of sub-image C are (256, 128), and the coordinates of sub-image D are (256, 256), then by comparing the coordinates, it can be seen that there is no positional difference between the position of the preset shared sub-image in the decoded image and the position of the preset shared sub-image in the preset shared image. If the coordinates of sub-image A are (128, 272), the coordinates of sub-image B are (262, 306), the coordinates of sub-image C are (256, 272), and the coordinates of sub-image D are (260, 384), then by comparing the coordinates, it can be seen that some image content has not shifted vertically in the decoded image, but has shifted to the right by 176 - 128 = 48 pixels.

[0140] In addition, if the decoded image does not include part of the image content in the preset shared image, the method further includes: moving the detection window according to the preset number of moves and the preset moving direction; and performing the step of decoding the data sequence by the decoder according to the window length of the detection window to obtain the decoded image.

[0141] Step S111: The receiving end determines the number of bits and direction of movement of the detection window based on the position difference.

[0142] In this embodiment, the first feature sequence obtained after encoding the pre-defined shared image is a semantic vector, and its shape can be represented by [1,C,H,W]. Wherein, C represents the number of channels; H represents the number of symbols in the vertical direction; and W represents the number of symbols in the horizontal direction.

[0143] When the distance between the current position of the detection window and the synchronization position is not a multiple of C, the decoded picture obtained by the decoder is noise; when the distance between the current position and the synchronization position is C×m (m is a positive integer, and m<H), the decoded picture contains partial picture content of the shared image, and the decoded picture will shift left and right at this time; when the distance between the current position and the synchronization position is C×H×n (n is a positive integer, n<W), the decoded picture will shift up and down at this time; when the distance between the current position and the synchronization position is C×H×n+C×m (both m and n are positive integers, and m<H, n<W), the decoded picture will undergo both vertical shift and horizontal shift.

[0144] Based on this, in this embodiment, the shift direction and shift distance corresponding to the decoded picture are obtained through the coordinate difference between the position coordinate of the preset shared sub-picture in the preset shared picture and the position coordinate of the preset shared sub-picture in the decoded picture, and then the moving bit count and moving direction of the detection window are determined according to the shift direction and shift distance.

[0145] In step S112, the receiving end moves the detection window according to the moving bit count and the moving direction, and determines the synchronization position in the data sequence.

[0146] Specifically, that the receiving end moves the detection window according to the moving bit count and the moving direction and determines the synchronization position in the data sequence includes: when the moving direction is left, subtracting the moving bit count from the current position index of the detection window to determine the synchronization position in the data sequence; when the moving direction is right, adding the moving bit count to the current position index of the detection window to determine the synchronization position in the data sequence.

[0147] Traditional synchronous communication usually relies on strict clock synchronization and a deterministic transmission sequence, which may limit the scalability of the network and the flexibility of applications in some cases. In large-scale heterogeneous networks, different devices, transmission media and network topologies may have differences, resulting in heterogeneous transmission delays and bandwidths. Traditional synchronous communication cannot adapt to such heterogeneity, which may lead to degradation of transmission performance and system instability. In addition, transmission of complex transmission tasks usually requires high bandwidth, low latency and high throughput. The fixed transmission sequence and clock synchronization mechanism of traditional synchronous communication may fail to meet these requirements, thereby limiting the real-time performance and efficiency of transmission tasks.

[0148] In this embodiment, the number of bits and direction of movement of the detection window are determined based on the positional difference of the preset shared subgraph in the decoded image and the preset shared image. Then, the detection window is moved according to the number of bits and direction of movement to determine the synchronization position in the data sequence, thereby achieving semantic synchronization. This avoids dependence on strict clock synchronization and deterministic transmission order, thus avoiding the impact of transmission delay and bandwidth heterogeneity in large-scale heterogeneous networks. It improves transmission performance and stability in large-scale heterogeneous networks, while also improving the real-time performance and efficiency of transmission tasks.

[0149] The semantic synchronization method based on fixed images described in this invention will be explained in detail below with reference to several examples:

[0150] Example 1: The shape of the first semantic feature matrix obtained after encoding the shared image is preset to [1, C, H, W]. Here, C represents the number of channels; H represents the number of pixels in the vertical direction; and W represents the number of pixels in the horizontal direction. The preset shared image is as follows: Figure 3 The image shown has a resolution of 512×512. The sending end encodes this preset shared image to obtain a first semantic feature matrix. The shape of the first semantic feature matrix is ​​[1,32,32,32], and its size is 1×32×32×32. At this time, C is 32, H is 32, and W is 32. The first semantic feature matrix is ​​expanded into a first vector sequence according to a first expansion strategy, and then concatenated with the second vector sequence corresponding to the received transmission data before being sent to the receiving end. The initial position of the detection window is 20 bits ahead of the synchronization position (less than C).

[0151] The receiving end uses a right-sliding detection window to send the data within the window to a preset decoder for decoding, obtaining a decoded image. At this point, the window length is 32768. The decoded image is then fed into a preset neural network classification model for classification. If the decoded image is classified as noise, the sliding window continues until the preset neural network classification model determines that the decoded image contains part of the content in the preset shared image, and records the detection window's position as 20. At this point, the decoder outputs the decoded image as shown below. Figure 4 As shown.

[0152] Four preset shared sub-images are extracted from the preset shared image, namely sub-image A, sub-image B, sub-image C and sub-image D, with corresponding top-left corner icons of (128, 128), (128, 256), (256, 128), and (256, 256), respectively, with a length and width of 128.

[0153] Sub-images A, B, C, and D are used as targets for template matching with the decoded image. If the positions of sub-images A, B, C, and D in the decoded image are found to be the same as their positions in the preset shared image, then synchronization is complete. The index of the synchronization position is 20, and the data content corresponding to the data to be transmitted starts from the 21st position.

[0154] Example 2: The first semantic feature matrix obtained after encoding the shared image is pre-defined as having the shape [1, C, H, W]. Here, C represents the number of channels; H represents the number of symbols in the vertical direction; and W represents the number of symbols in the horizontal direction. The pre-defined shared image is as follows: Figure 3 The image shown has a resolution of 512×512. The sending end encodes this preset shared image to obtain a first semantic feature matrix. The shape of the first semantic feature matrix is ​​[1,32,32,32], and its size is 1×32×32×32. At this time, C is 32, H is 32, and W is 32. The first semantic feature matrix is ​​expanded into a first vector sequence according to a first expansion strategy, and then concatenated with the second vector sequence corresponding to the received transmission data before being sent to the receiving end. The initial position of the detection window is 310 bits ahead of the synchronization position (less than C×H).

[0155] The receiving end uses a right-sliding detection window to send the data within the window to a preset decoder for decoding, obtaining a decoded image. At this point, the window length is 32768. The decoded image is then fed into a preset neural network classification model for classification. If the decoded image is classified as noise, the sliding window continues until the preset neural network classification model determines that the decoded image contains part of the content in the preset shared image, and records the position of the detection window as 22. At this point, the decoder outputs the decoded image as follows: Figure 5 As shown.

[0156] Four preset shared sub-images are extracted from the preset shared image, namely sub-image A, sub-image B, sub-image C and sub-image D, with corresponding top-left corner icons of (128, 128), (128, 256), (256, 128), and (256, 256), respectively, with a length and width of 128.

[0157] Sub-images A, B, C, and D were used as targets for template matching against the decoded image. After matching, the positions of sub-images A and C in the decoded image were found to be (128, 272) and (256, 272), respectively, while the positions of sub-images B and D were (262, 306) and (260, 384), respectively. This indicates that sub-images B and D have shifted. Using sub-images A and C as references, coordinate comparison shows that there was no vertical shift, but a rightward shift of 272 - 128 = 144 pixels.

[0158] In this embodiment, by encoding the preset shared image of 3×512×512 and expanding it into a one-dimensional vector of 1×32×32×32, it can be seen that whenever the vector shifts by C bits, the image moves by 512 / 32 = 16 pixels. Therefore, the detection window needs to be shifted to the right by 144 / 16×C = 288 bits. From the initial position to the synchronization position, it needs to be shifted to the right by 22 + 288 = 310 bits to achieve synchronization. At this time, the data content corresponding to the data to be transmitted starts from the 311th bit.

[0159] Example 3: The shape of the first semantic feature matrix obtained after encoding the shared image is preset to [1, C, H, W]. Where C represents the number of channels; H represents the number of symbols in the vertical direction; and W represents the number of symbols in the horizontal direction. The preset shared image is as follows: Figure 3 The image shown has a resolution of 512×512. The sending end encodes this preset shared image to obtain a first semantic feature matrix. The shape of the first semantic feature matrix is ​​[1,32,32,32], and its size is 1×32×32×32. At this time, C is 32, H is 32, and W is 32. The first semantic feature matrix is ​​expanded into a first vector sequence according to a first expansion strategy, and then concatenated with the second vector sequence corresponding to the data to be transmitted before being sent to the receiving end. The initial position of the detection window is 3072 bits ahead of the synchronization position (an integer multiple of C×H).

[0160] The receiving end uses a right-sliding detection window to send the data within the window to a preset decoder for decoding, obtaining a decoded image. At this point, the window length is 32768. The decoded image is then fed into a preset neural network classification model for classification. If the decoded image is classified as noise, the sliding window continues until the preset neural network classification model determines that the decoded image contains part of the content in the preset shared image, and records the position of the detection window movement as 0. At this point, the decoder outputs the decoded image as shown below. Figure 6 As shown.

[0161] Four preset shared sub-images are extracted from the preset shared image, namely sub-image A, sub-image B, sub-image C and sub-image D, with corresponding top-left corner icons of (128, 128), (128, 256), (256, 128), and (256, 256), respectively, with a length and width of 128.

[0162] Sub-images A, B, C, and D are used as targets for template matching with the decoded image.

[0163] After matching, it was found that the positions of sub-image A and sub-image C in the decoded image were (176, 128) and (304, 128) respectively, and the positions of sub-image B and sub-image D in the decoded image were (176, 256) and (304, 256) respectively. Comparing the coordinates, it can be seen that there was no left or right shift, but it was shifted downward by 176-128=48 pixels.

[0164] In this embodiment, by encoding the preset shared image of 3×512×512 and expanding it into a one-dimensional vector of 1×32×32×32, it can be seen that whenever the vector shifts by C bits, the image moves by 512 / 32 = 16 pixels. Therefore, the detection window needs to be shifted to the right by 48 / 16×C×H = 3072 bits. A shift of 3072 bits to the right is required to complete synchronization from the initial position to the synchronization position. At this point, the data content corresponding to the data to be transmitted begins from the 3073rd bit.

[0165] Example 4: The shape of the first semantic feature matrix obtained after encoding the shared image is preset to [1, C, H, W]. Where C represents the number of channels; H represents the number of symbols in the vertical direction; and W represents the number of symbols in the horizontal direction. The preset shared image is as follows: Figure 3 The image shown has a resolution of 512×512. The sending end encodes this preset shared image to obtain a first semantic feature matrix. The shape of the first semantic feature matrix is ​​[1,32,32,32], and its size is 1×32×32×32. At this time, C is 32, H is 32, and W is 32. The first semantic feature matrix is ​​expanded into a first vector sequence according to a first expansion strategy, and then concatenated with the second vector sequence corresponding to the received transmission data before being sent to the receiving end. The initial position of the detection window is 3382 bits ahead of the synchronization position.

[0166] The receiving end uses a right-sliding detection window to send the data within the window to a preset decoder for decoding, obtaining a decoded image. At this point, the window length is 32768. The decoded image is then fed into a preset neural network classification model for classification. If the decoded image is classified as noise, the sliding window continues until the preset neural network classification model determines that the decoded image contains part of the content in the preset shared image, and records the position of the detection window as 22. At this point, the decoder outputs the decoded image as follows: Figure 7 As shown.

[0167] Four preset shared sub-images are extracted from the preset shared image, namely sub-image A, sub-image B, sub-image C and sub-image D, with corresponding top-left corner icons of (128, 128), (128, 256), (256, 128), and (256, 256), respectively, with a length and width of 128.

[0168] Sub-images A, B, C, and D were used as targets for template matching with the decoded image. After matching, it was found that sub-images A and C were located at (176, 272) and (304, 272) in the decoded image, respectively, while sub-images B and D were located at (310, 306) and (308, 384) in the decoded image, respectively. This indicates that sub-images B and D have shifted. Using sub-images A and C as reference coordinates, they have shifted downwards by 176 - 128 = 48 pixels and to the right by 272 - 128 = 144 pixels.

[0169] In this embodiment, by encoding the preset shared image of 3×512×512 and expanding it into a one-dimensional vector of 1×32×32×32, it can be seen that whenever the vector shifts by C bits, the image moves by 512 / 32 = 16 pixels. Therefore, the number of bits that the detection window needs to shift to the right is 48 / 16×C×H + 144 / 16×C = 3360. From the initial position to the synchronization position, it needs to shift to the right by 3360 + 22 = 3382 bits to achieve synchronization. At this point, the data content corresponding to the data to be transmitted starts from the 3383rd bit.

[0170] The semantic synchronization method based on fixed images provided in this embodiment obtains a preset shared image and data to be sent at the sending end; encodes and expands the preset shared image and data to be sent respectively to obtain a first vector sequence and a second vector sequence; concatenates the second vector sequence after the first vector sequence to obtain a data sequence; sends the data sequence to the receiving end based on the communication channel; the receiving end decodes the data sequence according to the window length of the detection window; when the decoded image includes part of the image content in the preset shared image, the position of the preset shared sub-image in the decoded image is compared with its position in the preset shared image to determine the number of bits and the direction of movement of the detection window, thereby determining the synchronization position in the data sequence. This system addresses the instability of synchronization caused by interference and clock drift during transmission. By determining the positional differences of partial image content within the decoded image and the preset shared image, the system identifies the number of bits and direction of the detection window movement, thus determining the synchronization position. This eliminates the need for a synchronization clock, avoiding the impact of clock drift and improving synchronization stability. Furthermore, when the decoded image includes partial content from the preset shared image, it is categorized as part of the preset shared image and its position is compared. Conversely, when the decoded image does not include partial content from the preset shared image, it is categorized as noise. This avoids the influence of noise and multipath effects on the preset shared image, further enhancing synchronization stability. Therefore, it prevents data transmission errors or loss, increasing the success rate of data transmission.

[0171] Furthermore, based on the positional differences of partial image content in the decoded image and the preset shared image, the number of bits and direction of movement of the detection window are determined. Then, the detection window is moved according to the number of bits and direction of movement to determine the synchronization position in the data sequence, achieving semantic synchronization. This avoids dependence on strict clock synchronization and deterministic transmission order, thereby avoiding the impact of transmission delay and bandwidth heterogeneity in large-scale heterogeneous networks. It improves transmission performance and stability in large-scale heterogeneous networks, thus meeting the requirements of high bandwidth, low latency, and high throughput for complex transmission tasks, and improving the real-time performance and efficiency of transmission tasks.

[0172] Figure 8 This is a block diagram of a semantic synchronization device based on a fixed image provided in one embodiment of this application. In this embodiment, the device is applied to a transmitting end, and the device includes at least the following modules: a data acquisition module 810, a data encoding module 820, a first expansion module 830, a second expansion module 840, a sequence splicing module 850, and a sequence sending module 860.

[0173] The data acquisition module 810 is used to acquire preset shared images and data to be sent.

[0174] The data encoding module 820 is used to encode the preset shared image and the data to be sent by the encoder respectively, so as to obtain the first feature matrix corresponding to the preset shared image and the second feature matrix corresponding to the data to be sent.

[0175] The first expansion module 830 is used to expand the first feature matrix into a one-dimensional vector according to the first expansion strategy to obtain the first vector sequence.

[0176] The second expansion module 840 is used to expand the second feature matrix into a one-dimensional vector according to the second expansion strategy to obtain the second vector sequence; the first expansion strategy is different from the second expansion strategy.

[0177] The sequence concatenation module 850 is used to concatenate the second vector sequence after the first vector sequence to obtain a data sequence.

[0178] The sequence transmission module 860 is used to transmit a data sequence to the receiving end based on the communication channel.

[0179] For relevant details, please refer to the above methods and system implementation examples.

[0180] Figure 9 This is a block diagram of a semantic synchronization device based on a fixed image provided in one embodiment of this application. In this embodiment, the device is applied to a receiving end, and the device includes at least the following modules: a sequence receiving module 910, an image decoding module 920, an image comparison module 930, a position difference module 940, a window moving module 950, and a position determination module 960.

[0181] The sequence receiving module 910 is used to receive data sequences based on the communication channel between the transmitter and the receiver.

[0182] The image decoding module 920 is used to decode the data sequence according to the window length of the detection window to obtain a decoded image.

[0183] Image comparison module 930 is used to classify decoded images with preset shared images to obtain classification results.

[0184] The position difference module 940 is used to compare the position of the preset shared sub-image in the decoded image with the position of the preset shared sub-image in the preset shared image when the classification result indicates that the decoded image includes part of the image content in the preset shared image, and obtain the position difference.

[0185] The window movement module 950 is used to determine the number of bits and the direction of movement of the detection window based on the position difference.

[0186] The position determination module 960 is used to move the detection window according to the number of moves and the direction of movement to determine the synchronization position in the data sequence.

[0187] For relevant details, please refer to the above methods and system implementation examples.

[0188] It should be noted that the semantic synchronization device based on fixed images provided in the above embodiments is only illustrated by the division of the functional modules described above when performing semantic synchronization based on fixed images. In practical applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the semantic synchronization device based on fixed images can be divided into different functional modules to complete all or part of the functions described above. In addition, the semantic synchronization device based on fixed images provided in the above embodiments and the semantic synchronization method embodiments based on fixed images belong to the same concept, and the specific implementation process is detailed in the method embodiments, which will not be repeated here.

[0189] This embodiment provides a semantic synchronization based on a fixed image, such as... Figure 10 As shown, the device includes at least a processor 1010 and a memory 1020.

[0190] Processor 1010 may include one or more processing cores, such as a quad-core processor, an octa-core processor, etc. Processor 1010 may be implemented using at least one hardware form selected from DSP (Digital Signal Processing), FPGA (Field-Programmable Gate Array), and PLA (Programmable Logic Array). Processor 1010 may also include a main processor and a coprocessor. The main processor, also known as a CPU (Central Processing Unit), is used to process data in the wake-up state; the coprocessor is a low-power processor used to process data in the standby state. In some embodiments, processor 1010 may integrate a GPU (Graphics Processing Unit), which is responsible for rendering and drawing the content to be displayed on the screen. In some embodiments, processor 1010 may also include an AI (Artificial Intelligence) processor, which is used to handle computational operations related to machine learning.

[0191] The memory 1020 may include one or more computer-readable storage media, which may be non-transitory. The memory 1020 may also include high-speed random access memory and non-volatile memory, such as one or more disk storage devices or flash memory devices. In some embodiments, the non-transitory computer-readable storage media in the memory 1020 are used to store at least one instruction, which is executed by the processor 1010 to implement the semantic synchronization method based on a fixed image provided in the method embodiments of this application.

[0192] In some embodiments, the device may also optionally include a peripheral device interface and at least one peripheral device. The processor 1010, memory 1020, and peripheral device interface can be connected via a bus or signal lines. Each peripheral device can be connected to the peripheral device interface via a bus, signal lines, or a circuit board. Indicatively, peripheral devices include, but are not limited to, radio frequency circuits, touch displays, audio circuits, and power supplies.

[0193] Of course, the semantic synchronization device based on fixed images may also include fewer or more components, and this embodiment does not limit this.

[0194] Optionally, this application also provides a computer-readable storage medium storing a program that is loaded and executed by a processor to implement the fixed-image-based semantic synchronization method of the above-described method embodiments.

[0195] Those skilled in the art will understand that the exemplary components, systems, and methods described in conjunction with the embodiments disclosed herein can be implemented in hardware, software, or a combination of both. Whether implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this invention. When implemented in hardware, it can be, for example, electronic circuits, application-specific integrated circuits (ASICs), appropriate firmware, plug-ins, function cards, etc. When implemented in software, the elements of this invention are programs or code segments used to perform the desired tasks. The programs or code segments can be stored in a machine-readable medium or transmitted over a transmission medium or communication link via data signals carried in a carrier wave.

[0196] It should be clarified that the present invention is not limited to the specific configurations and processes described above and shown in the figures. For the sake of brevity, detailed descriptions of known methods are omitted here. In the above embodiments, several specific steps are described and shown as examples. However, the method process of the present invention is not limited to the specific steps described and shown. Those skilled in the art can make various changes, modifications, and additions, or change the order of steps, after understanding the spirit of the present invention.

[0197] In this invention, features described and / or illustrated for one embodiment may be used in the same or similar manner in one or more other embodiments, and / or combined with or in place of features of other embodiments.

[0198] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention. For those skilled in the art, various modifications and variations can be made to the embodiments of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A semantic synchronization method based on a fixed image, characterized in that, In the sending end, a communication channel is established between the sending end and the receiving end. The method includes the following steps: Retrieve preset shared images and data to be sent; The preset shared image and the data to be sent are encoded by an encoder to obtain a first feature matrix corresponding to the preset shared image and a second feature matrix corresponding to the data to be sent. The first feature matrix is ​​expanded into a one-dimensional vector according to the first expansion strategy to obtain the first vector sequence. The second feature matrix is ​​expanded into a one-dimensional vector according to the second expansion strategy to obtain the second vector sequence; the first expansion strategy is different from the second expansion strategy. The second vector sequence is concatenated to the first vector sequence to obtain the data sequence; The data sequence is sent to the receiving end via the communication channel, so that after receiving the data sequence, the receiving end decodes the data sequence according to the window length of the detection window using a decoder; if the decoded image includes part of the image content in the preset shared image, the position of the preset shared sub-image in the decoded image is compared with the position of the preset shared sub-image in the preset shared image to obtain the position difference; based on the position difference, the number of moves and the direction of movement of the detection window are determined; the detection window is moved according to the number of moves and the direction of movement to determine the synchronization position in the data sequence.

2. The semantic synchronization method based on a fixed image according to claim 1, characterized in that, The first expansion strategy is to expand sequentially according to the order of submatrices in the feature matrix, the row number order within each submatric, and the column number order within each submatric; The second expansion strategy includes either an expansion strategy based on row number order or an expansion strategy based on column number order.

3. The semantic synchronization method based on a fixed image according to claim 1, characterized in that, The data to be sent includes image data, voice data, text data, video data, point cloud data, or augmented reality data.

4. The semantic synchronization method based on a fixed image according to claim 3, characterized in that, The encoder includes a semantic encoder and / or a basic encoder, and the semantic encoder includes a first semantic encoder and a second semantic encoder; the model structures of the first semantic encoder and the second semantic encoder are different; The step of encoding the preset shared image and the data to be sent using an encoder includes: The preset shared image is encoded using a first semantic encoder to obtain the first feature matrix; the data to be sent is encoded using a second semantic encoder or a basic encoder to obtain the second feature matrix. or, The preset shared image is encoded using a second semantic encoder to obtain the first feature matrix; the data to be sent is encoded using either the first semantic encoder or a basic encoder to obtain the second feature matrix.

5. The semantic synchronization method based on a fixed image according to claim 1, characterized in that, The preset shared sub-image is an image cropped from the preset shared image, including rectangular images with the top left corner as the vertex, rectangular images with the top right corner as the vertex, rectangular images with the bottom left corner as the vertex, rectangular images with the bottom right corner as the vertex, or rectangular images of the central area in the preset shared image.

6. A semantic synchronization method based on a fixed image, characterized in that, For use in a receiving end, the method includes the following steps: The data sequence is received based on the communication channel between the sender and the receiver; the data sequence is obtained by the sender encoding a preset shared image and the data to be sent, expanding them into a first vector sequence and a second vector sequence, and then concatenating them. The data sequence is decoded by the decoder according to the window length of the detection window to obtain a decoded image; The decoded images are classified to obtain classification results; the classification results are used to indicate whether the decoded images include part of the image content in the preset shared images. If the classification result indicates that the decoded image includes part of the image content in the preset shared image, the position of the preset shared sub-image in the decoded image is compared with the position of the preset shared sub-image in the preset shared image to obtain the position difference; Based on the positional differences, the number of positions and the direction of movement of the detection window are determined; The detection window is moved according to the specified number of moves and the specified direction of movement to determine the synchronization position in the data sequence.

7. The semantic synchronization method based on a fixed image according to claim 6, characterized in that, If the decoded image does not include some image content from the preset shared image, the method further includes: The detection window is moved according to a preset number of moves and a preset moving direction; The step of decoding the data sequence using a decoder according to the window length of the detection window to obtain a decoded image is performed.

8. The semantic synchronization method based on a fixed image according to claim 6, characterized in that, The step of classifying the decoded image to obtain the classification result includes: The decoded image is classified using a preset neural network classification model to obtain the classification result; or, The decoded image is classified using a preset edge detection algorithm to obtain the classification result; or, The classification result is obtained by calculating the peak signal-to-noise ratio between the decoded image and the preset shared image; or, The classification result is obtained by calculating the structural similarity index between the decoded image and the preset shared image.

9. A semantic synchronization device based on a fixed image, comprising a processor and a memory, characterized in that, The memory stores computer instructions, and the processor executes the computer instructions stored in the memory. When the computer instructions are executed by the processor, the device implements the steps of the method as described in any one of claims 1 to 8.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the steps of the method as described in any one of claims 1 to 8.

Images