Two-way signal synchronization method and device based on semantic heterogeneous model

Abstract

The invention relates to the technical field of communication, in particular to a two-way signal synchronization method and device based on a semantic heterogeneous model. The specific implementation scheme is as follows: a sending end encodes a shared image by using a first semantic encoding model and a second semantic encoding model to obtain a first semantic vector and a second semantic vector; first transmission information obtained by combining the first semantic vector and the first service information and second transmission information obtained by combining the second semantic vector and the second service information are sent to a receiving end through a channel; the receiving end decodes the received semantic information by using the first semantic decoding model and the second semantic decoding model to obtain a first decoded image and a second decoded image; and determining the synchronization position of the first transmission information and the second transmission information according to the similarity between the first decoded image and the shared image and the similarity between the second decoded image and the shared image. The method is wider in application scene, the robustness of the semantic synchronization method can be improved, and the probability of missing detection and error detection of synchronization is further reduced.

Description

Technical Field

[0001] The present disclosure relates to the field of communication technologies, and in particular, to a dual-channel signal synchronization method and apparatus based on a semantic heterogeneous model. Background Art

[0002] In modern communication systems, the synchronous transmission of data is of crucial importance. Synchronous transmission refers to maintaining the timing consistency of data between the sending end and the receiving end to ensure that the data can be correctly decoded and processed. In some application fields, such as image transmission, video stream transmission, and audio transmission, synchronous transmission is crucial for maintaining the integrity and quality of data.

[0003] Traditional data synchronous transmission methods usually rely on inserting specific synchronization signals or markers into the data stream to achieve synchronization. However, these methods have some limitations: Firstly, the insertion and detection of synchronization signals may be affected by transmission noise, interference, or channel instability at low signal-to-noise ratios, resulting in synchronization errors. In addition, different types of data usually require different synchronization methods, which is a relatively complex process and difficult to adapt to various different application scenarios. Summary of the Invention

[0004] The present disclosure provides a dual-channel signal synchronization method, apparatus, electronic device, and storage medium based on a semantic heterogeneous model.

[0005] According to a first aspect of the present disclosure, there is provided a dual-channel signal synchronization method based on a semantic heterogeneous model, including:

[0006] The sending end encodes a shared image using a first semantic encoding model to obtain a first semantic vector, and encodes the shared image using a second semantic encoding model to obtain a second semantic vector;

[0007] The sending end combines the first semantic vector and first service information to obtain first transmission information, and combines the second semantic vector and second service information to obtain second transmission information;

[0008] The sending end sends the first transmission information and the second transmission information to the receiving end through a channel; wherein, the first transmission information and the second transmission information are provided to the receiving end to decode a first decoded image and a second decoded image through a first semantic decoding model and a second semantic decoding model, and determine the synchronization positions of the first transmission information and the second transmission information according to the similarities between the first decoded image, the second decoded image and the shared image.

[0009] According to a second aspect of the present disclosure, there is provided a dual-channel signal synchronization method based on a semantic heterogeneous model, including:

[0010] The receiving end receives the first transmission information and the second transmission information; wherein, the first transmission information is obtained by the sending end encoding a shared image using a first semantic encoding model to obtain a first semantic vector and combining the first semantic vector and first service information; the second transmission information is obtained by the sending end encoding the shared image using a second semantic encoding model to obtain a second semantic vector and combining the second semantic vector and second service information;

[0011] The receiving end decodes the first transmission information using a first semantic decoding model to obtain a first decoded image, and decodes the second transmission information using a second semantic decoding model to obtain a second decoded image;

[0012] The receiving end determines the synchronization positions of the first transmission information and the second transmission information according to the similarities between the first decoded image, the second decoded image and the shared image.

[0013] According to a third aspect of the present disclosure, there is provided a dual-channel signal synchronization device based on a semantic heterogeneous model, including:

[0014] An encoding module configured to encode a shared image using a first semantic encoding model to obtain a first semantic vector, and encode the shared image using a second semantic encoding model to obtain a second semantic vector;

[0015] An information combining module configured to combine the first semantic vector and first service information to obtain first transmission information, and combine the second semantic vector and second service information to obtain second transmission information;

[0016] An information sending module configured to send the first transmission information and the second transmission information to a receiving end through a channel; wherein, the first transmission information and the second transmission information are used to be provided to the receiving end to decode a first decoded image and a second decoded image through a first semantic decoding model and a second semantic decoding model, and determine the synchronization positions of the first transmission information and the second transmission information according to the similarities between the first decoded image, the second decoded image and the shared image.

[0017] According to a fourth aspect of the present disclosure, there is provided a dual-channel signal synchronization device based on a semantic heterogeneous model, including:

[0018] A receiving module, configured to receive first transmission information and second transmission information; wherein, the first transmission information is obtained by a sending end encoding a shared image using a first semantic encoding model to obtain a first semantic vector, and combining the first semantic vector and first service information; the second transmission information is obtained by the sending end encoding the shared image using a second semantic encoding model to obtain a second semantic vector, and combining the second semantic vector and second service information;

[0019] A decoding module, configured to decode the first transmission information using a first semantic decoding model to obtain a first decoded image, and decode the second transmission information using a second semantic decoding model to obtain a second decoded image;

[0020] A synchronization module, configured to determine a synchronization position of the first transmission information and the second transmission information according to similarities between the first decoded image, the second decoded image and the shared image.

[0021] According to a fifth aspect of the present disclosure, there is provided an electronic device, including:

[0022] At least one processor; and

[0023] A memory communicatively connected to the at least one processor; wherein,

[0024] The memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the method described in any one of the above technical solutions.

[0025] According to a sixth aspect of the present disclosure, there is provided a non-transitory computer-readable storage medium storing computer instructions, wherein the computer instructions are used to cause a computer to execute the method described in any one of the above technical solutions.

[0026] According to a seventh aspect of the present disclosure, there is provided a computer program product, including a computer program which, when executed by a processor, implements the method described in any one of the above technical solutions.

[0027] The present disclosure provides a dual-channel signal synchronization method and apparatus based on a semantic heterogeneous model. The synchronization method is applicable to a wider range of scenarios, can improve the robustness of the semantic synchronization method, and further reduce the probability of missed detection and misdetection in synchronization.

[0028] It should be understood that the content described in this part is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it used to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] The accompanying drawings are used to better understand the present solution and do not constitute a limitation to the present disclosure. Among them:

[0030] Figure 1 is a schematic diagram of the steps of the dual-channel signal synchronization method based on a semantic heterogeneous model for the sending end in the embodiments of the present disclosure;

[0031] Figure 2 is a schematic flowchart of the dual-channel signal synchronization method based on a semantic heterogeneous model in the embodiments of the present disclosure;

[0032] Figure 3 is a schematic diagram of the principle of converting a semantic vector into a one-dimensional vector in the embodiments of the present disclosure;

[0033] Figure 4 is a schematic diagram of the steps of the dual-channel signal synchronization method based on a semantic heterogeneous model for the receiving end in the embodiments of the present disclosure;

[0034] Figure 5 is a principle block diagram of the dual-channel signal synchronization device based on a semantic heterogeneous model for the sending end in the embodiments of the present disclosure;

[0035] Figure 6 is a principle block diagram of the dual-channel signal synchronization device based on a semantic heterogeneous model for the receiving end in the embodiments of the present disclosure;

[0036] Figure 7 is a block diagram of an electronic device for implementing the dual-channel signal synchronization method based on a semantic heterogeneous model in the embodiments of the present disclosure. Detailed implementation manners

[0037] The following makes an explanation of the exemplary embodiments of the present disclosure with reference to the accompanying drawings. Various details of the embodiments of the present disclosure are included to facilitate understanding, and they should be considered merely exemplary. Therefore, those of ordinary skill in the art should recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Similarly, for the sake of clarity and conciseness, the description of well-known functions and structures is omitted below.

[0038] Sending end and receiving end: There is a channel connection between the sending end and the receiving end. The sending end (transmitter) has an encoder, and the receiving end (receiver) has a decoder. The present disclosure is based on semantic encoding and semantic decoding. Therefore, the sending end of the present disclosure has a semantic encoding model; the receiving end of the present disclosure has a semantic decoding model. The underlying layers of the sending end and the receiving end of the present disclosure are still the Shannon physical layer.

[0039] Business information: Information involved in information service. Depending on the type of information, it can be further divided into text business information, image business information, voice business information, video business information, point cloud business information, etc. It can also be divided into structured information and unstructured information. It can also be all possible types of business information for communication transmission.

[0040] Artificial intelligence model: including artificial intelligence semantic encoding model and artificial intelligence semantic decoding model. The artificial intelligence semantic encoding model is used to encode the business information from the information source into semantic information. The semantic information is transmitted over the channel. The artificial intelligence semantic decoding model is used to decode the semantic information transmitted over the channel into business information. Depending on the type of semantics, it can be divided into text artificial intelligence model, audio artificial intelligence model, image artificial intelligence model, video artificial intelligence model, point cloud artificial intelligence model, one-dimensional waveform model, radar data model, etc. The category of the artificial intelligence model is related to the category of semantic encoding and the category of business information, and can be all possible types of models for communication transmission.

[0041] Semantic encoding: The semantic information contained in business information. Depending on the type of semantics, it can be further divided into text semantic encoding, image semantic encoding, audio semantic encoding, video semantic encoding, point cloud semantic encoding, etc. The possible categories of semantic encoding are related to the possible categories of business information, and can be all possible types of semantic encoding for communication transmission. The generation of semantic encoding is also related to the semantic encoding model, and the semantic encoding model can adopt models from all possible disciplines such as artificial intelligence, deep learning, and pattern recognition.

[0042] In view of the technical problems that the traditional data synchronization method may be affected by transmission noise, interference or channel instability in the case of low signal-to-noise ratio, resulting in synchronization errors; and different types of data usually require different synchronization methods, which are difficult to adapt to different application scenarios, the present disclosure provides a dual-channel signal synchronization method based on a semantic heterogeneous model, as Figure 1 shown, including:

[0043] Step S101, the sending end encodes the shared image using the first semantic encoding model to obtain a first semantic vector, and encodes the shared image using the second semantic encoding model to obtain a second semantic vector;

[0044] Step S102, the sending end combines the first semantic vector and the first business information to obtain a first transmission information, and combines the second semantic vector and the second business information to obtain a second transmission information;

[0045] Step S103, the sending end sends the first transmission information and the second transmission information to the receiving end through a channel; wherein, the first transmission information and the second transmission information are used to be provided to the receiving end, and the first decoded image and the second decoded image are obtained by decoding through the first semantic decoding model and the second semantic decoding model, and the synchronization positions of the first transmission information and the second transmission information are determined according to the similarities between the first decoded image, the second decoded image and the shared image.

[0046] Specifically, as Figure 2 shown, the same shared image 107 can be pre-stored in the sending end 100 and the receiving end 200. The sending end pre-sets two heterogeneous semantic encoding models (the first semantic encoding model 101, the second semantic encoding model 102). Among them, the number of channels of the semantic vector encoded by the first semantic encoding model 101 is C1, and the number of channels of the semantic vector encoded by the second semantic encoding model 102 is C2. The shared image 107 is semantically encoded through the first semantic encoding model 101 and the second semantic encoding model 102 respectively. The first semantic vector obtained by semantic encoding is merged with the first service information to be transmitted, and the second semantic vector obtained by semantic encoding is merged with the second service information to be transmitted. Two signals, namely the first transmission information and the second transmission information, are sent to the receiving end through the channel. It should be noted that the first service information and the second service information can be data of the same type or different types, and can be any one or more types among images, voices, texts, videos, point clouds, etc.

[0047] In this embodiment, two heterogeneous semantic encoding models are used to encode the shared image respectively, and the first semantic vector obtained by encoding is merged with the first service information, and the second semantic vector obtained by encoding is merged with the second service information, that is, the shared image and the service information to be transmitted are sent to the receiving end together. As Figure 2 shown, the sending end 100 includes a first semantic encoding model 101, a second semantic encoding model 102, a first encoder 103, and a second encoder 104. The first service information can be obtained by encoding the first data to be transmitted 105 using the first encoder 103, and the second service information can be obtained by encoding the second data to be transmitted 106 using the second encoder 104. Among them, the first encoder 103 and the second encoder 104 can use semantic encoders or basic encoders, that is, the data to be transmitted can be encoded by semantic encoders or basic encoders, and is not limited to semantic encoders.

[0048] Furthermore, as Figure 2As shown, after the receiving end 200 receives the first transmission information 108 and the second transmission information 109, it decodes them through two heterogeneous first semantic decoding models 201 and second semantic decoding models 202 respectively, and determines the synchronization position of the data according to the similarity between the decoded images of the two semantic decoding models and the shared image 107, so as to achieve the synchronization of different types of data. The synchronization method is applicable to a wider range of scenarios, which can improve the robustness of the semantic synchronization method and further reduce the probability of missed detection and misdetection in synchronization.

[0049] As an optional implementation manner, the first semantic encoding model and the second semantic encoding model are heterogeneous encoding models; the first semantic decoding model and the second semantic decoding model are heterogeneous decoding models; the first semantic encoding model corresponds to the first semantic decoding model, and the second semantic encoding model corresponds to the second semantic decoding model.

[0050] Specifically, the semantic heterogeneous model means that the structures of the two models are different and the output vector dimensions are also different. For example, the output dimension after encoding by model A is [1, 4, 16, 16], and the output dimension after encoding by model B is [1, 16, 16, 16].

[0051] As an optional implementation manner, in step S102, the sending end combines the first semantic vector and the first service information to obtain the first transmission information, and combines the second semantic vector and the second service information to obtain the second transmission information, including:

[0052] Convert the first semantic vector into a one-dimensional vector, and connect it as the synchronization header of the first transmission information to the first service information to obtain the first transmission information.

[0053] Convert the second semantic vector into a one-dimensional vector, and connect it as the synchronization header of the second transmission information to the second service information to obtain the second transmission information.

[0054] Exemplarily, in step S101, as Figure 2 shown, the sending end 100 can encode the shared image 103 of 512×512 using the first semantic encoding model 101 and the second semantic encoding model 102. The size of the first semantic vector generated by the first semantic encoding model 101 is 1×9×32×32, and the size of the second semantic vector generated by the second semantic encoding model is 1×7×32×32. As Figure 3As shown in the figure, the encoded first semantic vector and second semantic vector are unfolded into one-dimensional vectors in the channel direction. These two vectors are unfolded into one-dimensional vectors in the channel direction and used as the synchronization headers of two sequences respectively. At this time, the lengths of the synchronization headers of the two sequences are 9216 and 7168 respectively, and then two paths of data to be transmitted are connected respectively. During the channel coherence time, these two paths of signals (the first transmission information and the second transmission information) are sent. At the receiving end, the sliding window detection method is used to process the two paths of signals: for each path of signal, the corresponding semantic decoding model is used to decode the data in the sliding window. As Figure 2 shown, the first semantic decoding model 201 is used to decode the sequence data (the first transmission information) in the first sliding window 203, and the second semantic decoding model 202 is used to decode the sequence data (the second transmission information) in the second sliding window 204. When the outputs of the two semantic decoding models both contain partial contents of the shared image, the synchronization position can be further determined. This method uses two heterogeneous semantic decoding models at the receiving end, and only when the outputs of the decoding models both contain the shared image can the synchronization position be determined, which improves the accuracy of data synchronization and reduces the probability of missed detection and misdetection of synchronization.

[0055] As an optional implementation manner, in step S103, the sending end sends the first transmission information and the second transmission information to the receiving end through a channel, including:

[0056] During the coherence time of the channel, the first transmission information and the second transmission information are sent to the receiving end.

[0057] Specifically, the coherence time refers to the maximum time difference range during which the channel remains constant. The same signal of the sending end arrives at the receiving end within the coherence time, and the fading characteristics of the signal are completely similar. It can be considered that the sending end sends two paths of signals to the receiving end within the coherence time, and the misalignment of these two paths of signals is the same. The channel coherence time is the time during which the channel state remains unchanged. Two paths of signals are sent within the channel coherence time to ensure that the synchronization misalignment of the two paths of signals is the same.

[0058] The present disclosure also provides another method for synchronizing two paths of signals based on a semantic heterogeneous model, as Figure 4 shown, including:

[0059] Step S104, the receiving end receives the first transmission information and the second transmission information; wherein, the first transmission information is obtained by the sending end encoding the shared image using the first semantic encoding model to obtain a first semantic vector and combining the first semantic vector with the first service information; the second transmission information is obtained by the sending end encoding the shared image using the second semantic encoding model to obtain a second semantic vector and combining the second semantic vector with the second service information;

[0060] Step S105: The receiving end decodes the first transmission information by using the first semantic decoding model to obtain a first decoded image, and decodes the second transmission information by using the second semantic decoding model to obtain a second decoded image;

[0061] Step S106: The receiving end determines the synchronization positions of the first transmission information and the second transmission information according to the similarities between the first decoded image, the second decoded image and the shared image.

[0062] As Figure 2 shown, after receiving the first transmission information 108 and the second transmission information 109, the receiving end 200 decodes them respectively through two heterogeneous first semantic decoding models 201 and second semantic decoding models 202, and determines the synchronization positions of the data according to the similarities between the first decoded image, the second decoded image obtained by decoding through the two semantic decoding models and the shared image 107, so as to realize the synchronization of different types of data. The applicable scenario of this synchronization method is wider, which can improve the robustness of the semantic synchronization method and further reduce the probability of missed detection and misdetection in synchronization.

[0063] As an optional implementation manner, before step S106 where the receiving end determines the synchronization positions of the first transmission information and the second transmission information according to the similarities between the first decoded image, the first decoded image and the shared image, it further includes:

[0064] Judging whether at least part of the content of the shared image is included in both the first decoded image and the second decoded image;

[0065] In response to that at least part of the content of the shared image is included in both the first decoded image and the second decoded image, in step S106, the receiving end determines the synchronization positions of the first transmission information and the second transmission information according to the similarities between the first decoded image, the second decoded image and the shared image;

[0066] In response to that at least part of the content of the shared image is not included in both the first decoded image and the second decoded image, return to step S105, and the receiving end decodes the first transmission information by using the first semantic decoding model to obtain a first decoded image, and decodes the second transmission information by using the second semantic decoding model to obtain a second decoded image.

[0067] Specifically, during the decoding process, when both semantic decoding models have decoded partial content of the shared image, record the current decoding position, the n positions forward from the current position, and the n positions backward from the current position. The decoded image can be compared with the shared image, and calculate the SSIM (Structural Similarity, a metric for measuring the similarity between two images) between the first decoded image, the second decoded image, and the shared image. The SSIM between the first decoded image and the shared image, and the SSIM between the second decoded image and the shared image are both greater than a preset threshold (for example, the preset threshold is 0.6), and the position at this time is the synchronization position.

[0068] As an alternative implementation, as Figure 2 shown, the receiving end 200 uses the first semantic decoding model 201 to decode the first transmission information to obtain the first decoded image, and uses the second semantic decoding model to decode the second transmission information to obtain the second decoded image, including:

[0069] Decode the first transmission information through the first sliding window 203 and the first semantic decoding model 201 to obtain the first decoded image; during the decoding process, the first sliding window 203 slides to the right from the starting position of the sequence corresponding to the first transmission information, and the first semantic decoding model 201 decodes the data within the first sliding window 203 to obtain the first decoded image.

[0070] Decode the second transmission information through the second sliding window 204 and the second semantic decoding model 202 to obtain the second decoded image. During the decoding process, the second sliding window 204 slides to the right from the starting position of the sequence corresponding to the second transmission information, and the second semantic decoding model 202 decodes the data within the second sliding window 204 to obtain the second decoded image.

[0071] Specifically, as Figure 2 shown, the receiving end 200 can use the right-sliding window detection method to decode the two received signals (the first transmission information and the second transmission information), and simultaneously output the first decoded image 205 and the second decoded image 206 through the first semantic decoding model 201 and the second semantic decoding model 202.

[0072] As an alternative implementation, the receiving end determines the synchronization position of the first transmission information and the second transmission information according to the similarity between the first decoded image, the second decoded image, and the shared image, including:

[0073] Obtain the current positions of the first sliding window and the second sliding window, the n positions forward from the current position, and the n positions backward from the current position when the content of the shared image appears in both the first decoded image and the second decoded image; where n is equal to the product of the first channel number C1 corresponding to the first semantic vector and the second channel number C2 corresponding to the second semantic vector.

[0074] Compare the similarities between the first decoded image, the second decoded image and the shared image when the first sliding window and the second sliding window are at the current position, n positions forward and n positions backward, respectively.

[0075] Determine that the positions where the similarities between the first decoded image, the second decoded image and the shared image are greater than the preset similarity are synchronization positions.

[0076] Exemplarily, as Figure 2 shown, at the receiving end 200, a method of using a right sliding window detection for two signals is adopted. For each signal transmitted through the channel, the corresponding semantic decoding model is used to decode the data in the signal window, and the decoded image is sent into a classification network for classification. If the classification network classifies the decoded image as a noise map, continue sliding the window until the classification network believes that both decoded images contain parts of the shared image, and record the current moving position.

[0077] As Figure 2 shown, the first semantic decoding model 201 is used to decode the signal in the first sliding window 203, and the second semantic decoding model 202 is used to decode the signal in the second sliding window 204. The sliding window detection starts for both signals simultaneously. When sliding to 43, both signals are decoded to parts of the shared image, then record 43 as the current position, and record n positions forward and n positions backward (i.e., the number of channels C1 * C2). For example, if C1 = 9 and C2 = 7, then C1 * C2 = 63, and record the current position 43, the forward position -20 (43 minus 63 positions forward, 43 - 63 = -20), and the backward position 106 (43 plus 63 positions backward, 43 + 63 = 106).

[0078] Further, calculate the SSIM between the decoded image and the original shared image when the sliding window positions are at the current position 43, the forward position -20, and the backward position 106, respectively. The SSIM between the first decoded image and the shared image and the SSIM between the second decoded image and the shared image are both greater than the preset threshold (for example, the preset threshold is 0.6), and the position at this time is the synchronization position. For example, when the sliding window position is 20, the SSIMs between the first decoded image, the second decoded image and the shared image are all greater than 0.6, and the sliding window position 20 at this time is the synchronization position, and the two signals are synchronized according to this position. It should be noted that the present disclosure is not limited to calculating the SSIM between images, and other metrics such as PSNR (Peak Signal to Noise Ratio) can also be calculated.

[0079] In this embodiment, the value of n depends on the number of channels C1 of the first semantic decoding model and the number of channels C2 of the second semantic decoding model. The semantic encoding model for encoding the shared image consists of multiple two-dimensional convolutional layers. When performing repeated two-dimensional convolutional operations on the shared image, each convolutional layer generates a new set of semantic feature maps. These semantic feature maps are stacked in the channel direction to form a three-dimensional tensor, where each element represents a feature. Each convolutional kernel contains a set of trainable weight parameters in the channel direction. The convolutional kernel extracts the corresponding sequential feature information by convolving the input feature maps. Therefore, the vector group in the channel direction can be understood as a set of semantic features describing certain specific features in the input image.

[0080] Based on the above principle, when the vector after semantic encoding is unfolded in the channel direction, every C (representing the number of channels) symbols represent a set of semantic features. If the shift of the sequence is an integer multiple of C, the decoded result will contain part of the content of the shared image. If the shift of the sequence is not an integer multiple of C, the order of the semantic features corresponding to each feature after the shift will be disturbed. Since the semantic decoding model and the semantic encoding model share context knowledge, the semantic decoding model cannot correctly decode the semantic features with disturbed order. Therefore, two groups of vectors with the number of channels C1 and C2 in the encoding result are respectively used as the synchronization headers of the two paths of signals. At the receiving end, through the method of sliding window detection, two heterogeneous semantic decoding models are respectively used to decode the semantic information in the sliding window; when the decoded results both contain part of the content of the shared image, it may be at the synchronization position at this time, or the difference from the correct synchronization position is an integer multiple of C1*C2. Therefore, in this embodiment, the current position when the decoded image simultaneously shows part of the content of the shared image, the C1*C2 positions forward from the current position, and the C1*C2 positions backward from the current position are judged to avoid the order of the semantic features after decoding being disturbed.

[0081] The present disclosure also provides a dual-channel signal synchronization device 500 based on a semantic heterogeneous model, as Figure 5 shown, including:

[0082] An encoding module 501, configured to encode the shared image using the first semantic encoding model to obtain a first semantic vector, and encode the shared image using the second semantic encoding model to obtain a second semantic vector.

[0083] An information merging module 502, configured to merge the first semantic vector and the first service information to obtain first transmission information, and merge the second semantic vector and the second service information to obtain second transmission information.

[0084] An information sending module 503, configured to send first transmission information and second transmission information to a receiving end through a channel; wherein, the first transmission information and the second transmission information are used to be provided to the receiving end, and first decoded images and second decoded images are obtained by decoding through a first semantic decoding model and a second semantic decoding model, and the synchronization positions of the first transmission information and the second transmission information are determined according to the similarities between the first decoded images, the second decoded images and a shared image.

[0085] Specifically, as Figure 2 shown, the same shared image 107 can be pre-stored at a sending end 100 and a receiving end 200 in advance. Two heterogeneous semantic encoding models (a first semantic encoding model 101 and a second semantic encoding model 102) are pre-set at the sending end. Among them, the number of channels of the first semantic encoding model 101 is C1, and the number of channels of the second semantic encoding model 102 is C2. The shared image 107 is respectively subjected to semantic encoding through the first semantic encoding model 101 and the second semantic encoding model 102. The first semantic vector obtained by semantic encoding is merged with the first service information to be transmitted, and the second semantic vector obtained by semantic encoding is merged with the second service information to be transmitted. Two signals, namely first transmission information and second transmission information, are sent to the receiving end through the channel. It should be noted that the first service information and the second service information can be data of the same type or different types, and can be any one or more types among images, voices, texts, videos, point clouds, etc.

[0086] In this embodiment, two heterogeneous semantic encoding models are used to respectively encode the shared image, and the first semantic vector obtained by encoding is merged with the first service information, and the second semantic vector obtained by encoding is merged with the second service information, that is, the shared image and the service information to be transmitted are sent to the receiving end together. As Figure 2 shown, the sending end 100 includes a first semantic encoding model 101, a second semantic encoding model 102, a first encoder 103, and a second encoder 104. The first service information can be obtained by encoding first data to be transmitted 105 through the first encoder 103, and the second service information can be obtained by encoding second data to be transmitted 106 through the second encoder 104. Among them, the first encoder 103 and the second encoder 104 can use semantic encoders or basic encoders, that is, the data to be transmitted can be encoded through semantic encoders or basic encoders, and is not limited to semantic encoders.

[0087] Further, after receiving the first transmission information and the second transmission information, the receiving end decodes them through two heterogeneous first semantic decoding models and second semantic decoding models respectively, and determines the synchronization position of the data according to the similarity between the decoded images corresponding to the two semantic decoding models and the shared image, so as to realize the synchronization of different types of data. The method is applicable to a wider range of scenarios, can improve the robustness of the semantic synchronization method, and further reduce the probability of missed detection and misdetection in synchronization.

[0088] As an optional implementation manner, the first semantic encoding model and the second semantic encoding model are heterogeneous encoding models; the first semantic decoding model and the second semantic decoding model are heterogeneous decoding models; the first semantic encoding model corresponds to the first semantic decoding model, and the second semantic encoding model corresponds to the second semantic decoding model.

[0089] Specifically, the semantic heterogeneous model means that the structures of the two models are different and the output vector dimensions are also different. For example, the output dimension after encoding by model A is [1, 4, 16, 16], and the output dimension after encoding by model B is [1, 16, 16, 16]. The first semantic encoding model and the first semantic decoding model are paired so that the first semantic decoding model can decode the encoded content of the first semantic encoding model; the second semantic encoding model and the second semantic decoding model are paired so that the second semantic decoding model can decode the encoded content of the second semantic encoding model.

[0090] As an optional implementation manner, the information merging module 502 merging the first semantic vector and the first service information to obtain the first transmission information, and merging the second semantic vector and the second service information to obtain the second transmission information includes:

[0091] Convert the first semantic vector into a one-dimensional vector and connect it to the first service information as the synchronization header of the first transmission information to obtain the first transmission information.

[0092] Convert the second semantic vector into a one-dimensional vector and connect it to the second service information as the synchronization header of the second transmission information to obtain the second transmission information.

[0093] Exemplarily, as Figure 2 shown, the encoding module 501 can be set at the sending end 100. The encoding module 501 can use the first semantic encoding model 101 and the second semantic encoding model 102 to encode the shared image 103 of 512×512. The size of the first semantic vector generated by the first semantic encoding model 101 is 1×9×32×32, and the size of the second semantic vector generated by the second semantic encoding model is 1×7×32×32. As Figure 3As shown, the encoded first semantic vector and second semantic vector are unfolded into one-dimensional vectors in the channel direction. These two vectors are unfolded into one-dimensional vectors in the channel direction and used as the synchronization headers of two sequences respectively. At this time, the lengths of the synchronization headers of the two sequences are 9216 and 7168 respectively, and the two paths of data to be transmitted are connected behind. Within the channel coherence time, these two paths of signals (the first transmission information and the second transmission information) are sent. At the receiving end 200, the sliding window detection method is used to process the two paths of signals: for each path of signal, the corresponding semantic decoding model is used to decode the data in the sliding window. As Figure 2 shown, the first semantic decoding model 201 is used to decode the sequence data (the first transmission information) in the first sliding window 203, and the second semantic decoding model 202 is used to decode the sequence data (the second transmission information) in the second sliding window 204. When the outputs of the two semantic decoding models both contain partial contents of the shared image, the synchronization position can be further determined. This method uses two heterogeneous semantic decoding models at the receiving end, and only when the outputs of the decoding models both contain the shared image can the synchronization position be determined, improving the accuracy of data synchronization and reducing the probability of missed detection and misdetection in synchronization.

[0094] As an optional implementation manner, the information sending module 503 sends the first transmission information and the second transmission information to the receiving end through the channel, including:

[0095] Within the coherence time of the channel, the first transmission information and the second transmission information are sent to the receiving end.

[0096] Specifically, the coherence time refers to the maximum time difference range during which the channel remains constant. The same signal at the sending end arrives at the receiving end within the coherence time, and the fading characteristics of the signal are completely similar. It can be considered that the sending end sends two paths of signals to the receiving end within the coherence time, and the misalignment of these two paths of signals is the same. The channel coherence time is the time during which the channel state remains unchanged. Two paths of signals are sent within the channel coherence time to ensure that the synchronization misalignment of the two paths of signals is the same.

[0097] The present disclosure also provides another dual-channel signal synchronization device 600 based on a semantic heterogeneous model, as Figure 6 shown, including:

[0098] A receiving module 601, configured to receive the first transmission information and the second transmission information.

[0099] A decoding module 602, configured to decode the first transmission information using the first semantic decoding model to obtain a first decoded image, and decode the second transmission information using the second semantic decoding model to obtain a second decoded image.

[0100] The synchronization module 603 is configured to determine the synchronization positions of the first transmission information and the second transmission information according to the similarities between the first decoded image, the second decoded image, and the shared image.

[0101] As Figure 2 shown, after the receiving end 200 receives the first transmission information and the second transmission information sent by the sending end 100, it decodes them through two heterogeneous first semantic decoding models 201 and second semantic decoding models 202 respectively, and determines the synchronization positions of the data according to the similarities between the first decoded image, the second decoded image, and the shared image 107, so as to achieve the synchronization of different types of data. The method is applicable to a wider range of scenarios, can improve the robustness of the semantic synchronization method, and further reduce the probability of missed detection and false detection in synchronization.

[0102] As an optional implementation manner, the device 600 further includes:

[0103] A judgment module, configured to judge whether at least part of the content of the shared image is included in both the first decoded image and the second decoded image before the synchronization module 603 determines the synchronization positions of the first transmission information and the second transmission information according to the similarities between the first decoded image, the first decoded image, and the shared image.

[0104] In response to that at least part of the content of the shared image is included in both the first decoded image and the second decoded image, the receiving end determines the synchronization positions of the first transmission information and the second transmission information according to the similarities between the first decoded image, the second decoded image, and the shared image.

[0105] In response to that at least part of the content of the shared image is not included in both the first decoded image and the second decoded image, return that the receiving end decodes the first transmission information using the first semantic decoding model to obtain the first decoded image, and decodes the second transmission information using the second semantic decoding model to obtain the second decoded image.

[0106] Specifically, during the decoding process, when both semantic decoding models decode to obtain part of the content of the shared image, record the current position of the decoding, the n positions forward and the n positions backward from the current position. The decoded image can be compared with the shared image, and the SSIM (Structural Similarity, a metric for measuring the similarity between two images) between the first decoded image, the second decoded image, and the shared image is calculated. The SSIM between the first decoded image and the shared image, and the SSIM between the second decoded image and the shared image are both greater than a preset threshold (for example, the preset threshold is 0.6), and the position at this time is the synchronization position.

[0107] As an optional implementation manner, the decoding module 602 includes:

[0108] The first decoding unit is configured to decode the first transmission information through a first sliding window and a first semantic decoding model to obtain a first decoded image.

[0109] The second decoding unit is configured to decode the second transmission information through a second sliding window and a second semantic decoding model to obtain a second decoded image.

[0110] Specifically, as Figure 2 shown, the receiving end 200 can adopt a right-sliding window detection method to decode the two received signals (the first transmission information and the second transmission information), and simultaneously output a first decoded image 205 and a second decoded image 206 through the first semantic decoding model 201 and the second semantic decoding model 202.

[0111] As an optional implementation manner, the synchronization module 603 determines the synchronization positions of the first transmission information and the second transmission information according to the similarities between the first decoded image, the second decoded image and the shared image, including:

[0112] When the content of the shared image appears in both the first decoded image and the second decoded image, obtain the current positions corresponding to the first sliding window and the second sliding window, the first n positions forward from the current position, and the first n positions backward from the current position.

[0113] Compare the similarities between the first decoded image, the second decoded image and the shared image when the first sliding window and the second sliding window are at the current position, the first n positions forward, and the first n positions backward, respectively.

[0114] Determine the positions where the similarities between the first decoded image, the second decoded image and the shared image are greater than a preset similarity as the synchronization positions.

[0115] Exemplarily, as Figure 2 shown, at the receiving end 200, a right-sliding window detection method is used for the two signals. For each signal transmitted through the channel, the corresponding semantic decoding model is used to decode the data in the signal window, and the decoded images are sent into a classification network for classification. If the classification network classifies the decoded images as noise maps, continue to slide the window until the classification network believes that both decoded images contain some content in the shared image, and record the moving position at this time.

[0116] As Figure 2As shown, the first semantic decoding model 201 decodes the signals (first transmission information) in the first sliding window 203, and the second semantic decoding model 202 decodes the signals (second transmission information) in the second sliding window 204. The two-way signals start sliding window detection simultaneously. When sliding to 43, partial contents of the shared image are decoded for both signals. Then record 43 as the current position, and record n positions forward and n positions backward (i.e., the number of channels C1*C2). For example, if C1 = 9 and C2 = 7, then C1*C2 = 63. Then record the current position 43, the forward position -20 (43 - 63 = -20, 63 positions forward from 43), and the backward position 106 (43 + 63 = 106, 63 positions backward from 43).

[0117] Further, calculate the SSIM between the decoded images and the original shared image when the sliding window positions are at the current position 43, the forward position -20, and the backward position 106 respectively. The SSIM between the first decoded image and the shared image, and the SSIM between the second decoded image and the shared image are both greater than a preset threshold (for example, the preset threshold is 0.6). The position at this time is the synchronization position. For example, when the sliding window position is 20, the SSIMs between the first decoded image, the second decoded image, and the shared image are all greater than 0.6. At this time, the sliding window position 20 is the synchronization position, and the two-way signals are synchronized according to this position. It should be noted that the present disclosure is not limited to calculating the SSIM between images, and other metrics such as PSNR can also be calculated.

[0118] In this embodiment, the value of n depends on the number of channels C1 of the first semantic decoding model and the number of channels C2 of the second semantic decoding model. The semantic encoding model for encoding the shared image consists of multiple two-dimensional convolutional layers. When performing repeated two-dimensional convolutional operations on the shared image, each convolutional layer generates a new set of semantic feature maps. These semantic feature maps are stacked in the channel direction to form a three-dimensional tensor, where each element represents a feature. Each convolutional kernel contains a set of trainable weight parameters in the channel direction. The convolutional kernel extracts the corresponding sequential feature information by convolving the input feature maps. Therefore, the vector group in the channel direction can be understood as a set of semantic features describing certain specific features in the input image.

[0119] Based on the above principles, when the vectors after semantic encoding are unfolded in the channel direction, every C (representing the number of channels) symbols represent a group of semantic features. If the shift of the sequence is an integer multiple of C, the decoded result will contain partial content of the shared image. If the shift of the sequence is not an integer multiple of C, the order of the semantic features corresponding to each feature after the shift will be disturbed. Since the semantic decoding model and the semantic encoding model share context knowledge, the semantic decoding model cannot correctly decode the semantic features with disturbed order. Therefore, two groups of vectors with the number of channels C1 and C2 in the encoding result are respectively used as the synchronization headers of the two signals. At the receiving end, through the method of sliding window detection, two heterogeneous semantic decoding models are respectively used to decode the semantic information in the sliding window; when the decoded results both contain partial content of the shared image, it may be at the synchronization position at this time, or the difference between the current position and the correct synchronization position is an integer multiple of C1*C2. Therefore, in this embodiment, the current position when the decoded image simultaneously shows partial content of the shared image, the position C1*C2 forward from the current position, and the position C1*C2 backward from the current position are judged to avoid the order of the semantic features after decoding being disturbed.

[0120] In the technical solution of the present disclosure, the acquisition, storage, and application of the user's personal information involved all comply with the provisions of relevant laws and regulations and do not violate public order and good customs.

[0121] According to the embodiments of the present disclosure, the present disclosure also provides an electronic device, a readable storage medium, and a computer program product.

[0122] Figure 7 FIG. shows a schematic block diagram of an exemplary electronic device 700 that can be used to implement the embodiments of the present disclosure. The electronic device is intended to represent various forms of digital computers, such as, laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as, personal digital processors, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are only examples and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0123] As Figure 7As shown, device 700 includes a computing unit 701, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 702 or a computer program loaded from a storage unit 708 into a random access memory (RAM) 703. In the RAM 703, various programs and data required for the operation of the device 700 can also be stored. The computing unit 701, the ROM 702, and the RAM 703 are connected to each other via a bus 704. An input / output (I / O) interface 705 is also connected to the bus 704.

[0124] Multiple components in the device 700 are connected to the I / O interface 705, including: an input unit 706, such as a keyboard, a mouse, etc.; an output unit 707, such as various types of displays, speakers, etc.; a storage unit 708, such as a magnetic disk, an optical disc, etc.; and a communication unit 709, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 709 allows the device 700 to exchange information / data with other devices via a computer network such as the Internet and / or various telecommunication networks.

[0125] The computing unit 701 can be various general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 701 include but are not limited to a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning objective function algorithms, a digital signal processor (DSP), and any appropriate processor, controller, microcontroller, etc. The computing unit 701 executes the various methods and processes described above, such as the dual-channel signal synchronization method based on a semantic heterogeneous model. For example, in some embodiments, the dual-channel signal synchronization method based on a semantic heterogeneous model can be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as the storage unit 708. In some embodiments, part or all of the computer program can be loaded and / or installed onto the device 700 via the ROM 702 and / or the communication unit 709. When the computer program is loaded into the RAM 703 and executed by the computing unit 701, one or more steps of the dual-channel signal synchronization method based on a semantic heterogeneous model described above can be executed. Alternatively, in other embodiments, the computing unit 701 can be configured to execute the dual-channel signal synchronization method based on a semantic heterogeneous model in any other appropriate manner (e.g., by means of firmware).

[0126] The various embodiments of the systems and techniques described above in this document can be implemented in digital electronic circuitry, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on a chip (SOCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special-purpose or general-purpose programmable processor that receives data and instructions from a storage system, at least one input device, and at least one output device, and transmits the data and instructions to the storage system, the at least one input device, and the at least one output device.

[0127] The program code for implementing the methods of the present disclosure can be written in any combination of one or more programming languages. These program codes can be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, such that when the program codes are executed by the processor or controller, the functions / operations specified in the flowcharts and / or block diagrams are implemented. The program codes can be executed entirely on the machine, partially on the machine, as an independent software package partially on the machine and partially on a remote machine, or entirely on a remote machine or server.

[0128] In the context of the present disclosure, a machine-readable medium can be a tangible medium that can contain or store a program for use by or in connection with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media would include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, compact disc read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0129] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device for displaying information to the user (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor); and a keyboard and a pointing device (e.g., a mouse or a trackball) by which the user can provide input to the computer. Other kinds of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0130] The systems and techniques described herein can be implemented in a computing system including backend components (e.g., as a data server), or a computing system including middleware components (e.g., an application server), or a computing system including frontend components (e.g., a user computer having a graphical user interface or a web browser through which the user can interact with an implementation of the systems and techniques described herein), or a computing system including any combination of such backend components, middleware components, or frontend components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: local area network (LAN), wide area network (WAN), and the Internet.

[0131] A computer system can include a client and a server. The client and the server are generally remote from each other and typically interact through a communication network. The client-server relationship is created by computer programs running on the respective computers and having a client-server relationship with each other. The server can be a cloud server, a server of a distributed system, or a server incorporating a blockchain.

[0132] It should be understood that various forms of the processes shown above can be used, with steps reordered, added, or deleted. For example, the steps recited in this disclosure can be executed in parallel, sequentially, or in a different order, as long as the desired results of the technical solution of this disclosure can be achieved, and no limitation is imposed herein.

[0133] The above specific embodiments do not constitute a limitation on the protection scope of this disclosure. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this disclosure shall be included within the protection scope of this disclosure.

Claims

1. A dual-channel signal synchronization method based on a semantic heterogeneous model, characterized in that, Including: The sending end encodes the shared image using the first semantic encoding model to obtain a first semantic vector, and encodes the shared image using the second semantic encoding model to obtain a second semantic vector; The sending end combines the first semantic vector and the first service information to obtain first transmission information, and combines the second semantic vector and the second service information to obtain second transmission information; The sending end sends the first transmission information and the second transmission information to the receiving end through a channel; wherein, the first transmission information and the second transmission information are used to be provided to the receiving end, and a first decoded image and a second decoded image are obtained through decoding by the first semantic decoding model and the second semantic decoding model, and the synchronization position of the first transmission information and the second transmission information is determined according to the similarities between the first decoded image, the second decoded image and the shared image.

2. The method according to claim 1, wherein The first semantic encoding model and the second semantic encoding model are heterogeneous encoding models; the first semantic decoding model and the second semantic decoding model are heterogeneous decoding models; the first semantic encoding model corresponds to the first semantic decoding model, and the second semantic encoding model corresponds to the second semantic decoding model.

3. The method according to claim 1, wherein The sending end combines the first semantic vector and the first service information to obtain first transmission information, including: Converting the first semantic vector into a one-dimensional vector, and connecting the first service information as the synchronization header of the first transmission information to obtain the first transmission information.

4. The method according to claim 1, wherein The sending end combines the second semantic vector and the second service information to obtain second transmission information, including: Converting the second semantic vector into a one-dimensional vector, and connecting the second service information as the synchronization header of the second transmission information to obtain the second transmission information.

5. The method according to any one of claims 1-4, wherein The sending end sends the first transmission information and the second transmission information to the receiving end through a channel, including: During the coherence time of the channel, sending the first transmission information and the second transmission information to the receiving end.

6. A dual-channel signal synchronization method based on a semantic heterogeneous model, characterized in that, Including: The receiving end receives the first transmission information and the second transmission information; wherein, the first transmission information is obtained by the sending end encoding the shared image using the first semantic encoding model to obtain a first semantic vector, and combining the first semantic vector and the first service information; the second transmission information is obtained by the sending end encoding the shared image using the second semantic encoding model to obtain a second semantic vector, and combining the second semantic vector and the second service information; The receiving end decodes the first transmission information using the first semantic decoding model to obtain a first decoded image, and decodes the second transmission information using the second semantic decoding model to obtain a second decoded image; The receiving end determines the synchronization position of the first transmission information and the second transmission information according to the similarities between the first decoded image, the second decoded image and the shared image.

7. The method according to claim 6, wherein Before the receiving end determines the synchronization position of the first transmission information and the second transmission information according to the similarities between the first decoded image, the first decoded image and the shared image, further including: Determine whether at least part of the content of the shared image is included in both the first decoded image and the second decoded image; In response to both the first decoded image and the second decoded image including at least part of the content of the shared image, the receiving end determines the synchronization positions of the first transmission information and the second transmission information according to the similarities between the first decoded image, the second decoded image and the shared image; In response to the first decoded image and the second decoded image not including at least part of the content of the shared image, return that the receiving end decodes the first transmission information using a first semantic decoding model to obtain a first decoded image, and decodes the second transmission information using a second semantic decoding model to obtain a second decoded image.

8. The method according to claim 6, wherein The receiving end decodes the first transmission information using a first semantic decoding model to obtain a first decoded image, and decodes the second transmission information using a second semantic decoding model to obtain a second decoded image, including: Decode the first transmission information through a first sliding window and the first semantic decoding model to obtain the first decoded image; Decode the second transmission information through a second sliding window and the second semantic decoding model to obtain the second decoded image.

9. The method according to claim 8, wherein The receiving end determines the synchronization positions of the first transmission information and the second transmission information according to the similarities between the first decoded image, the second decoded image and the shared image, including: Obtain the current positions of the first sliding window and the second sliding window, the n positions forward from the current position, and the n positions backward from the current position when the content of the shared image appears in both the first decoded image and the second decoded image; Compare the similarities between the first decoded image, the second decoded image and the shared image when the first sliding window and the second sliding window are at the current position, the n positions forward, and the n positions backward respectively; Determine the positions where the similarities between the first decoded image, the second decoded image and the shared image are greater than a preset similarity as the synchronization positions.

10. The method according to claim 9, characterized in that, n is equal to the product of the number of channels of the first channel corresponding to the first semantic vector and the number of channels of the second channel corresponding to the second semantic vector.

11. A dual-channel signal synchronization device based on a semantic heterogeneity model, characterized in that, Including: An encoding module, configured to encode a shared image using a first semantic encoding model to obtain a first semantic vector, and encode the shared image using a second semantic encoding model to obtain a second semantic vector; An information merging module, configured to merge the first semantic vector and first service information to obtain first transmission information, and merge the second semantic vector and second service information to obtain second transmission information; An information sending module, configured to send the first transmission information and the second transmission information to a receiving end through a channel; wherein, the first transmission information and the second transmission information are used to be provided to the receiving end, decoded by a first semantic decoding model and a second semantic decoding model to obtain a first decoded image and a second decoded image, and determine a synchronization position of the first transmission information and the second transmission information according to similarities between the first decoded image, the second decoded image and the shared image.

12. The device according to claim 11, characterized in that, The first semantic encoding model and the second semantic encoding model are heterogeneous encoding models; the first semantic decoding model and the second semantic decoding model are heterogeneous decoding models; the first semantic encoding model corresponds to the first semantic decoding model, and the second semantic encoding model corresponds to the second semantic decoding model.

13. A dual-channel signal synchronization device based on a semantic heterogeneity model, characterized in that, Including: A receiving module, configured to receive the first transmission information and the second transmission information; wherein, the first transmission information is obtained by a sending end encoding a shared image using a first semantic encoding model to obtain a first semantic vector and combining the first semantic vector with first service information; the second transmission information is obtained by the sending end encoding the shared image using a second semantic encoding model to obtain a second semantic vector and combining the second semantic vector with second service information; A decoding module, configured to decode the first transmission information using a first semantic decoding model to obtain a first decoded image, and decode the second transmission information using a second semantic decoding model to obtain a second decoded image; A synchronization module, configured to determine a synchronization position of the first transmission information and the second transmission information according to similarities between the first decoded image, the second decoded image and the shared image.

14. The device according to claim 13, characterized in that, Further including: A judgment module, configured to judge whether at least partial content of the shared image is included in both the first decoded image and the second decoded image before the synchronization module determines the synchronization position of the first transmission information and the second transmission information according to similarities between the first decoded image, the first decoded image and the shared image; In response to at least partial content of the shared image being included in both the first decoded image and the second decoded image, the receiving end determines the synchronization position of the first transmission information and the second transmission information according to similarities between the first decoded image, the second decoded image and the shared image; In response to at least partial content of the shared image not being included in both the first decoded image and the second decoded image, return the first decoded image obtained by the receiving end decoding the first transmission information using the first semantic decoding model, and the second decoded image obtained by decoding the second transmission information using the second semantic decoding model.

15. The device according to claim 13, characterized in that, The decoding module includes: A first decoding unit, configured to decode the first transmission information through a first sliding window and the first semantic decoding model to obtain the first decoded image; A second decoding unit, configured to decode the second transmission information through a second sliding window and the second semantic decoding model to obtain the second decoded image.

16. The device according to claim 15, characterized in that, The synchronization module determining the synchronization positions of the first transmission information and the second transmission information according to the similarities between the first decoded image, the second decoded image and the shared image includes: obtaining the current positions of the first sliding window and the second sliding window, the n positions forward of the current position, and the n positions backward of the current position when the content of the shared image appears in both the first decoded image and the second decoded image; respectively comparing the similarities between the first decoded image, the second decoded image and the shared image when the first sliding window and the second sliding window are at the current position, the n positions forward of the current position, and the n positions backward of the current position; determining the positions where the similarities between the first decoded image, the second decoded image and the shared image are greater than a preset similarity as the synchronization positions.

17. An electronic device, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein, the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to execute the method according to any one of claims 1-10.

18. A non-transitory computer-readable storage medium storing computer instructions, wherein, The computer instructions are used to cause the computer to execute the method according to any one of claims 1-10.

19. A computer program product, comprising a computer program, where the computer program, when executed by a processor, implements the method according to any one of claims 1-10.

Images