Jump connection network, coding and decoding method and device thereof, equipment and medium
通过在跳连网络中设置隐空间编解码网络,将高维特征数据编码为低维隐空间表示,解决了跳连网络中数据传输时延和带宽占用的问题,实现了更高效的数据传输和 Better base network performance.
Patent Information
- Application Number
- CN202411689670.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-11-25
- Publication Date
- 2025-01-28
AI Technical Summary
Existing hopping networks are prone to long data transmission delays and large bandwidth usage during actual hardware deployment, affecting the performance of the baseline network.
Set up a hidden space coding network in the jumping network, encode high -dimensional characteristics data as low -dimensional hidden space, and then transmit the jump network to reduce data transmission volume and bandwidth occupation.
Through low -dimensional hidden space, the key information in high -dimensional feature data is retained, the data transmission delay and bandwidth occupation are reduced, and the performance of the baseline network where the jumping network is located.
Smart Images
Figure CN119378608A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of coding and decoding technology, and in particular to a skip connection network and a coding and decoding method, device, equipment and medium thereof. Background Art
[0002] Skip connections are widely used in neural networks. Skip connections refer to skipping some intermediate layers in the neural network and directly passing shallow features extracted from some network layers before the intermediate layers to the network layers after the intermediate layers, so that subsequent network layers can make up for the limitations of relying only on deep features based on shallow features.
[0003] The current mainstream skip connection network can be applied to scenarios such as super-resolution and image denoising. Take the baseline network based on the U-Net network as an example, which contains an encoder module, a decoder module and a skip connection module.
[0004] However, in actual hardware deployment, the skip connection module usually faces some challenges. For example, since the skip connection module needs to transfer data between different stages in the network, it is prone to long data transmission delays and large bandwidth occupancy, which has a negative impact on the performance of the baseline network where the skip connection network is located. Summary of the invention
[0005] In view of this, the purpose of the present invention is to provide a skip-connection network and its encoding and decoding method, device, equipment, and medium, which can reduce the data transmission delay and bandwidth occupancy of the skip-connection network transmission by setting a latent space encoding and decoding network in the skip-connection network to encode high-dimensional feature data into a low-dimensional latent space representation and then transmit it through the skip-connection network, and the low-dimensional latent space representation retains the key information in the high-dimensional feature data, so that the second network can obtain relatively complete decoded high-dimensional feature data, thereby ensuring the performance of the baseline network where the skip-connection network is located.
[0006] The specific plan is as follows:
[0007] In a first aspect, the present application provides a coding and decoding method based on a skip connection network, wherein the skip connection network includes a latent space coding and decoding network, and the method includes:
[0008] Acquire first high-dimensional feature data from the first network;
[0009] The first high-dimensional feature data is encoded into a low-dimensional latent space representation through the latent space encoding and decoding network, and the low-dimensional latent space representation is transmitted through a skip connection network and then decoded into a second high-dimensional feature data, so that the second network receives the second high-dimensional feature data; the skip connection network is arranged between the first network and the second network.
[0010] Optionally, the latent space encoding and decoding network includes a first encoding network arranged close to the first network and a first decoding network arranged close to the second network;
[0011] The step of encoding the first high-dimensional feature data into a low-dimensional latent space representation through the latent space encoding and decoding network, and decoding the low-dimensional latent space representation into second high-dimensional feature data after transmitting it through a skip connection network, comprises:
[0012] Encoding the first high-dimensional feature data into a low-dimensional latent space representation through the first encoding network, and transmitting the low-dimensional latent space representation to the first decoding network through a skip connection network;
[0013] The low-dimensional latent space representation transmitted through the skip connection network is received through the first decoding network, and the low-dimensional latent space representation is decoded into second high-dimensional feature data, so as to transmit the second high-dimensional feature data to the second network.
[0014] Optionally, the first network, the second network and the jump connection network constitute a target network;
[0015] Accordingly, the process of constructing the target network includes:
[0016] The target network is obtained according to the baseline network and the skip-connection network to be trained; wherein the skip-connection network to be trained includes an initial latent space codec network; and the baseline network includes an initial first network, an initial second network and an initial skip-connection network.
[0017] Optionally, obtaining the target network according to the baseline network and the skip-connection network to be trained includes:
[0018] Obtain the trained baseline network;
[0019] Replacing the initial skip connection network in the baseline network with the skip connection network to be trained, so as to update the baseline network to obtain an intermediate network;
[0020] The intermediate network is trained to obtain the trained target network.
[0021] Optionally, the training the intermediate network to obtain the trained target network includes:
[0022] The weight parameters of the first network and the second network in the intermediate network are frozen, and the skip-connection network to be trained in the intermediate network is fine-tuned to obtain the trained target network.
[0023] In a second aspect, the present application provides a jump connection network, wherein the jump connection network is arranged between a first network and a second network, and the jump connection network includes:
[0024] A latent space encoding and decoding network is used to encode the first high-dimensional feature data obtained from the first network into a low-dimensional latent space representation, and decode the low-dimensional latent space representation into a second high-dimensional feature data after passing it through a skip connection network, so that the second network receives the second high-dimensional feature data.
[0025] Optionally, the latent space encoding and decoding network includes a first encoding network and a first decoding network;
[0026] The first encoding network is arranged close to the first network, and is used to encode the first high-dimensional feature data obtained from the first network into a low-dimensional latent space representation, and transmit the low-dimensional latent space representation to the first decoding network through a skip connection network;
[0027] The first decoding network is arranged close to the second network, and is used to receive the low-dimensional latent space representation transmitted via the skip connection network, and decode the low-dimensional latent space representation into second high-dimensional feature data, so as to transmit the second high-dimensional feature data to the second network.
[0028] In a third aspect, the present application provides a coding and decoding device based on a skip connection network, wherein the skip connection network includes a latent space coding and decoding network, and the device includes:
[0029] An acquisition module, used for acquiring first high-dimensional feature data from the first network;
[0030] A coding and decoding module is used to encode the first high-dimensional feature data into a low-dimensional latent space representation through the latent space coding and decoding network, and decode the low-dimensional latent space representation into a second high-dimensional feature data after passing it through a skip connection network, so that the second network receives the second high-dimensional feature data; the skip connection network is arranged between the first network and the second network.
[0031] In a fourth aspect, the present application provides an electronic device, including:
[0032] Memory for storing computer programs;
[0033] A processor is used to implement the steps of the aforementioned method when executing the computer program.
[0034] In a fifth aspect, the present application provides a computer storage medium having a computer program stored thereon, and the computer program implements the steps of the aforementioned method when executed by a processor.
[0035] The present application discloses a coding and decoding method based on a skip connection network, wherein the skip connection network includes a latent space coding and decoding network, and the coding and decoding method based on the skip connection network includes: obtaining first high-dimensional feature data from a first network; encoding the first high-dimensional feature data into a low-dimensional latent space representation through the latent space coding and decoding network, and decoding the low-dimensional latent space representation into second high-dimensional feature data after passing it through the skip connection network, so that the second network receives the second high-dimensional feature data; the skip connection network is arranged between the first network and the second network.
[0036] It can be seen that the present application sets a latent space codec network in the skip network to encode the high-dimensional feature data in the first network into a low-dimensional latent space representation through the latent space codec network and then transmits it through the skip network. Compared with the current skip network structure that directly transmits the high-dimensional feature data through the skip network, it not only reduces the amount of data transmission in the skip network, thereby reducing the data transmission delay and bandwidth occupancy of the skip network transmission, but also because the low-dimensional latent space representation is learned and encoded from the high-dimensional feature data, the low-dimensional latent space representation retains the key feature information in the high-dimensional feature data, and finally enables the second network to obtain relatively complete high-dimensional feature data after decoding, thereby ensuring the performance of the baseline network where the skip network is located to a certain extent.
[0037] In summary, the embodiments of the present application solve the technical problem that the prior art is prone to long data transmission delays and large bandwidth occupancy, thereby negatively affecting the performance of the baseline network where the jump network is located, while maximally retaining the necessary key feature information in the high-dimensional feature data, so that the overall baseline network has higher performance. For example, when the baseline network is a noise reduction network, the overall network has an efficient and accurate noise reduction effect. BRIEF DESCRIPTION OF THE DRAWINGS
[0038] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying creative work.
[0039] Figure 1 A flow chart of a coding and decoding method based on a skip-connection network disclosed in this application;
[0040] Figure 2A flowchart of the composition of a target network disclosed in this application;
[0041] Figure 3 A schematic diagram of a target network structure disclosed in this application;
[0042] Figure 4 A schematic diagram of a specific target network structure disclosed in this application;
[0043] Figure 5 A schematic diagram of the structure of a coding and decoding device based on a skip connection network disclosed in this application;
[0044] Figure 6 This is a structural diagram of an electronic device disclosed in this application. DETAILED DESCRIPTION
[0045] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0046] In actual hardware deployment, skip connections usually face some challenges. For example, since skip connections need to transfer data between different stages in the baseline network, they are prone to long data transmission delays and large bandwidth occupancy, thereby reducing the utilization of hardware accelerators in the network. Hardware accelerators in the network can be, for example, GPUs (Graphics Processing Units) and NPUs (Neural Processing Units), which has a negative impact on the performance of the baseline network where the skip connection network is located.
[0047] Therefore, how to ensure the feature transmission capability of the skip connection module while reducing its occupation of hardware resources to better balance the network bandwidth utilization and the effect of the baseline network has become a challenge that needs to be solved urgently.
[0048] In order to solve the above technical problems, the present application provides a skip connection network and its encoding and decoding method, device, equipment, and medium, wherein the skip connection network is set between the first network and the second network, and includes a latent space encoding and decoding network. Specifically, the high-dimensional feature data in the first network is encoded into a low-dimensional latent space representation through the latent space encoding and decoding network and then transmitted through the skip connection network. Compared with the current skip connection network structure that directly transmits the high-dimensional feature data through the skip connection network, it not only reduces the amount of data transmission in the skip connection, thereby reducing the data transmission delay and bandwidth occupancy of the skip connection network transmission, but also because the low-dimensional latent space representation is learned from the high-dimensional feature data. The encoding is obtained, the low-dimensional latent space representation retains the key feature information in the high-dimensional feature data, and finally enables the second network to obtain relatively complete high-dimensional feature data after decoding, thereby ensuring the performance of the baseline network where the skip connection network is located to a certain extent, thereby ensuring the feature transmission capability of the skip connection module while reducing its occupation of hardware resources, and can better balance the network bandwidth utilization and the effect of the baseline network.
[0049] Overall, the technical problem that the existing technology is prone to long data transmission delays and large bandwidth occupancy, which has a negative impact on the performance of the baseline network where the skip-connection network is located, is solved. At the same time, the necessary key feature information in the high-dimensional feature data is retained to the maximum extent, so that the overall skip-connection network has efficient and accurate performance.
[0050] The following first introduces a coding and decoding method based on a skip connection network involved in the solution.
[0051] See also Figure 1 As shown, an embodiment of the present invention discloses a coding and decoding method based on a skip connection network, wherein the skip connection network includes a latent space coding and decoding network, and the method includes:
[0052] Step S11: Acquire first high-dimensional feature data from the first network.
[0053] Step S12: encode the first high-dimensional feature data into a low-dimensional latent space representation through the latent space encoding and decoding network, and decode the low-dimensional latent space representation into second high-dimensional feature data after passing it through a skip connection network, so that the second network receives the second high-dimensional feature data; the skip connection network is set between the first network and the second network.
[0054] In this embodiment, the first network has shallow feature data, and the shallow feature data may include first high-dimensional feature data obtained from the first network by the skip connection network or the latent space encoding and decoding network.
[0055] It should be noted that the first high-dimensional feature data obtained by the latent space codec network can be the first high-dimensional feature data obtained by the latent space codec network directly from the first network, or the first high-dimensional feature data can be first obtained from the first network through a skip connection network and then the first high-dimensional feature data is passed to the latent space codec network.
[0056] The latent space encoding and decoding network can encode the first high-dimensional feature data to obtain a low-dimensional latent space representation, and then decode the low-dimensional latent space representation into second high-dimensional feature data after passing it through a skip connection network, and pass the second high-dimensional feature data to the second network, so that the second network receives the second high-dimensional feature data.
[0057] It should also be noted that the skip-connection network is arranged between the first network and the second network, and the skip-connection network includes a latent space codec network. Therefore, the latent space codec network can be arranged on a pipeline transmitted by the skip-connection network in the skip-connection network, which connects the first network and the second network.
[0058] The latent space encoding and decoding network can play the role of dimensionality reduction and dimensionality increase in the pipeline, that is, the latent space encoding and decoding network can make the first high-dimensional feature data obtained from the first network be reduced in dimension and then transmitted through the skip-connection network, and finally the low-dimensional latent space representation transmitted by the skip-connection network is upgraded to the second high-dimensional feature data and then sent to the second network, so that the second network can receive the second high-dimensional feature data obtained by upgrading the low-dimensional latent space representation transmitted by the skip-connection network.
[0059] Since the low-dimensional latent space representation is learned and encoded from shallow high-dimensional feature data, the low-dimensional latent space representation retains the key feature information in the shallow high-dimensional feature data. The second network obtains the high-dimensional feature data obtained by upgrading the low-dimensional latent space representation, thereby obtaining the key information of the shallow features contained therein, that is, the second high-dimensional feature data belongs to shallow feature data.
[0060] It should be noted that there are also multiple network layers connected in sequence between the first network and the second network, and none of the multiple network layers are connected to the aforementioned skip-connection network. After obtaining the first high-dimensional feature data from the first network, it also includes: transferring the first high-dimensional feature data from the first network layer to the second network layer in a non-skip-connection network transfer manner through multiple network layers to obtain third feature data; and determining the output result based on the second high-dimensional feature data and the third feature data.
[0061] That is, for the second network, in addition to obtaining the second high-dimensional feature data obtained by dimensionalizing the low-dimensional latent space representation, the first high-dimensional feature data can also be transmitted from the first network to the second network in a non-skip network transmission manner through multiple network layers between the first network and the second network, so that the second network receives the third feature data, and fuses or splices the second high-dimensional feature data and the third feature data to obtain the output result. It should be noted that the third feature data is the data obtained after the first high-dimensional feature data is processed by multiple network layers, that is, the third feature data belongs to deep feature data; and the third feature data and the second high-dimensional feature data have the same number of channels; in addition, after the second high-dimensional feature and the third feature data are input into the second network layer, the output result can be obtained by decoding the two.
[0062] The second network can make up for the limitation of the second network relying only on deep features by fusing or splicing the shallow second high-dimensional feature data and the deep third feature data, and can help improve the capabilities of the entire baseline network where the skip connection network is located. Taking the baseline network as a denoising network as an example, the setting of the skip connection network can help improve the entire baseline network's ability to reconstruct detailed texture information.
[0063] This embodiment sets a latent space codec network in the skip network to encode the high-dimensional feature data in the first network into a low-dimensional latent space representation through the latent space codec network and then transmit it through the skip network. Compared with the current skip network structure that directly transmits the high-dimensional feature data through the skip network, it not only reduces the amount of data transmission in the skip network, thereby reducing the data transmission delay and bandwidth occupancy of the skip network transmission, but also because the low-dimensional latent space representation is learned and encoded from the high-dimensional feature data, the low-dimensional latent space representation retains the key feature information in the high-dimensional feature data, and finally enables the second network to obtain relatively complete high-dimensional feature data after decoding, thereby ensuring the performance of the baseline network where the skip network is located to a certain extent.
[0064] In summary, the embodiments of the present application solve the technical problem that the prior art is prone to long data transmission delays and large bandwidth occupancy, thereby negatively affecting the performance of the baseline network where the jump network is located, while maximally retaining the necessary key feature information in the high-dimensional feature data, so that the overall baseline network has higher performance. For example, when the baseline network is a noise reduction network, the overall network has an efficient and accurate noise reduction effect.
[0065] It should be noted that, on this basis, the latent space encoding and decoding network in the skip connection network includes a first encoding network arranged close to the first network and a first decoding network arranged close to the second network;
[0066] Correspondingly, the above step S12 may include:
[0067] S121, encoding the first high-dimensional feature data into a low-dimensional latent space representation through the first encoding network, and transmitting the low-dimensional latent space representation to the first decoding network through a skip connection network;
[0068] S122. Receive, through the first decoding network, the low-dimensional latent space representation transmitted via the skip connection network, and decode the low-dimensional latent space representation into second high-dimensional feature data, so as to transmit the second high-dimensional feature data to the second network.
[0069] Since the latent space encoding and decoding network is set on the pipeline of the skip-connection network transmission in the skip-connection network, the pipeline connects the first network and the second network, and the latent space encoding and decoding network includes the first encoding network and the first decoding network, the first encoding network and the first decoding network are also set on the pipeline of the skip-connection network transmission in the skip-connection network. Moreover, in the pipeline, the first decoding network is closer to the second network than the first encoding network and the first network, and the first encoding network is closer to the first network than the first decoding network and the second network.
[0070] When performing latent space encoding and decoding, the first encoding network performs dimensionality reduction at the front end so that the first high-dimensional feature data is encoded into a low-dimensional latent space representation, so that the pipeline transmitted by the skip connection network transmits low-dimensional feature data; when the low-dimensional latent space representation is transmitted to the first decoding network set close to the second network through the pipeline, the first decoding network receives the low-dimensional latent space representation transmitted through the skip connection network, and reconstructs the low-dimensional latent space representation into a second high-dimensional feature data, and then transmits the second high-dimensional feature data to the second network.
[0071] Based on the above description, it can be understood that the embodiment of the present application can be understood as adding a first encoding network and a first decoding network to the original pipeline of the skip-connection network transmission, so that only the low-dimensional features encoded by the first encoding network need to be transmitted in the transmission stream of the pipeline, until the end of the skip-connection network transmission, the low-dimensional features are decoded by the first decoding network. That is, the entire first encoding network and the first decoding network, as skip-connection intermediaries, can complete the corresponding encoding and decoding at different stages in the forward flow of the skip-connection network, thereby reducing the transmission bandwidth.
[0072] It should be noted that the first encoding network can use the first convolution to encode the high-dimensional feature data, and the first decoding network can use the second convolution to decode the low-dimensional latent space representation. And because the second high-dimensional feature data and the first high-dimensional feature data have the same number of channels, the first convolution and the second convolution can have the same convolution kernel size and convolution step.
[0073] It should also be noted that in order to reduce the computing power consumed by the first encoding network and the first decoding network and increase the network processing speed, the first encoding network and the first decoding network may both use fewer network parameters.
[0074] Exemplarily, the network scale of the first encoding network is smaller than that of the first network, and the network scale of the first decoding network is smaller than that of the second network. Furthermore, the first encoding network and the first decoding network are two corresponding network layers in the latent space encoding and decoding network, and the first network and the second network are two corresponding network layers in the baseline network where the skip connection network is located.
[0075] Exemplarily, when each stage of the first network corresponds to each stage of the second network, a latent space encoding and decoding module is added to each stage of the first network, so that the shallow first high-dimensional feature data therein is encoded into a low-dimensional latent space representation. In the corresponding stage of the second network, the second high-dimensional feature data obtained by dimensional upgrading the low-dimensional latent space representation transmitted through the skip connection network is received, so that the second network can obtain the shallow features / key information of the shallow features contained therein.
[0076] Exemplarily, in order to reduce network parameters, convolution with both convolution kernel size and convolution step size set to 1 can be used, that is, the convolution kernel size and convolution step size of the first convolution used by the first encoding network and the second convolution used by the first decoding network can be set to 1.
[0077] Exemplarily, the baseline network in which the skip connection network is located can be a UNet network, such as a denoising network based on UNet, or a super-resolution network, such as a super-resolution network based on a resnet straight network, and of course it can also be other baseline networks with skip connections.
[0078] In the actual application of the skip-connected network, the skip-connected network proposed in this embodiment can be used for image denoising, as well as for voice / audio signals and audio and video denoising, etc. Correspondingly, the feature data is image data or voice / audio / audio and video data. Of course, it can also be applied to other fields or scenarios, which will not be elaborated here.
[0079] It can be seen that the present application sets a latent space codec network in the skip network to encode the high-dimensional feature data in the first network into a low-dimensional latent space representation through the latent space codec network and then transmits it through the skip network. Compared with the current skip network structure that directly transmits the high-dimensional feature data through the skip network, it not only reduces the amount of data transmission in the skip network, thereby reducing the data transmission delay and bandwidth occupancy of the skip network transmission, but also because the low-dimensional latent space representation is learned and encoded from the high-dimensional feature data, the low-dimensional latent space representation retains the key feature information in the high-dimensional feature data, and finally enables the second network to obtain relatively complete high-dimensional feature data after decoding, thereby ensuring the performance of the baseline network where the skip network is located to a certain extent.
[0080] Based on the above embodiments, it can be known that the above embodiments describe how the skip network transmits the high-dimensional feature data obtained from the first network to the second network after dimensionality reduction encoding, skip network transmission, and dimensionality increase decoding, wherein the first network, the second network and the skip network as a whole constitute the target network. The first network, the second network and the skip network mentioned here are all the final trained networks, and correspondingly, the target network is also the final trained network.
[0081] It should be noted that the process of constructing the target network includes: constructing the target network according to the baseline network and the skip-connection network to be trained; wherein the skip-connection network to be trained includes the initial latent space codec network, and the baseline network includes the initial first network, the initial second network and the initial skip-connection network. The baseline network mentioned here is a trained network, but not the final trained network.
[0082] Specifically, Figure 2 As shown, an embodiment of the present invention discloses a method for constructing a target network, including:
[0083] Step S21: Obtain a trained baseline network.
[0084] In this embodiment, a pre-trained baseline network is obtained, wherein the baseline network includes an initial first network, an initial second network, and an initial skip-connection network; the structure of the initial skip-connection network is a standard skip-connection network structure. It should be noted that "trained" here means that the baseline network has been trained to convergence at the current stage, and is a pre-training stage for subsequent training of other networks based on the baseline network.
[0085] Step S22: replacing the initial skip-connection network in the baseline network with the skip-connection network to be trained, so as to update the baseline network to obtain an intermediate network.
[0086] In this embodiment, the initial skip-connected network in the baseline network is replaced by the skip-connected network to be trained to update the baseline network, thereby obtaining an intermediate network based on the initial first network, the initial second network and the skip-connected network to be trained, wherein the skip-connected network to be trained includes an initial latent space codec network.
[0087] Step S23: training the intermediate network to obtain a trained target network.
[0088] In this embodiment, in the first training stage, the baseline network needs to be trained first to obtain a trained baseline network. In this stage, the baseline network can be trained to convergence, or after the number of iterations, weight parameters, etc. meet the requirements, the trained baseline network is used as the trained baseline network.
[0089] For example, the hyperparameters of the baseline network can be pre-set, and then the baseline network can be iteratively trained using a pre-constructed training set and based on a loss function to obtain a trained baseline network, wherein the hyperparameters include but are not limited to an initial learning rate, a batch size, and a total number of iterations.
[0090] For the baseline network in this embodiment, the initial learning rate can be set to 1e-4, 2e-4 or 1e-3, etc., and the cosine annealing learning rate can be used to adjust the learning rate later; the batch size can be set to 10-20, for example, the batch size can be set to 15; the total iteration rounds can be set to 150-250.
[0091] The above loss function can adopt the L1 loss function. The L1 loss function can be used to compare the output result of the baseline network with the true label, and the baseline network can be adjusted and trained based on the comparison result, thereby speeding up the training speed of the baseline network.
[0092] For the construction of the training set, a training data pair can be constructed based on clean data and noisy data, so as to construct a training set based on several training data pairs. Among them, the clean data can be data without noise, or data with relatively small noise, that is, the noise has almost no effect on the data, and the noisy data corresponds to the clean data.
[0093] Furthermore, in order to ensure the training effect of the baseline network, the number of training data pairs should not be too small, for example, more than 500 pairs, or more than 1000 pairs, or even more than 1200 pairs of training data pairs can be used. In order to improve the generalization of the baseline network, the training data pairs can also be enhanced to enrich the diversity of the training set.
[0094] For another example, the baseline network is iteratively trained to obtain a trained baseline network. The trained baseline network can be obtained when the baseline network obtained after the current round of iterative training reaches convergence, or when the current iteration round reaches a preset total iteration round.
[0095] For another example, the above training set can be used to iteratively train the baseline network based on the AdamW optimizer. During the training process, the cosine annealing algorithm is used to adjust the learning rate, and the training loss between the output result of the baseline network and the corresponding true label in the training set is calculated based on the L1 loss function, thereby training the baseline network to obtain a trained baseline network.
[0096] After obtaining the trained baseline network, the second training phase can be entered, that is, the initial skip connection network in the baseline network is replaced with the skip connection network to be trained to update the baseline network, thereby obtaining an intermediate network based on the initial first network, the initial second network and the skip connection network to be trained.
[0097] It should be noted that the skip connection network to be trained includes the initial latent space codec network, but the initial latent space codec network has not yet completed training and belongs to the initial latent space codec network, that is, it does not belong to the final trained latent space codec network.
[0098] After obtaining the intermediate network, it is necessary to fine-tune the intermediate network to obtain a final trained target network, which includes the final trained first network, the second network and the skip-connection network. The final trained skip-connection network is the skip-connection network involved in the aforementioned embodiment.
[0099] It should be noted that this embodiment forms a two-stage skip connection reconstruction training strategy, in which the first stage is based on the initial skip connection network, and pre-training obtains a trained baseline network. In the second stage, the initial skip connection network in the trained baseline network is replaced with a new skip connection network with an initial latent space codec network added to obtain an intermediate network, thereby training the intermediate network to obtain a trained target network. This embodiment effectively alleviates the accuracy loss introduced by latent space coding by using the pre-trained weights of the first stage as a good initialization.
[0100] In addition, this training method forms an encoding-transmission-decoding process in the entire skip-connection network, which can greatly reduce the amount of data transmission in the skip-connection, thereby reducing network bandwidth occupancy and latency, and improving the utilization rate of hardware accelerators in the network.
[0101] In some examples, the training process of the intermediate network may include:
[0102] The weight parameters of the first network and the second network in the intermediate network are frozen, and the skip connection network to be trained in the intermediate network is fine-tuned to obtain a trained target network.
[0103] Specifically, the weight parameters of the trained baseline network can be loaded into the intermediate network, and the weight parameters of the first network and the second network in the intermediate network can be frozen, and then the training set can be used to fine-tune the skip connection network to be trained in the intermediate network to obtain the final trained target network. The training set used to train the intermediate network can be the training set used to train the baseline network.
[0104] Of course, the corresponding parameters of the first network and the second network can also be adjusted according to actual needs, and the parameters of the skip-connection network to be trained in the intermediate network can be optimized at the same time to reconstruct the skip-connection network in the target network.
[0105] For fine-tuning training of the skip connection network to be trained in the intermediate network, hyperparameters can also be set in advance, for example, the initial learning rate can be set to 4e-5, 3e-5 or 4e-4, etc.; the batch size can be set to 10-20, specifically the batch size can be set to 16; the total iteration rounds can be set to 5-15, specifically the total iteration rounds can be set to 5, 10 or 15.
[0106] In addition, for fine-tuning the skip connection network to be trained in the intermediate network, the AdamW optimizer, L1 loss function and cosine annealing algorithm can also be used. Among them, for the loss function, in addition to the L1 loss function, the mean square error loss function, the mean absolute error loss function, etc. can also be used.
[0107] It can be seen that in this embodiment, the baseline network is first trained in the first stage to obtain a trained baseline network, and then the standard skip connection network (i.e., the initial skip connection network) in the baseline network is replaced with the skip connection network to be trained including the initial latent space codec network to obtain an intermediate network; then, the weight parameters of the first network and the second network in the intermediate network are frozen or adaptively adjusted in the second stage, and the weight parameters of the skip connection network to be trained in the intermediate network are fine-tuned to obtain the final trained target network. In this way, this embodiment uses the weight parameters of the baseline network trained in the first stage as the initialization parameters for the fine-tuning training of the intermediate network in the second stage, which not only alleviates the accuracy loss caused by the introduction of the latent space codec network to a certain extent, but also avoids directly training the skip connection network constructed based on the initial latent space codec network through two-stage training, thereby reducing the difficulty of network training.
[0108] See also Figure 3As shown, an embodiment of the present invention discloses a skip connection network, which is arranged between a first network 11 and a second network 12, and the skip connection network includes a latent space encoding and decoding network 13.
[0109] The latent space encoding and decoding network 13 is used to encode the first high-dimensional feature data obtained from the first network 11 into a low-dimensional latent space representation, and decode the low-dimensional latent space representation into second high-dimensional feature data after passing it through a skip connection network, so that the second network 12 receives the second high-dimensional feature data.
[0110] In this embodiment, the first network 11 has shallow feature data, and the shallow feature data may include first high-dimensional feature data obtained from the first network 11 by the skip connection network or the latent space encoding and decoding network 13.
[0111] It should be noted that the first high-dimensional feature data obtained by the latent space codec network 13 can be the first high-dimensional feature data obtained by the latent space codec network 13 directly from the first network 11, or the first high-dimensional feature data can be first obtained from the first network 11 through a skip connection network and then the first high-dimensional feature data is passed to the latent space codec network 13.
[0112] The latent space encoding and decoding network 13 can encode the first high-dimensional feature data to obtain a low-dimensional latent space representation, and then decode the low-dimensional latent space representation into second high-dimensional feature data after passing it through a skip connection network, and pass the second high-dimensional feature data to the second network 12, so that the second network 12 receives the second high-dimensional feature data.
[0113] It should also be noted that the skip-connection network is set between the first network 11 and the second network 12, and the skip-connection network includes a latent space codec network 13. Therefore, the latent space codec network 13 can be set on the pipeline of the skip-connection network transmission in the skip-connection network, which connects the first network 11 and the second network 12.
[0114] The latent space codec network 13 can play the role of dimensionality reduction and dimensionality increase in the pipeline, that is, the latent space codec network 13 can make the first high-dimensional feature data obtained from the first network 11 be reduced in dimension and then transmitted through the skip-connection network, and finally the low-dimensional latent space representation transmitted by the skip-connection network is upgraded to the second high-dimensional feature data and then sent to the second network 12, so that the second network 12 can receive the second high-dimensional feature data obtained by upgrading the low-dimensional latent space representation transmitted by the skip-connection network.
[0115] Since the low-dimensional latent space representation is learned and encoded from shallow high-dimensional feature data, the low-dimensional latent space representation retains the key feature information in the shallow high-dimensional feature data. The second network 12 obtains the high-dimensional feature data obtained by dimensionalizing the low-dimensional latent space representation, thereby obtaining the key information of the shallow features contained therein, that is, the second high-dimensional feature data belongs to shallow feature data.
[0116] For the second network 12, in addition to obtaining the second high-dimensional feature data obtained by dimensionalizing the low-dimensional latent space representation, the first high-dimensional feature data can also be transmitted from the first network 11 to the second network 12 in a non-skip network transmission manner through multiple network layers between the first network 11 and the second network 12, so that the second network 12 receives the third feature data, and fuses or splices the second high-dimensional feature data and the third feature data to obtain an output result. It should be noted that the third feature data is the data obtained after the first high-dimensional feature data is processed by multiple network layers, that is, the third feature data belongs to deep feature data.
[0117] The second network 12 can make up for the limitation of the second network 12 relying only on deep features by fusing or splicing the shallow second high-dimensional feature data and the deep third feature data, and can help improve the capabilities of the entire baseline network where the skip network is located. Taking the baseline network as a denoising network as an example, the setting of the skip network can help improve the entire baseline network's ability to reconstruct detailed texture information.
[0118] This embodiment sets a latent space codec network in the skip network to encode the high-dimensional feature data in the first network into a low-dimensional latent space representation through the latent space codec network and then transmit it through the skip network. Compared with the current skip network structure that directly transmits the high-dimensional feature data through the skip network, it not only reduces the amount of data transmission in the skip network, thereby reducing the data transmission delay and bandwidth occupancy of the skip network transmission, but also because the low-dimensional latent space representation is learned and encoded from the high-dimensional feature data, the low-dimensional latent space representation retains the key feature information in the high-dimensional feature data, and finally enables the second network to obtain relatively complete high-dimensional feature data after decoding, thereby ensuring the performance of the baseline network where the skip network is located to a certain extent.
[0119] In summary, the embodiments of the present application solve the technical problem that the prior art is prone to long data transmission delays and large bandwidth occupancy, thereby negatively affecting the performance of the baseline network where the jump network is located, while maximally retaining the necessary key feature information in the high-dimensional feature data, so that the overall baseline network has higher performance. For example, when the baseline network is a noise reduction network, the overall network has an efficient and accurate noise reduction effect.
[0120] It should be noted that on this basis, Figure 4 As shown, the latent space encoding and decoding network 13 in the skip connection network includes a first encoding network 131 and a first decoding network 132, wherein:
[0121] A first encoding network 131 is arranged close to the first network 11, and is used to encode the first high-dimensional feature data obtained from the first network 11 into a low-dimensional latent space representation, and transmit the low-dimensional latent space representation to the first decoding network 132 through a skip connection network;
[0122] The first decoding network 132 is arranged close to the second network 12, and is used to receive the low-dimensional latent space representation transmitted via the skip connection network, and decode the low-dimensional latent space representation into second high-dimensional feature data to transmit the second high-dimensional feature data to the second network 12.
[0123] Since the latent space encoding and decoding network 13 is set on the pipeline of the skip-connection network transmission in the skip-connection network, the pipeline connects the first network 11 and the second network 12, and the latent space encoding and decoding network 13 includes the first encoding network 131 and the first decoding network 132, the first encoding network 131 and the first decoding network 132 are also set on the pipeline of the skip-connection network transmission in the skip-connection network. Moreover, in the pipeline, the first decoding network 132 is closer to the second network 12 than the first encoding network 131 and the first network 11, and the first encoding network 131 is closer to the first network 11 than the first decoding network 132 and the second network 12.
[0124] When performing latent space encoding and decoding, the first encoding network 131 performs dimensionality compression at the front end so that the first high-dimensional feature data is encoded into a low-dimensional latent space representation, so that the pipeline transmitted by the skip connection network transmits low-dimensional feature data; when the low-dimensional latent space representation is transmitted to the first decoding network 132 set close to the second network 12 through the pipeline, the first decoding network 132 receives the low-dimensional latent space representation transmitted through the skip connection network, and reconstructs the low-dimensional latent space representation into a second high-dimensional feature data, and then passes the second high-dimensional feature data to the second network 12.
[0125] Based on the above description, it can be understood that the embodiment of the present application can be understood as adding the first encoding network 131 and the first decoding network 132 to the original skip-connection network transmission pipeline, so that only the low-dimensional features encoded by the first encoding network 131 need to be transmitted in the pipeline transmission stream, until the end of the skip-connection network transmission, the low-dimensional features are decoded by the first decoding network 132. That is, the entire first encoding network 131 and the first decoding network 132, as skip-connection intermediaries, can complete the corresponding encoding and decoding at different stages in the forward flow of the skip-connection network, thereby reducing the transmission bandwidth.
[0126] See also Figure 5As shown, an embodiment of the present invention discloses a coding and decoding device based on a skip connection network, wherein the skip connection network includes a latent space coding and decoding network, and the device includes:
[0127] An acquisition module 21, used to acquire first high-dimensional feature data from the first network;
[0128] The encoding and decoding module 22 is used to encode the first high-dimensional feature data into a low-dimensional latent space representation through the latent space encoding and decoding network, and decode the low-dimensional latent space representation into second high-dimensional feature data after passing it through a skip connection network, so that the second network receives the second high-dimensional feature data; the skip connection network is set between the first network and the second network.
[0129] It can be seen that the present application sets a latent space codec network in the skip network to encode the high-dimensional feature data in the first network into a low-dimensional latent space representation through the latent space codec network and then transmits it through the skip network. Compared with the current skip network structure that directly transmits the high-dimensional feature data through the skip network, it not only reduces the amount of data transmission in the skip network, thereby reducing the data transmission delay and bandwidth occupancy of the skip network transmission, but also because the low-dimensional latent space representation is learned and encoded from the high-dimensional feature data, the low-dimensional latent space representation retains the key feature information in the high-dimensional feature data, and finally enables the second network to obtain relatively complete high-dimensional feature data after decoding, thereby ensuring the performance of the baseline network where the skip network is located to a certain extent.
[0130] In some specific embodiments, the latent space encoding and decoding network includes a first encoding network arranged close to the first network and a first decoding network arranged close to the second network;
[0131] The encoding and decoding module 22 includes:
[0132] an encoding unit, configured to encode the first high-dimensional feature data into a low-dimensional latent space representation through the first encoding network, and transmit the low-dimensional latent space representation to the first decoding network through a skip connection network;
[0133] A decoding unit is used to receive the low-dimensional latent space representation transmitted via the skip connection network through the first decoding network, and decode the low-dimensional latent space representation into second high-dimensional feature data to transmit the second high-dimensional feature data to the second network.
[0134] In some specific embodiments, the first network, the second network and the jump connection network constitute a target network;
[0135] Accordingly, the process of constructing the target network in the encoding and decoding device based on the skip connection network includes:
[0136] A target network acquisition module is used to obtain the target network according to a baseline network and a skip-connection network to be trained; wherein the skip-connection network to be trained includes the latent space codec network; and the baseline network includes an initial first network, an initial second network and an initial skip-connection network.
[0137] In some specific embodiments, the target network acquisition module includes:
[0138] A baseline network acquisition unit, used to acquire the trained baseline network;
[0139] A baseline network updating unit, configured to replace the initial skip-connection network in the baseline network with the skip-connection network to be trained, so as to update the baseline network to obtain an intermediate network;
[0140] The intermediate network training unit is used to train the intermediate network to obtain the trained target network.
[0141] In some specific embodiments, the intermediate network training unit includes:
[0142] The skip-connection network fine-tuning unit is used to freeze the weight parameters of the first network and the second network in the intermediate network, and fine-tune the skip-connection network to be trained in the intermediate network to obtain the trained target network.
[0143] Furthermore, the present application also discloses an electronic device. Figure 6 This is a structural diagram of an electronic device 30 according to an exemplary embodiment. The content in the diagram cannot be regarded as any limitation on the scope of use of the present application.
[0144] Figure 6 The present invention provides a schematic diagram of the structure of an electronic device 30 provided in an embodiment of the present application. The electronic device 30 may specifically include: at least one processor 31, at least one memory 32, a power supply 33, a communication interface 34, an input / output interface 35, and a communication bus 36. The memory 32 is used to store a computer program, which is loaded and executed by the processor 31 to implement the relevant steps in the method disclosed in any of the aforementioned embodiments. In addition, the electronic device 30 in this embodiment may specifically be an electronic computer.
[0145] In this embodiment, the power supply 33 is used to provide working voltage for each hardware device on the electronic device 30; the communication interface 34 can create a data transmission channel between the electronic device 30 and the external device, and the communication protocol it follows is any communication protocol that can be applied to the technical solution of the present application, and is not specifically limited here; the input and output interface 35 is used to obtain external input data or output data to the outside world, and its specific interface type can be selected according to specific application needs and is not specifically limited here.
[0146] In addition, the memory 32, as a carrier for storing resources, can be a read-only memory, a random access memory, a disk or an optical disk, etc. The resources stored thereon can include an operating system 321, a computer program 322, etc., and the storage method can be temporary storage or permanent storage.
[0147] The operating system 321 is used to manage and control the hardware devices and computer programs 322 on the electronic device 30, which can be Windows Server, Netware, Unix, Linux, etc. In addition to including computer programs that can be used to complete the method performed by the electronic device 30 disclosed in any of the aforementioned embodiments, the computer program 322 can further include computer programs that can be used to complete other specific tasks.
[0148] Furthermore, the present application also discloses a computer-readable storage medium for storing a computer program; wherein the computer program, when executed by a processor, implements the aforementioned disclosed method. The specific steps of the method can refer to the corresponding contents disclosed in the aforementioned embodiments, and will not be repeated here.
[0149] In this specification, each embodiment is described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the embodiments can be referred to each other. For the device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part.
[0150] Professionals may further appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in the above description according to function. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians may use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.
[0151] The steps of the method or algorithm described in conjunction with the embodiments disclosed herein may be implemented directly using hardware, a software module executed by a processor, or a combination of the two. The software module may be placed in a random access memory (RAM), a memory, a read-only memory (ROM), an electrically programmable ROM, an electrically erasable programmable ROM, a register, a hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art.
[0152] Finally, it should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the presence of other identical elements in the process, method, article or device including the elements.
[0153] The technical solution provided by the present application is introduced in detail above. Specific examples are used in this article to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method of the present application and its core idea. At the same time, for general technicians in this field, according to the idea of the present application, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as a limitation on the present application.
Claims
1. A coding and decoding method based on a skip connection network, characterized in that: The skip connection network includes a latent space codec network, and the method includes: Acquire first high-dimensional feature data from the first network; The first high-dimensional feature data is encoded into a low-dimensional latent space representation through the latent space encoding and decoding network, and the low-dimensional latent space representation is transmitted through a skip connection network and then decoded into a second high-dimensional feature data, so that the second network receives the second high-dimensional feature data; the skip connection network is arranged between the first network and the second network.
2. The method according to claim 1, characterized in that: The latent space encoding and decoding network includes a first encoding network arranged close to the first network and a first decoding network arranged close to the second network; The step of encoding the first high-dimensional feature data into a low-dimensional latent space representation through the latent space encoding and decoding network, and decoding the low-dimensional latent space representation into second high-dimensional feature data after transmitting it through a skip connection network, comprises: Encoding the first high-dimensional feature data into a low-dimensional latent space representation through the first encoding network, and transmitting the low-dimensional latent space representation to the first decoding network through a skip connection network; The low-dimensional latent space representation transmitted through the skip connection network is received through the first decoding network, and the low-dimensional latent space representation is decoded into second high-dimensional feature data, so as to transmit the second high-dimensional feature data to the second network.
3. The method according to claim 1, characterized in that: The first network, the second network and the jump connection network constitute a target network; Accordingly, the process of constructing the target network includes: The target network is obtained according to the baseline network and the skip-connection network to be trained; wherein the skip-connection network to be trained includes an initial latent space codec network; and the baseline network includes an initial first network, an initial second network and an initial skip-connection network.
4. The method according to claim 3, characterized in that The step of obtaining the target network according to the baseline network and the skip connection network to be trained includes: Obtain the trained baseline network; Replacing the initial skip connection network in the baseline network with the skip connection network to be trained, so as to update the baseline network to obtain an intermediate network; The intermediate network is trained to obtain the trained target network.
5. The method according to claim 4, characterized in that The step of training the intermediate network to obtain the trained target network includes: The weight parameters of the first network and the second network in the intermediate network are frozen, and the skip-connection network to be trained in the intermediate network is fine-tuned to obtain the trained target network.
6. A skip connection network, characterized in that: The jump connection network is arranged between the first network and the second network, and the jump connection network includes: A latent space encoding and decoding network is used to encode the first high-dimensional feature data obtained from the first network into a low-dimensional latent space representation, and decode the low-dimensional latent space representation into a second high-dimensional feature data after passing it through a skip connection network, so that the second network receives the second high-dimensional feature data.
7. The jump connection network according to claim 6, characterized in that: The latent space encoding and decoding network includes a first encoding network and a first decoding network; The first encoding network is arranged close to the first network, and is used to encode the first high-dimensional feature data obtained from the first network into a low-dimensional latent space representation, and transmit the low-dimensional latent space representation to the first decoding network through a skip connection network; The first decoding network is arranged close to the second network, and is used to receive the low-dimensional latent space representation transmitted via the skip connection network, and decode the low-dimensional latent space representation into second high-dimensional feature data, so as to transmit the second high-dimensional feature data to the second network.
8. A coding and decoding device based on a skip connection network, characterized in that: The skip connection network includes a latent space codec network, and the device includes: An acquisition module, used for acquiring first high-dimensional feature data from the first network; A coding and decoding module is used to encode the first high-dimensional feature data into a low-dimensional latent space representation through the latent space coding and decoding network, and decode the low-dimensional latent space representation into a second high-dimensional feature data after passing it through a skip connection network, so that the second network receives the second high-dimensional feature data; the skip connection network is arranged between the first network and the second network.
9. An electronic device, characterized in that: include: Memory for storing computer programs; A processor, configured to implement the steps of the method according to any one of claims 1 to 5 when executing the computer program.
10. A computer storage medium, characterized in that: The computer storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 5 are implemented.