Data processing methods and new skip connection structures applied to neural networks

Abstract

The present disclosure provides a data processing method and a novel skip connection structure applied to a neural network, which can be applied to the field of artificial intelligence technology. The method includes: reading input data from a storage medium; inputting the input data into a first computing bypass through a processor, so that the input data is calculated in sequence through a first normalization layer, a first fully connected layer, an activation layer, and a second fully connected layer to obtain first intermediate data; inputting the input data into a second computing bypass through a processor, so that the input data is scaled by a first scaling unit and fused with the first intermediate data to obtain second intermediate data; inputting the input data into a main computing branch through a processor, so that the input data is scaled by a second scaling unit and fused with the second intermediate data to obtain output data.

Description

Technical Field

[0001] The present disclosure relates to the field of artificial intelligence technology, and more specifically, to a data processing method, a novel skip connection structure applied to a neural network, a method for machine translation applied to the novel skip connection structure applied to a neural network, and a method for binocular depth estimation applied to the novel skip connection structure applied to a neural network. Background Art

[0002] Currently, artificial intelligence technology is widely used in image classification, object recognition, photo news headline generation, video understanding, and speech recognition. Deep learning-based neural network technology, due to its excellent performance, has also attracted widespread attention across various industries.

[0003] In the process of realizing the concept of the present disclosure, the inventors found that there are at least the following problems in the related art: when performing calculations through models such as machine translation and binocular depth estimation, the computer's computing efficiency is low and data processing cannot be performed efficiently. Summary of the Invention

[0004] In view of this, the present disclosure provides a method for data processing, a novel skip connection structure applied to a neural network, a method for machine translation applied to the novel skip connection structure applied to a neural network, and a method for binocular depth estimation applied to the novel skip connection structure applied to a neural network.

[0005] One aspect of the present disclosure provides a method for data processing based on a novel skip connection structure applied to a neural network, comprising:

[0006] Reading input data from a storage medium, wherein the input data is obtained by processing original technical data in the target technical field using a neural network model applied in the target technical field;

[0007] Inputting input data into a first calculation bypass through a processor so as to sequentially calculate the input data through a first normalization layer, a first fully connected layer, an activation layer, and a second fully connected layer to obtain first intermediate data;

[0008] Inputting the input data into the second calculation bypass through the processor, so that the input data is scaled by the first scaling unit and fused with the first intermediate data to obtain second intermediate data; and

[0009] The processor inputs the input data into the main calculation branch, so that the second scaling unit performs scaling processing on the input data and fuses the data with the second intermediate data to obtain output data.

[0010] Another aspect of the present disclosure provides a novel skip connection structure applied to a neural network, comprising:

[0011] A first calculation bypass is used to calculate the input data through the first normalization layer, the first fully connected layer, the activation layer, and the second fully connected layer in sequence to obtain first intermediate data;

[0012] A second calculation bypass is used to scale the input data using the first scaling unit and fuse the data with the first intermediate data to obtain second intermediate data; and

[0013] The main computing branch is used to perform scaling processing on the input data through the second scaling unit and fuse it with the second intermediate data to obtain output data.

[0014] According to an embodiment of the present disclosure, the first scaling unit is provided with a first hyperparameter, the first hyperparameter being used to characterize a scaling factor for scaling input data passing through the second computation bypass; and

[0015] The second scaling unit is provided with a second hyperparameter, and the second hyperparameter is used to represent a scaling coefficient for scaling the input data passing through the main computing branch.

[0016] According to an embodiment of the present disclosure, the novel skip connection structure applied to the neural network further includes:

[0017] The second normalization layer is used to perform normalization processing on the second intermediate data.

[0018] Another aspect of the present disclosure provides a method for performing machine translation using the novel skip connection structure applied to a neural network, comprising:

[0019] Preprocessing the original sentence data to obtain word vector data, wherein the original sentence data is used to represent the original sentence to be translated;

[0020] Input the word vector data into the multi-head attention layer of the pre-trained machine translation model, and output the first mapping relationship table data of the word vector data, wherein the first mapping relationship table data is used to represent the mapping relationship between the original sentence and the target language;

[0021] Inputting the first mapping relationship table data as input data into the novel skip connection structure in the machine translation model, and outputting output data;

[0022] The output data is activated to obtain target data, wherein the target data is used to represent a target sentence corresponding to the original sentence.

[0023] According to an embodiment of the present disclosure, before activating the output data to obtain the target data, the machine translation method further includes:

[0024] Perform linear processing on the output data to output the output data linearly.

[0025] According to an embodiment of the present disclosure, before inputting the word vector data into the multi-head attention layer in the pre-trained machine translation model and outputting the first mapping relationship table data of the word vector data, the above-mentioned machine translation method further includes:

[0026] Perform position encoding on the word vector data to determine the word position in the original sentence.

[0027] Another aspect of the present disclosure provides a method for binocular depth estimation using the novel skip connection structure applied to a neural network, comprising:

[0028] Obtain left feature map data and right feature map data, wherein the left feature map data is used to represent the pixel points in the left image taken by the binocular camera, and the right feature map data is used to represent the pixel points in the right image taken by the binocular camera;

[0029] Input the left feature map data and the right feature map data into a pre-trained binocular depth estimation model, and output the first distance data between the pixel point in the left image and the target object, and the second distance data between the pixel point in the right image and the target object, respectively. The binocular depth estimation model includes a novel skip connection structure, and the target object is an object captured by the binocular camera.

[0030] A depth map of the target object is determined according to the first distance data and the second distance data, wherein the depth map records the distance between the target object and the binocular camera.

[0031] According to an embodiment of the present disclosure, the method for binocular depth estimation using the novel skip connection structure applied to the neural network further includes:

[0032] The binocular depth estimation model includes at least one computing layer, wherein a novel skip connection structure is set in each computing layer.

[0033] According to an embodiment of the present disclosure, a method for training a binocular depth estimation model includes:

[0034] Acquire a training sample, wherein the training sample includes a first data sample and a second data sample, the first data sample represents distance data between the target object and the binocular camera, and the second data sample is used to represent left feature map data and right feature map data;

[0035] The first data sample is used as a label and the second data sample is used as an input to train an untrained binocular depth estimation model to obtain a binocular depth estimation model.

[0036] According to an embodiment of the present disclosure, input data is read from a storage medium, first intermediate data is calculated by a first computation bypass, and second intermediate data is obtained by fusing the data with the input data scaled by a second computation bypass. The second intermediate data is then fused with the input data scaled by the main computation branch to obtain output data. Input data is obtained by processing the original technical data in the target technical field. The input data is not scaled by the first computation bypass, thereby increasing the accuracy of the neural network model. By adding a scaling unit through the second computation bypass and the main computation branch, more input data can be introduced into the output, enabling the computer to process data more efficiently and improving the computer's computational efficiency. Therefore, the technical problem of low computational efficiency and inability to efficiently process data when performing calculations through models such as machine translation and binocular depth estimation is at least partially overcome. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] The above and other objects, features and advantages of the present disclosure will become more apparent through the following description of the embodiments of the present disclosure with reference to the accompanying drawings, in which:

[0038] Figure 1 A flowchart schematically illustrates a method for data processing based on a novel skip connection structure applied to a neural network according to an embodiment of the present disclosure;

[0039] Figure 2 A schematic diagram schematically illustrates a novel skip connection structure applied to a neural network according to an embodiment of the present disclosure;

[0040] Figure 3 A schematic diagram schematically illustrates a method for performing machine translation using a novel skip connection structure applied to a neural network according to an embodiment of the present disclosure; and

[0041] Figure 4 A schematic diagram of a method for binocular depth estimation using a novel skip connection structure applied to a neural network according to an embodiment of the present disclosure is schematically shown. DETAILED DESCRIPTION

[0042] Hereinafter, embodiments of the present disclosure will be described with reference to the accompanying drawings. However, it should be understood that these descriptions are merely exemplary and are not intended to limit the scope of the present disclosure. In the detailed description below, for ease of explanation, many specific details are set forth to provide a comprehensive understanding of the embodiments of the present disclosure. However, it is apparent that one or more embodiments may also be implemented without these specific details. In addition, in the following description, descriptions of well-known structures and technologies are omitted to avoid unnecessary confusion of the concepts of the present disclosure.

[0043] The terms used herein are only for describing specific embodiments and are not intended to limit the present disclosure. The terms "comprise," "include," etc. used herein indicate the presence of the features, steps, operations, and / or components, but do not exclude the presence or addition of one or more other features, steps, operations, or components.

[0044] All terms used herein (including technical and scientific terms) have the meanings commonly understood by those skilled in the art unless otherwise defined. It should be noted that the terms used herein should be interpreted as having a meaning consistent with the context of this specification and should not be interpreted in an idealized or overly rigid manner.

[0045] When expressions such as “at least one of A, B, and C, etc.” are used, they should generally be interpreted in accordance with the meaning commonly understood by those skilled in the art (e.g., “a system having at least one of A, B, and C” should include but is not limited to systems having A alone, B alone, C alone, A and B, A and C, B and C, and / or A, B, and C, etc.). When expressions such as “at least one of A, B, or C, etc.” are used, they should generally be interpreted in accordance with the meaning commonly understood by those skilled in the art (e.g., “a system having at least one of A, B, or C” should include but is not limited to systems having A alone, B alone, C alone, A and B, A and C, B and C, and / or A, B, and C, etc.).

[0046] In related technologies, neural networks generally use a feedforward layer subnetwork with a nonlinear activation function to introduce nonlinearity. However, nonlinear operations can cause optimization difficulties. To solve this problem, the neural network can be alleviated by propagating a linear component between the input and output. The linear component can scale the input data according to a predetermined value, thereby regulating the output data. This can be achieved by having a linear skip connection structure. The skip connection structure transmits the input data of the neural network to the output through an identity mapping and fuses it with the output of other branches. This operation allows the input component in the output to continue reasoning about the neural network even if erroneous data passes through the nonlinear unit. Common skip connections, such as traditional residual units, are directly integrated into the feedforward layer propagation of the neural network and do not scale the input. By utilizing skip connections in neural networks, the computational performance of the neural network and the computational efficiency of the computer are optimized, so that the expressive power of the neural network can be fully utilized.

[0047] The Transformer model, which has achieved significant progress in binocular depth estimation and machine translation tasks, has a skip connection structure in its feedforward layer. This improves the performance and convergence of the Transformer model during training, resolving the problem of poor gradient performance in deep Transformer structures. Consequently, skip connections are widely used in Transformers. However, existing skip connection structures are simple and lack performance, resulting in low computational efficiency and inefficient data processing.

[0048] In view of this, an embodiment of the present disclosure provides a method for data processing based on a novel skip connection structure applied to a neural network. The method includes reading input data from a storage medium, wherein the input data is obtained by processing original technical data in the target technical field using a neural network model applied in the target technical field; inputting the input data into a first computing bypass through a processor so as to sequentially calculate the input data through a first normalization layer, a first fully connected layer, an activation layer, and a second fully connected layer to obtain first intermediate data; inputting the input data into a second computing bypass through a processor so as to scale the input data through a first scaling unit and fuse it with the first intermediate data to obtain second intermediate data; and inputting the input data into a main computing branch through a processor so as to scale the input data through a second scaling unit and fuse it with the second intermediate data to obtain output data.

[0049] Figure 1 The flowchart of the method for data processing based on the novel skip connection structure applied to the neural network according to an embodiment of the present disclosure is schematically shown.

[0050] like Figure 1 As shown, the method includes operations S101 to S104.

[0051] In operation S101 , input data is read from a storage medium, wherein the input data is obtained by processing original technical data in the target technical field using a neural network model applied in the target technical field.

[0052] According to an embodiment of the present disclosure, the storage medium may be a read-only memory (ROM) or a random access memory (RAM).

[0053] According to an embodiment of the present disclosure, the target technology field may be a technology field in which a neural network model is applied, such as the image classification technology field, the object recognition technology field, the picture news title generation technology field, the video understanding technology field, the speech recognition technology field, and the like.

[0054] According to an embodiment of the present disclosure, the neural network model may be a model used in the target technical field, for example, it may be a neural network model for image classification, a neural network model for object recognition, a neural network model for picture news headline generation, a neural network model for video understanding, a neural network model for speech recognition, and the like.

[0055] According to an embodiment of the present disclosure, the original technical data may be image data obtained in the field of image classification technology, object data obtained in the field of object recognition technology, picture news data obtained in the field of picture news title generation technology, video data obtained in the field of video understanding technology, voice data obtained in the field of voice recognition technology, and so on.

[0056] According to an embodiment of the present disclosure, input data may be read from a storage medium through program instructions.

[0057] In operation S102, the processor inputs input data into the first calculation bypass so as to sequentially calculate the input data through the first normalization layer, the first fully connected layer, the activation layer, and the second fully connected layer to obtain first intermediate data.

[0058] According to an embodiment of the present disclosure, the processor may include a general-purpose microprocessor (e.g., a CPU), an instruction set processor and / or a related chipset and / or a special-purpose microprocessor (e.g., an application-specific integrated circuit (ASIC)), etc. The processor may also include onboard memory for caching purposes. The processor may include a single processing unit or multiple processing units for executing different actions of the method flow according to an embodiment of the present disclosure.

[0059] According to an embodiment of the present disclosure, the first normalization layer can adjust the data format size in the input data to be the same to facilitate subsequent calculations.

[0060] According to embodiments of the present disclosure, the first fully connected layer, the activation layer, and the second fully connected layer may constitute a feedforward layer. The first fully connected layer may perform a fully connected calculation on the data output by the first normalization layer. The activation layer may perform an activation operation using a ReLU activation function. The second fully connected layer may perform a fully connected calculation on the data output by the activation layer, ultimately obtaining first intermediate data.

[0061] According to an embodiment of the present disclosure, the first computation bypass can operate on the input data so that the transmission of the input data is within a reasonable range, avoiding the gradient vanishing and exploding problems.

[0062] In operation S103 , the processor inputs the input data into the second calculation bypass, so that the first scaling unit performs scaling processing on the input data and fuses the data with the first intermediate data to obtain second intermediate data.

[0063] According to an embodiment of the present disclosure, the first scaling unit can perform a multiplication operation, and the second calculation can scale the input data through the multiplication operation of the first scaling unit, and obtain the second intermediate data by fusing it with the first intermediate data through an addition operation.

[0064] In operation S104, the processor inputs the input data into the main computing branch, so that the second scaling unit performs scaling processing on the input data and fuses the data with the second intermediate data to obtain output data.

[0065] According to an embodiment of the present disclosure, the second scaling unit can perform a multiplication operation, and the main computing branch can scale the input data through the multiplication operation of the second scaling unit, and obtain output data by fusing it with the second intermediate data through an addition operation.

[0066] According to an embodiment of the present disclosure, input data is read from a storage medium, first intermediate data is calculated by a first computation bypass, and second intermediate data is obtained by fusing the data with the input data scaled by a second computation bypass. The second intermediate data is then fused with the input data scaled by the main computation branch to obtain output data. Input data is obtained by processing the original technical data in the target technical field. The input data is not scaled by the first computation bypass, thereby increasing the accuracy of the neural network model. By adding a scaling unit through the second computation bypass and the main computation branch, more input data can be introduced into the output, enabling the computer to process data more efficiently and improving the computer's computational efficiency. Therefore, the technical problem of low computational efficiency and inability to efficiently process data when performing calculations through models such as machine translation and binocular depth estimation is at least partially overcome.

[0067] Figure 2 A schematic diagram of a novel skip connection structure applied to a neural network according to an embodiment of the present disclosure is schematically shown.

[0068] like Figure 2 As shown, the novel skip connection structure includes a first computation bypass 201 , a second computation bypass 202 and a main computation branch 203 .

[0069] The first calculation bypass 201 is used to calculate the input data through the first normalization layer, the first fully connected layer, the activation layer and the second fully connected layer in sequence to obtain first intermediate data.

[0070] The second calculation bypass 202 is configured to perform scaling processing on the input data through the first scaling unit, and fuse the scaling processing with the first intermediate data to obtain second intermediate data.

[0071] The main computing branch 203 is configured to perform scaling processing on the input data through the second scaling unit, and fuse the data with the second intermediate data to obtain output data.

[0072] According to an embodiment of the present disclosure, the first normalization layer can adjust the data format size in the input data to be the same to facilitate subsequent calculations.

[0073] According to embodiments of the present disclosure, the first fully connected layer, the activation layer, and the second fully connected layer may constitute a feedforward layer. The first fully connected layer may perform a fully connected calculation on the data output by the first normalization layer. The activation layer may perform an activation operation using a ReLU activation function. The second fully connected layer may perform a fully connected calculation on the data output by the activation layer, ultimately obtaining first intermediate data.

[0074] According to an embodiment of the present disclosure, the first computation bypass can operate on the input data so that the transmission of the input data is within a reasonable range, avoiding the gradient vanishing and exploding problems.

[0075] According to an embodiment of the present disclosure, the first scaling unit can perform a multiplication operation, and the second calculation can scale the input data through the multiplication operation of the first scaling unit, and obtain the second intermediate data by fusing it with the first intermediate data through an addition operation.

[0076] According to an embodiment of the present disclosure, the second scaling unit can perform a multiplication operation, and the main computing branch can scale the input data through the multiplication operation of the second scaling unit, and obtain output data by fusing it with the second intermediate data through an addition operation.

[0077] According to an embodiment of the present disclosure, the first scaling unit is provided with a first hyperparameter, and the first hyperparameter is used to characterize a scaling coefficient for scaling the input data passing through the second computation bypass.

[0078] The second scaling unit is provided with a second hyperparameter, and the second hyperparameter is used to represent a scaling coefficient for scaling the input data passing through the main computing branch.

[0079] According to an embodiment of the present disclosure, the first hyperparameter can be represented by λ, and the value of λ can be 1, indicating that the input data passing through the second calculation bypass is scaled by 1 times. The value of λ can take other values ​​as needed and is not limited here.

[0080] According to an embodiment of the present disclosure, the second hyperparameter can be represented by β, and the value of β can be 0.5, indicating that the input data passing through the main computing branch is scaled by 0.5 times. The value of β can take other values ​​as needed and is not limited here.

[0081] According to the embodiments of the present disclosure, when the scaling factor is greater than 1, it may cause a gradient explosion in the neural network model. When the scaling factor is less than 1, it may cause input data transmission errors. However, the first computational bypass in the novel skip connection structure proposed in the embodiments of the present disclosure is not scaled, and the input data can be accurately transmitted to the output, avoiding the propagation of erroneous information.

[0082] According to an embodiment of the present disclosure, the novel skip connection structure applied to the neural network further includes:

[0083] The second normalization layer is used to perform normalization processing on the second intermediate data.

[0084] According to an embodiment of the present disclosure, the second normalization layer may normalize the format and size of the second intermediate data so that the second intermediate data can be fused with the scaled input data of the main computing branch.

[0085] According to an embodiment of the present disclosure, a method for performing machine translation using the novel skip connection structure applied to a neural network is provided, comprising:

[0086] The original sentence data is preprocessed to obtain word vector data, wherein the original sentence data is used to represent the original sentence to be translated.

[0087] The word vector data is input into the multi-head attention layer of a pre-trained machine translation model, and the first mapping relationship table data of the word vector data is output, wherein the first mapping relationship table data is used to represent the mapping relationship between the original sentence and the target language.

[0088] The first mapping relationship table data is input as input data to the novel skip connection structure in the machine translation model, and output data is output.

[0089] The output data is activated to obtain target data, wherein the target data is used to represent a target sentence corresponding to the original sentence.

[0090] Figure 3 A schematic diagram of a method for machine translation using a novel skip connection structure applied to a neural network according to an embodiment of the present disclosure is schematically shown.

[0091] like Figure 3 As shown, the new skip connection structure can be applied to the machine translation model.

[0092] According to an embodiment of the present disclosure, the original sentence data can be converted into a digital form that can be recognized by a machine, and then preprocessed to obtain word vector data of uniform size.

[0093] According to an embodiment of the present disclosure, word vector data is input into a multi-head attention layer. The multi-head attention layer can use the word vector data as a query vector and then obtain first mapping table data. The first mapping table data can be data of the mapping relationship between the original sentence and the target language, and the target language can be the language into which the original sentence is to be translated. For example, if the original sentence is in Chinese, the target language can be English, French, etc.

[0094] According to an embodiment of the present disclosure, the first mapping table data can be input into a novel skip connection structure. The specific processing process can be seen in the description of other parts of the embodiments of the present disclosure and will not be elaborated here.

[0095] According to an embodiment of the present disclosure, the output data obtained from the novel skip connection structure can be processed through activation to obtain target data. For example, when performing the task of translating Chinese into English, inputting the original Chinese sentence "apple" into the machine translation model can output the target language in English "apple".

[0096] According to an embodiment of the present disclosure, before activating the output data to obtain the target data, the above-mentioned machine translation method further includes:

[0097] Performing linear processing on the output data to make the output data linearly output.

[0098] According to an embodiment of the present disclosure, the output data can be pre-processed by setting a linear layer so that the output data can be linearly output.

[0099] According to an embodiment of the present disclosure, before inputting the word vector data into the multi-head attention layer of a pre-trained machine translation model and outputting the first mapping table data of the word vector data, the above-mentioned machine translation method further includes:

[0100] Performing position encoding on the word vector data to determine the word positions in the original sentence.

[0101] According to an embodiment of the present disclosure, when performing machine translation, one word can be translated each time, and position encoding can be performed on the word vector data to determine the position of the word to be translated. For example, in "I like apples", the position encoding of "I" can be the first position, the position encoding of "like" can be the second position, and the position encoding of "apples" can be the third position.

[0102] According to an embodiment of the present disclosure, an experiment was conducted on machine translation using a novel skip connection structure applied to a neural network, and the effect was evaluated using the bilingual evaluation understudy (BLEU) evaluation indicator, as shown in Table 1 below. Machine translation task 1 is a baseline Transformer model, and machine translation task 2 is a machine translation model using the novel skip connection structure applied to a neural network. The experimental results show that the architecture using the novel skip connection structure for machine translation achieved a BLEU score of 33.56 in the German-English translation task of the dataset (IWSLT14), which is 2% higher than the baseline Transformer model. In other words, using the novel skip connection structure for machine translation in a neural network can improve the quality of machine translation.

[0103] Table 1 Comparison of machine translation results

[0104] Task Type BLEU Machine Translation Task 1 32.93 Machine Translation Task 2 33.56

[0105] According to an embodiment of the present disclosure, a method for binocular depth estimation using the novel skip connection structure applied to a neural network is provided, comprising:

[0106] Obtain left feature map data and right feature map data, wherein the left feature map data is used to represent the pixel points in the left image taken by the binocular camera, and the right feature map data is used to represent the pixel points in the right image taken by the binocular camera.

[0107] The left feature map data and the right feature map data are respectively input into a pre-trained binocular depth estimation model, and the first distance data between the pixel point in the left image and the target object and the second distance data between the pixel point in the right image and the target object are output. The binocular depth estimation model includes a new skip connection structure, and the target object is the object photographed by the binocular camera.

[0108] A depth map of the target object is determined according to the first distance data and the second distance data, wherein the depth map records the distance between the target object and the binocular camera.

[0109] According to an embodiment of the present disclosure, the method for binocular depth estimation using the novel skip connection structure applied to the neural network further includes:

[0110] The binocular depth estimation model includes at least one computing layer, wherein a novel skip connection structure is set in each computing layer.

[0111] Figure 4 A schematic diagram of a method for binocular depth estimation using a novel skip connection structure applied to a neural network according to an embodiment of the present disclosure is schematically shown.

[0112] like Figure 4 As shown, a new skip connection structure is added to the existing binocular depth estimation model, and then the left image and the right image taken by the binocular camera are obtained, the left feature map data is determined according to the pixel points of the left image, and the right feature map data is determined according to the pixel points of the right image, so that the left feature map data and the right feature map data are respectively input into the binocular depth estimation model.

[0113] According to an embodiment of the present disclosure, the binocular depth estimation model may include two left and right paths, and the left feature map data and the right feature map data are transmitted to the two paths respectively. The left feature map data passes through the multi-head attention module of the first layer network in the left path, which can use the pixel points in the left image as the query vector to obtain the left side distance mapping table data of the left image. The right feature map data passes through the multi-head attention module of the first layer network in the right path, which can use the pixel points in the right image as the query vector to obtain the right side distance mapping table data. Then, the left distance mapping table data and the right distance mapping table data are respectively passed through the new skip connection feedforward layer. The specific processing process of the new skip connection feedforward layer can be seen in the description of other parts of the embodiment of the present disclosure, and will not be repeated here.

[0114] According to an embodiment of the present disclosure, when performing second-layer network calculations, the left path can use the right distance mapping table of the right path to obtain the left distance mapping table data in the second layer, and the right path can use the left distance mapping table of the left path to obtain the right distance mapping table data in the second layer, and then the left distance mapping table data and the right distance mapping table data in the second layer are respectively passed through the new skip connection feedforward layer, and so on, until the last layer obtains the first distance data between the pixel point in the left image and the target object and the second distance data between the pixel point in the right image and the target object.

[0115] According to an embodiment of the present disclosure, a depth map of a target object photographed by a binocular camera may be determined based on the first distance data and the second distance data, and the depth map may record the distance between the target object and the binocular camera.

[0116] According to an embodiment of the present disclosure, a method for training a binocular depth estimation model includes:

[0117] A training sample is obtained, wherein the training sample includes a first data sample and a second data sample, the first data sample represents distance data between the target object and the binocular camera, and the second data sample is used to represent left feature map data and right feature map data.

[0118] The first data sample is used as a label and the second data sample is used as an input to train an untrained binocular depth estimation model to obtain a binocular depth estimation model.

[0119] According to an embodiment of the present disclosure, the training sample can be obtained through the KITTI dataset. The first data sample can be the distance data between the known target object and the binocular camera.

[0120] According to an embodiment of the present disclosure, an experiment on binocular depth estimation using a new skip connection structure applied to a neural network uses absolute error (EPE) and the percentage of errors greater than 3 pixels (3px Error) as evaluation indicators, and uses intersection over union (IOU) to evaluate occlusion estimation. Experiments are conducted on the DispNet dataset or the FlowNet2.0 dataset. Table 2 shows the results of the binocular depth estimation model with different skip connection structures. Model 1 is a binocular depth estimation model without adding a new skip connection structure, Model 1 is a binocular depth estimation model with a skip connection structure with only one bypass, Model 3 is a binocular depth estimation model with the new skip connection structure proposed in an embodiment of the present disclosure, where the first hyperparameter λ=2 and the second hyperparameter β=0.5, and Model 4 is a binocular depth estimation model with the new skip connection structure proposed in an embodiment of the present disclosure, where the first hyperparameter λ=1 and the second hyperparameter β=0.5. As can be seen from the table, the new skip connection structure applied in Model 4 achieves the best results in EPE, IOU, and 3PX ERROR indicators.

[0121] Table 2 Experimental results of binocular depth estimation model

[0122]

[0123] The flowcharts and block diagrams in the accompanying drawings illustrate the possible architectures, functions, and operations of the systems, methods, and computer program products according to various embodiments of the present disclosure. In this regard, each box in the flowchart or block diagram may represent a module, program segment, or portion of code, which contains one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the boxes may occur in an order different from that marked in the accompanying drawings. For example, two boxes shown in succession may actually be executed substantially in parallel, or they may sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flowchart, as well as the combination of boxes in the block diagram or flowchart, may be implemented using a dedicated hardware-based system that performs the specified function or operation, or may be implemented using a combination of dedicated hardware and computer instructions. It will be understood by those skilled in the art that the features described in the various embodiments and / or claims of the present disclosure may be combined and / or coupled in various ways, even if such combinations or couplings are not explicitly described in the present disclosure. In particular, without departing from the spirit and teachings of the present disclosure, the features described in the various embodiments and / or claims of the present disclosure may be combined and / or coupled in various ways, and all such combinations and / or couplings fall within the scope of the present disclosure.

[0124] The embodiments of the present disclosure are described above. However, these embodiments are for illustrative purposes only and are not intended to limit the scope of the present disclosure. Although each embodiment has been described separately above, this does not mean that the measures in each embodiment cannot be used in combination to advantage. The scope of the present disclosure is defined by the appended claims and their equivalents. Without departing from the scope of the present disclosure, those skilled in the art may make various substitutions and modifications, which should all fall within the scope of the present disclosure.

Claims

1. A method for machine translation using a novel skip connection structure applied to a neural network, comprising: Preprocessing the original sentence data to obtain word vector data, wherein the original sentence data is used to represent the original sentence to be translated; Input the word vector data into a multi-head attention layer in a pre-trained machine translation model, and output first mapping relationship table data of the word vector data, wherein the first mapping relationship table data is used to represent the mapping relationship between the original sentence and the target language; Inputting the first mapping relationship table data as input data into the novel skip connection structure in the machine translation model, and outputting output data; activating the output data to obtain target data, wherein the target data is used to represent a target sentence corresponding to the original sentence; The novel skip connection structure includes: A first calculation bypass is used to calculate the input data through the first normalization layer, the first fully connected layer, the activation layer, and the second fully connected layer in sequence to obtain first intermediate data; a second computing bypass, configured to scale the input data using a first scaling unit and fuse the scaled data with the first intermediate data to obtain second intermediate data, wherein the first scaling unit is provided with a first hyperparameter, and the first hyperparameter is used to represent a scaling factor for scaling the input data passing through the second computing bypass; and a main computing branch, configured to scale the input data using a second scaling unit and fuse the data with the second intermediate data to obtain output data, wherein the second scaling unit is provided with a second hyperparameter, and the second hyperparameter is used to represent a scaling factor for scaling the input data passing through the main computing branch; The second normalization layer is used to perform normalization processing on the second intermediate data.

2. The method according to claim 1, before activating the output data to obtain the target data, further comprising: The output data is linearly processed so as to be output linearly.

3. The method according to claim 1, before inputting the word vector data into a multi-head attention layer in a pre-trained machine translation model and outputting the first mapping relationship table data of the word vector data, further comprising: Position encoding is performed on the word vector data to determine the word position in the original sentence.

4. A method for binocular depth estimation using a novel skip connection structure applied to a neural network, comprising: Obtain left feature map data and right feature map data, wherein the left feature map data is used to represent pixel points in the left image taken by the binocular camera, and the right feature map data is used to represent pixel points in the right image taken by the binocular camera; Inputting the left feature map data and the right feature map data into a pre-trained binocular depth estimation model respectively, and outputting first distance data between the pixel point in the left image and the target object and second distance data between the pixel point in the right image and the target object, wherein the binocular depth estimation model includes the novel skip connection structure, and the target object is an object captured by the binocular camera; Determining a depth map of the target object according to the first distance data and the second distance data, wherein the depth map records the distance between the target object and the binocular camera; The novel skip connection structure includes: A first calculation bypass is used to calculate the input data through the first normalization layer, the first fully connected layer, the activation layer, and the second fully connected layer in sequence to obtain first intermediate data; a second computing bypass, configured to scale the input data using a first scaling unit and fuse the scaled data with the first intermediate data to obtain second intermediate data, wherein the first scaling unit is provided with a first hyperparameter, and the first hyperparameter is used to represent a scaling factor for scaling the input data passing through the second computing bypass; and a main computing branch, configured to scale the input data using a second scaling unit and fuse the data with the second intermediate data to obtain output data, wherein the second scaling unit is provided with a second hyperparameter, and the second hyperparameter is used to represent a scaling factor for scaling the input data passing through the main computing branch; The second normalization layer is used to perform normalization processing on the second intermediate data.

5. The method according to claim 4, further comprising: The binocular depth estimation model includes at least one computing layer, wherein the novel skip connection structure is set in each computing layer.

6. The method according to claim 4 or 5, wherein: The method for training a binocular depth estimation model comprises: Acquire a training sample, wherein the training sample includes a first data sample and a second data sample, the first data sample represents the distance data between the target object and the binocular camera, and the second data sample is used to represent the left feature map data and the right feature map data; An untrained binocular depth estimation model is trained using the first data sample as a label and the second data sample as an input to obtain the binocular depth estimation model.

Images