Multimedia information processing method and device, equipment and medium
By using a combined processing of convolutional layer and standardized layer in multimedia information processing, the problem of quantization accuracy loss in vehicle-machine system is solved, and the accuracy of multimedia information feature extraction is improved.
Patent Information
- Application Number
- CN202310538128.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2023-05-12
- Publication Date
- 2024-11-12
AI Technical Summary
In the prior art, the neural network model loses a large quantization accuracy during the process of quantization to the vehicle-machine system, resulting in a low accuracy in extracting multimedia information features such as video, images and voice.
By obtaining the features of multimedia information, input them to the first convolutional layer and the second convolutional layer for convolutional processing, and the results are added element by element, and then the processed features are input to the standardized layer for normalization processing. Finally, the normalized features are processed preset to obtain the target features of the multimedia information.
It reduces the quantization accuracy loss and improves the accuracy of multimedia information feature extraction.
Smart Images

Figure CN118939990A_ABST
Abstract
Description
Technical Field
[0001] The present disclosure relates to the field of computer technology, and in particular to a multimedia information processing method, device, equipment and medium. Background Art
[0002] With the increasing intelligence of vehicles, the vehicle computer system in the vehicle has the function of efficiently and accurately identifying and processing the video, images, voice, etc. obtained by the vehicle. However, when the vehicle computer system efficiently and accurately identifies and processes the video, images, voice, etc. obtained by the vehicle, it mainly relies on the neural network model with video, image, voice and other processing functions installed in the vehicle computer system.
[0003] Among them, the trained neural network model cannot be directly deployed on the vehicle system. It needs to be quantized by the quantization system installed on the vehicle system and converted into a format supported by the vehicle system. During the quantization process of the trained neural network model, the vehicle system has different support for neural network models with different structures. Therefore, the quantization accuracy loss of neural network models with different structures in the vehicle system during the quantization process is also different.
[0004] The neural network model in the prior art suffers from a large loss in quantization accuracy during quantization to the vehicle computer system, resulting in low accuracy in feature extraction of multimedia information such as video, image and voice by the neural network model. Summary of the invention
[0005] In order to solve the above technical problems, the present disclosure provides a multimedia information processing method, device, equipment and medium.
[0006] A first aspect of an embodiment of the present disclosure provides a multimedia information processing method, the method comprising:
[0007] Acquire a first feature of the multimedia information to be processed;
[0008] Input the first feature to the first convolution layer to obtain a first convolution result, and input the first feature to the second convolution layer to obtain a second convolution result. The first convolution layer and the second convolution layer are respectively used to perform convolution processing on the first feature;
[0009] Add the first convolution result and the second convolution result element by element to obtain the second feature;
[0010] Inputting the second feature into the first batch of normalization layers for normalization processing to obtain the normalized second feature, and the first batch of normalization layers is used to normalize the second feature;
[0011] The normalized second feature and the representation vector of the multimedia information are preset to obtain the target feature of the multimedia information.
[0012] A second aspect of the embodiments of the present disclosure provides a multimedia information processing device, the device comprising:
[0013] A first acquisition module, used to acquire a first feature of the multimedia information to be processed;
[0014] A convolution processing module, used for inputting the first feature into a first convolution layer to obtain a first convolution result, and inputting the first feature into a second convolution layer to obtain a second convolution result, wherein the first convolution layer and the second convolution layer are respectively used for performing convolution processing on the first feature;
[0015] A second acquisition module is used to add the first convolution result and the second convolution result element by element to obtain a second feature;
[0016] A normalization processing module, used for inputting the second feature into the first batch of normalization layers for normalization processing to obtain the normalized second feature, and the first batch of normalization layers is used for normalizing the second feature;
[0017] The third acquisition module is used to perform preset processing on the normalized second feature and the representation vector of the multimedia information to obtain the target feature of the multimedia information.
[0018] A third aspect of an embodiment of the present disclosure provides an electronic device, the device comprising:
[0019] Memory;
[0020] Processor; and
[0021] A computer program, wherein the computer program is stored in a memory and is configured to be executed by a processor to implement the multimedia information processing method of the first aspect as described above.
[0022] A fourth aspect of an embodiment of the present disclosure provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the multimedia information processing method of the first aspect described above is implemented.
[0023] Compared with the prior art, the technical solution provided by the embodiments of the present disclosure has the following advantages:
[0024] The multimedia information processing method, device, equipment and medium provided by the embodiments of the present disclosure can obtain a first feature of the multimedia information to be processed, input the first feature to the first convolution layer, and obtain a first convolution result, wherein the first convolution layer and the second convolution layer are respectively used to perform convolution processing on the first feature, and input the first feature to the second convolution layer to obtain a second convolution result, add the first convolution result and the second convolution result element by element to obtain a second feature, input the second feature to the first batch of standardization layers for normalization processing to obtain the normalized second feature, the first batch of standardization layers are used to normalize the second feature, and preset processing is performed on the normalized second feature and the representation vector of the multimedia information to obtain the target feature of the multimedia information, thereby, the first convolution result and the second convolution result respectively obtained by the first convolution layer and the second convolution layer can be added element by element to obtain the second feature, and then the second feature is input to the first batch of standardization layers for normalization processing, which reduces the quantization accuracy loss and thereby improves the accuracy of multimedia information feature extraction. BRIEF DESCRIPTION OF THE DRAWINGS
[0025] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present disclosure and, together with the description, serve to explain the principles of the present disclosure.
[0026] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.
[0027] Figure 1 is a flowchart of a multimedia information processing method provided by an embodiment of the present disclosure;
[0028] Figure 2-1 is a structural schematic diagram of a neural network model provided by an embodiment of the present disclosure;
[0029] Figure 2-2 is a structural schematic diagram of another neural network model provided by an embodiment of the present disclosure;
[0030] Figure 3 is a structural diagram of another neural network model provided by an embodiment of the present disclosure;
[0031] Figure 4 is a structural schematic diagram of a multimedia information processing device provided by an embodiment of the present disclosure;
[0032] Figure 5 It is a structural schematic diagram of an electronic device provided by an embodiment of the present disclosure. DETAILED DESCRIPTION
[0033] In order to more clearly understand the above-mentioned objectives, features and advantages of the present disclosure, the scheme of the present disclosure will be further described below. It should be noted that the embodiments of the present disclosure and the features in the embodiments can be combined with each other without conflict.
[0034] In the following description, many specific details are set forth to facilitate a full understanding of the present disclosure, but the present disclosure may also be implemented in other ways different from those described herein; it is obvious that the embodiments in the specification are only part of the embodiments of the present disclosure, rather than all of the embodiments.
[0035] It should be understood that the various steps described in the method embodiments of the present disclosure may be performed in different orders and / or in parallel. In addition, the method embodiments may include additional steps and / or omit the steps shown. The scope of the present disclosure is not limited in this respect.
[0036] It should be noted that, in this article, relational terms such as "first" and "second" are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.
[0037] It should be noted that the modifications of "one" and "plurality" mentioned in the present disclosure are illustrative rather than restrictive, and those skilled in the art should understand that unless otherwise clearly indicated in the context, it should be understood as "one or more".
[0038] Generally speaking, the neural network model in the prior art suffers from a large loss of quantization accuracy during quantization to the vehicle system, resulting in low accuracy in feature extraction of multimedia information such as video, image, and voice by the neural network model. To address this problem, the present disclosure provides a multimedia information processing method, which is described below in conjunction with a specific embodiment.
[0039] Figure 1It is a flowchart of a multimedia information processing method provided by an embodiment of the present disclosure. The method can be executed by a multimedia information processing device. The multimedia information processing device can be implemented in software and / or hardware. The multimedia information processing device can be configured in an electronic device, such as a server or a terminal or a server cluster, wherein the terminal can specifically include a mobile phone, a computer or a tablet computer, a vehicle-mounted terminal, or any device that can be used to process the multimedia information processing method.
[0040] The multimedia information processing method provided by the embodiment of the present disclosure is applicable to a neural network model, which includes a first convolutional layer, a second convolutional layer and a first batch of standardization layers, wherein the first convolutional layer and the second convolutional layer are respectively connected to the first batch of standardization layers, wherein the neural network model can be any network model that can process multimedia information, such as feature extraction, and is not limited here.
[0041] like Figure 1 As shown, the multimedia information processing method provided by the embodiment of the present disclosure includes the following steps.
[0042] S110: Acquire a first feature of the multimedia information to be processed.
[0043] In the embodiment of the present disclosure, when the electronic device needs to process multimedia information, it obtains the first feature of the multimedia information to be processed.
[0044] In the embodiments of the present disclosure, multimedia information may include video, image, text, voice, etc., which is not limited here.
[0045] Optionally, the first feature may be a feature obtained after performing at least one feature extraction on the multimedia information. For example, the first feature may be a feature obtained by extracting features by inputting the multimedia information into a convolutional layer, or may be a feature obtained by further extracting features by inputting the multimedia information into a convolutional layer for feature extraction. There is no limitation here.
[0046] S120, inputting the first feature into the first convolution layer to obtain a first convolution result, and inputting the first feature into the second convolution layer to obtain a second convolution result, the first convolution layer and the second convolution layer are respectively used to perform convolution processing on the first feature.
[0047] In the disclosed embodiment, after acquiring the first feature of the multimedia information to be processed, the electronic device inputs the first feature into the first convolution layer to obtain a first convolution result, and inputs the first feature into the second convolution layer to obtain a second convolution result.
[0048] The first convolution layer and the second convolution layer are respectively used to perform convolution processing on the first feature.
[0049] Optionally, the first convolution layer and the second convolution layer may be convolution layers with the same convolution kernel or convolution layers with different convolution kernels. For example, the first convolution layer may be a convolution layer with a convolution kernel of 3×3, and the second convolution layer may be a convolution layer with a convolution kernel of 1×1 or a 3×3 convolution layer.
[0050] S130, adding the first convolution result and the second convolution result element by element to obtain a second feature.
[0051] In the embodiment of the present disclosure, after obtaining the first convolution result and the second convolution result, the electronic device adds the first convolution result and the second convolution result element by element to obtain the second feature.
[0052] In the embodiment of the present disclosure, element-by-element addition uses the symbol Perform calculations.
[0053] Optionally, when the first convolution result and the second convolution result are of the same dimension, element-by-element addition is performed directly; when the first convolution result and the second convolution result are of different dimensions, the first convolution result and the second convolution result are first dimensionally matched and then element-by-element addition is performed.
[0054] S140, inputting the second feature into the first batch of normalization layers for normalization processing to obtain the normalized second feature, and the first batch of normalization layers are used to normalize the second feature.
[0055] In the disclosed embodiment, after obtaining the second feature, the electronic device inputs the second feature into the first batch of standardization layers for normalization processing to obtain the normalized second feature.
[0056] Among them, the role of the batch normalization (Batch Normalization, BN) layer is to standardize or normalize the input features.
[0057] S150, performing preset processing on the normalized second feature and the representation vector of the multimedia information to obtain a target feature of the multimedia information.
[0058] In the disclosed embodiment, after obtaining the normalized second feature, the electronic device performs preset processing on the normalized second feature and the representation vector of the multimedia information to obtain the target feature of the multimedia information.
[0059] The representation vector of the multimedia information can be understood as a vector obtained based on the original features of the multimedia information, namely, the representation vector.
[0060] Specifically, after obtaining the normalized second feature, the electronic device performs preset processing on the normalized second feature and the representation vector of the multimedia information to obtain the target feature of the multimedia information.
[0061] Optionally, the preset processing may include at least one of feature extraction, element-by-element addition, mapping processing, etc.
[0062] In the disclosed embodiment, a first feature of the multimedia information to be processed can be obtained, and the first feature is input into a first convolution layer to obtain a first convolution result, wherein the first convolution layer and the second convolution layer are respectively used to perform convolution processing on the first feature, and the first feature is input into the second convolution layer to obtain a second convolution result, the first convolution result and the second convolution result are added element by element to obtain a second feature, and the second feature is input into a first batch of standardization layers for normalization processing to obtain a normalized second feature, the first batch of standardization layers are used to normalize the second feature, and the normalized second feature and the representation vector of the multimedia information are preset to obtain a target feature of the multimedia information, thereby, the first convolution result and the second convolution result respectively obtained by the first convolution layer and the second convolution layer can be added element by element to obtain the second feature, and then the second feature is input into the first batch of standardization layers for normalization processing, which reduces the quantization precision loss and thereby improves the accuracy of feature extraction of multimedia information.
[0063] For example, Figure 2-1 is a schematic diagram of the structure of a neural network model provided by an embodiment of the present disclosure, such as Figure 2-1 As shown, the first feature of the acquired multimedia information is input into the first convolution layer and the second convolution layer respectively, the first convolution results and the second convolution results respectively obtained are added element by element, and the second feature obtained after the element-by-element addition is input into the first batch of standardization layers for normalization processing to obtain the normalized second feature, wherein the first convolution layer can be a 3×3 convolution layer, and the second convolution layer can be a 1×1 convolution layer.
[0064] Based on the above-mentioned embodiment of the present invention, the neural network model also includes a third convolutional layer, a second batch of normalization layers and a preset activation function, and the third convolutional layer is the upper layer of the second batch of normalization layers.
[0065] In the embodiment of the present disclosure, the third convolution layer may be a convolution layer with a convolution kernel size of 3×3, or may be a convolution layer with other convolution kernel sizes, which is not limited here.
[0066] Optionally, the preset activation function may be a ReLU function.
[0067] For example, Figure 3is a structural diagram of another neural network model provided by an embodiment of the present disclosure, such as Figure 3 The schematic diagram of the neural network model shown on the left shows that the third convolutional layer is above the second batch normalization layer, there is a preset activation function after the second batch normalization layer, and the first and second convolutional layers are below the preset activation function.
[0068] Among them, obtaining the first feature of the multimedia information to be processed in S110 can specifically include: obtaining input information of the third convolutional layer, the input information is obtained according to the representation vector of the multimedia information; normalizing the output information of the third convolutional layer according to the second batch of normalization layers to obtain the output information of the second batch of normalization layers; processing the output information of the second batch of normalization layers by a preset activation function to obtain the first feature of the multimedia information to be processed.
[0069] In the embodiment of the present disclosure, the input information of the third convolutional layer may be a representation vector of the multimedia information, and the output information of the third convolutional layer may be a feature vector obtained by extracting features from the representation vector of the multimedia information.
[0070] The input information of the third convolutional layer may also be a feature vector obtained by extracting features from the representation vector of the multimedia information, and the output information of the third convolutional layer may be a target feature vector obtained by extracting features from the feature vector obtained by extracting features from the representation vector.
[0071] In some embodiments of the present disclosure, the input information of the third convolutional layer may be a representation vector of multimedia information, the representation vector of the multimedia information is convolved in the third convolutional layer to obtain output information of the third convolutional layer, the output information of the third convolutional layer is input into the second batch standardization layer for normalization to obtain output information of the second batch standardization layer, and then the output information of the second batch standardization layer is mapped through a preset activation function to obtain the first feature of the multimedia information to be processed.
[0072] In some other embodiments of the present disclosure, the input information of the third convolutional layer may also be a feature vector obtained by extracting features from a representation vector of multimedia information. The feature vector obtained by extracting features from the representation vector of multimedia information is convolved in the third convolutional layer to obtain output information of the third convolutional layer. The output information of the third convolutional layer is input into the second batch of normalization layers for normalization to obtain output information of the second batch of normalization layers. The output information of the second batch of normalization layers is then mapped through a preset activation function to obtain the first feature of the multimedia information to be processed.
[0073] In the disclosed embodiment, the first feature can be obtained through the representation vector of multimedia information, so that the obtained first feature is more accurate. At the same time, the first feature obtained is different based on the different input information of the third convolutional layer, which improves the flexibility of obtaining the first feature.
[0074] Furthermore, in S130, the first convolution result and the second convolution result are added element by element to obtain the second feature, which may specifically include: the first convolution result, the second convolution result and the first feature are added element by element to obtain the second feature.
[0075] Figure 2-2 is a schematic diagram of the structure of another neural network model provided by an embodiment of the present disclosure, such as Figure 2-2 As shown, the first feature is introduced into the Shenjiang network model, and the two-branch feature extraction structure including the first convolution layer and the second convolution layer is expanded to a three-branch feature extraction structure. The first convolution result, the second convolution result and the first feature are added element by element to obtain the second feature, thereby improving the accuracy of the second feature, and further improving the accuracy of the feature extraction results obtained by the neural network model.
[0076] In the embodiment of the present disclosure, in S140, the normalized second feature and the representation vector of the multimedia information are preset to obtain the target feature of the multimedia information, which may specifically include: mapping the normalized second feature through a preset activation function to obtain a third feature; and obtaining the target feature of the multimedia information based on the third feature and the representation vector of the multimedia information.
[0077] In the embodiment of the present disclosure, mapping the normalized second feature through a preset activation function can increase nonlinear features and further improve the robustness of the neural network model.
[0078] like Figure 3 The schematic diagram of the neural network model shown on the left side shows that after obtaining the normalized second feature, the electronic device maps the second feature to the normalized second feature through a preset activation function to obtain a third feature, and then obtains the target feature of the multimedia information based on the third feature and the representation vector of the multimedia information.
[0079] In some embodiments of the present disclosure, obtaining a target feature of the multimedia information based on the third feature and the representation vector of the multimedia information may specifically include: adding the third feature and the representation vector of the multimedia information element by element to obtain the target feature of the multimedia information, thereby improving the accuracy of the obtained target feature.
[0080] In some other embodiments of the present disclosure, obtaining a target feature of the multimedia information based on the third feature and the representation vector of the multimedia information may specifically include: performing feature extraction on the representation vector of the multimedia information to obtain a fourth feature; and adding the third feature and the fourth feature element by element to obtain the target feature of the multimedia information, thereby increasing the rate of obtaining the target feature.
[0081] Exemplarily, the neural network model in the embodiment of the present disclosure can be used in a Qualcomm Snapdragon Neural Processing Engine (SNPE). When the neural network model with a RepResBlock structure is quantized by the quantization system in the SNPE, the SNPE has better support for the neural network model through a preset activation function, thereby improving the running rate. At the same time, by adding the first convolution result and the second convolution result element by element and then normalizing them, the quantization system in the SNPE quantizes the neural network model with a RepResBlock structure, and the quantization precision loss is reduced, thereby improving the accuracy of multimedia information feature extraction.
[0082] In the embodiments of the present disclosure, the neural network model can be simplified to further improve the running speed of the neural network model when extracting features from multimedia information. Figure 3 As shown, the neural network model is transformed from Figure 3 The neural network model diagram shown on the left is simplified to Figure 3 A schematic diagram of the neural network model shown on the right.
[0083] In the disclosed embodiment, the electronic device can obtain a fourth convolutional layer by using a preset fusion method and a reverse fusion of the third convolutional layer and the second batch normalization layer, wherein the preset fusion method is similar to the existing method of fusing the convolutional layer and the batch normalization layer, and will not be described in detail here. Furthermore, the first convolutional layer, the second convolutional layer, the first feature branch, the element-by-element addition and the first batch normalization layer are fused to obtain a fifth convolutional layer, and then multimedia information processing such as feature extraction is performed based on the simplified neural network model, thereby improving the accuracy of feature extraction of media information.
[0084] Exemplarily, the first convolution layer is a convolution layer with a convolution kernel of 3×3, and the second convolution layer is a convolution layer with a convolution kernel of 1×1. The second convolution layer is first expanded to a 3×3 convolution layer. Further, the first feature branch is converted into a 1×1 convolution, and then the 1×1 convolution obtained by the conversion of the first feature branch is expanded to a 3×3 convolution layer. Then, based on the parameters of the first convolution layer, the second convolution layer and the first feature branch, the 3×3 convolution layer obtained by expanding the first convolution layer, the second convolution layer, and the 3×3 convolution layer obtained by expanding the first feature branch are fused, wherein the element-by-element addition operation of the three-branch feature extraction structure is fused in the fusion process. Finally, the fused 3×3 convolution layer is further fused with the first batch of standardized layers to obtain the fifth convolution layer. Thus, a simplified neural network model is obtained, which improves the running rate of the neural network model when processing multimedia information.
[0085] Figure 4 : is a schematic diagram of the structure of a multimedia information processing device provided in an embodiment of the present disclosure. The multimedia information processing device in the embodiment of the present disclosure can be set in an electronic device, and the electronic device can be a server or a terminal or a server cluster, wherein the terminal can specifically include a mobile phone, a computer or a tablet computer, a vehicle-mounted terminal, or any device that can be used to process the multimedia information processing method, etc., which is not limited here.
[0086] like Figure 4 As shown, the multimedia information processing device 400 may include a first acquisition module 410 , a convolution processing module 420 , a second acquisition module 430 , a normalization processing module 440 and a third acquisition module 450 .
[0087] The first acquisition module 410 may be used to acquire a first feature of the multimedia information to be processed.
[0088] The convolution processing module 420 can be used to input the first feature into the first convolution layer to obtain a first convolution result, and input the first feature into the second convolution layer to obtain a second convolution result. The first convolution layer and the second convolution layer are respectively used to perform convolution processing on the first feature.
[0089] The second acquisition module 430 may be configured to perform element-by-element addition on the first convolution result and the second convolution result to obtain a second feature.
[0090] The normalization processing module 440 can be used to input the second feature into the first batch of normalization layers for normalization processing to obtain the normalized second feature. The first batch of normalization layers are used to normalize the second feature.
[0091] The third acquisition module 450 may be used to perform preset processing on the normalized second feature and the representation vector of the multimedia information to obtain the target feature of the multimedia information.
[0092] In the disclosed embodiment, a first feature of the multimedia information to be processed can be obtained, the first feature is input into a first convolution layer to obtain a first convolution result, and the first feature is input into a second convolution layer to obtain a second convolution result, wherein the first convolution layer and the second convolution layer are respectively used to perform convolution processing on the first feature, the first convolution result and the second convolution result are added element by element to obtain a second feature, the second feature is input into a first batch of standardization layers for normalization processing to obtain a normalized second feature, the first batch of standardization layers are used to normalize the second feature, the normalized second feature and the representation vector of the multimedia information are preset to obtain a target feature of the multimedia information, thereby, the first convolution result and the second convolution result respectively obtained by the first convolution layer and the second convolution layer can be added element by element to obtain the second feature, and then the second feature is input into the first batch of standardization layers for normalization processing, which reduces the quantization precision loss and thereby improves the accuracy of feature extraction of multimedia information.
[0093] In some embodiments of the present disclosure, the neural network model also includes a third convolutional layer, a second batch of normalization layers and a preset activation function, and the third convolutional layer is the upper layer of the second batch of normalization layers.
[0094] In some embodiments of the present disclosure, the first acquisition module 410 may include a first acquisition unit 4101 , a second acquisition unit 4102 , and a feature acquisition unit 4103 .
[0095] The first acquisition unit 4101 can be used to obtain input information of the third convolutional layer, where the input information is obtained based on the representation vector of the multimedia information.
[0096] The second acquisition unit 4102 can be used to normalize the output information of the third convolutional layer according to the second batch of normalization layers to obtain the output information of the second batch of normalization layers.
[0097] The feature acquisition unit 4103 can be used to process the output information of the second batch of normalization layers through a preset activation function to obtain the first feature of the multimedia information to be processed.
[0098] In some embodiments of the present disclosure, the second acquisition module 430 may be specifically configured to perform element-by-element addition on the first convolution result, the second convolution result, and the first feature to obtain the second feature.
[0099] In some embodiments of the present disclosure, the third acquisition module 450 may include a mapping processing unit 4501 and a third acquisition unit 4502 .
[0100] The mapping processing unit 4501 can be used to map the normalized second feature through a preset activation function to obtain a third feature.
[0101] The third acquisition unit 4502 may be configured to obtain a target feature of the multimedia information according to the third feature and a representation vector of the multimedia information.
[0102] In some embodiments of the present disclosure, the third acquisition unit 4502 may be specifically configured to perform element-by-element addition of the third feature and the representation vector of the multimedia information to obtain the target feature of the multimedia information.
[0103] In some embodiments of the present disclosure, the third acquisition unit 4502 may include a feature extraction subunit and a target feature acquisition subunit.
[0104] The feature extraction subunit may be used to perform feature extraction on a representation vector of multimedia information to obtain a fourth feature.
[0105] The target feature acquisition subunit may be used to add the third feature and the fourth feature element by element to obtain the target feature of the multimedia information.
[0106] In some embodiments of the present disclosure, the preset activation function is a ReLU activation function.
[0107] It should be noted that Figure 4 The multimedia information processing device 400 shown can execute each step in the above method embodiment and realize each process and effect in the above method embodiment, which will not be described in detail here.
[0108] Figure 5 A schematic structural diagram of an electronic device provided by an embodiment of the present disclosure is shown.
[0109] In the disclosed embodiment, Figure 5 The electronic device shown may be a server or a terminal or a server cluster, wherein the terminal may specifically include a mobile phone, a computer or a tablet computer, a vehicle-mounted terminal, or any device that can be used for the multimedia information processing method, etc., and is not limited here.
[0110] like Figure 5 As shown, the electronic device may include a processor 510 and a memory 520 storing computer program instructions.
[0111] Specifically, the processor 510 may include a central processing unit (CPU), or an application specific integrated circuit (ASIC), or may be configured to implement one or more integrated circuits of the embodiments of the present application.
[0112] The memory 520 may include a large capacity memory for information or instructions. By way of example and not limitation, the memory 520 may include a hard disk drive (HDD), a floppy disk drive, a flash memory, an optical disk, a magneto-optical disk, a tape, or a universal serial bus (USB) drive, or a combination of two or more of these. Where appropriate, the memory 520 may include a removable or non-removable (or fixed) medium. Where appropriate, the memory 520 may be inside or outside the integrated gateway device. In a particular embodiment, the memory 520 is a non-volatile solid-state memory. In a particular embodiment, the memory 520 includes a read-only memory (ROM). Where appropriate, the ROM may be a mask-programmed ROM, a programmable ROM (PROM), an erasable PROM (EPROM), an electrically erasable PROM (Electrically Erasable Programmable ROM, EEPROM), an electrically rewritable ROM (EAROM), or a flash memory, or a combination of two or more of these.
[0113] The processor 510 reads and executes the computer program instructions stored in the memory 520 to perform the steps of the multimedia information processing method provided in the embodiment of the present disclosure.
[0114] In one example, the electronic device may further include a transceiver 530 and a bus 540. Figure 5 As shown, the processor 510, the memory 520 and the transceiver 530 are connected via a bus 540 and communicate with each other.
[0115] The bus 540 includes hardware, software, or both. For example, but not limitation, the bus may include an Accelerated Graphics Port (AGP) or other graphics bus, an Extended Industry Standard Architecture (EISA) bus, a Front Side Bus (FSB), a Hyper Transport (HT) interconnect, an Industrial Standard Architecture (ISA) bus, an InfiniBand interconnect, a Low Pin Count (LPC) bus, a Memory bus, a MicroChannel Architecture (MCA) bus, a Peripheral Component Interconnect (PCI) bus, a PCI-Express (PCI-X) bus, a Serial Advanced Technology Attachment (SATA) bus, a Video Electronics Standards Association Local Bus (VLB) bus, or other suitable buses or a combination of two or more of these. Where appropriate, the bus 540 may include one or more buses.
[0116] The embodiments of the present disclosure further provide a computer-readable storage medium, which may store a computer program. When the computer program is executed by a processor, the processor implements the multimedia information processing method provided by the embodiments of the present disclosure.
[0117] The above-mentioned storage medium may, for example, include a memory 520 of computer program instructions, and the above-mentioned instructions may be executed by the processor 510 of the electronic device to complete the multimedia information processing method provided by the embodiment of the present disclosure. Optionally, the storage medium may be a non-temporary computer-readable storage medium, for example, the non-temporary computer-readable storage medium may be a ROM, a random access memory (Random Access Memory, RAM), a compact disc read-only memory (Compact Disc ROM, CD-ROM), a magnetic tape, a floppy disk, and an optical data storage device, etc.
[0118] It should be noted that, in this article, relational terms such as "first" and "second" are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the term "comprising" is intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device.
[0119] The above description is only a specific embodiment of the present disclosure, so that those skilled in the art can understand or implement the present disclosure. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present disclosure. Therefore, the present disclosure will not be limited to the embodiments described herein, but will conform to the widest scope consistent with the principles and novel features disclosed herein.
Claims
1. A multimedia information processing method, characterized in that: The method comprises: Acquire a first feature of the multimedia information to be processed; Inputting the first feature to the first convolution layer to obtain a first convolution result, and inputting the first feature to the second convolution layer to obtain a second convolution result, wherein the first convolution layer and the second convolution layer are respectively used to perform convolution processing on the first feature; Adding the first convolution result and the second convolution result element by element to obtain a second feature; Inputting the second feature into the first batch of normalization layers for normalization processing to obtain the normalized second feature, wherein the first batch of normalization layers is used to normalize the second feature; Preset processing is performed on the normalized second feature and the representation vector of the multimedia information to obtain the target feature of the multimedia information.
2. The method according to claim 1, characterized in that The neural network model also includes a third convolutional layer, a second batch of normalized layers, and a preset activation function, wherein the third convolutional layer is a layer above the second batch of normalized layers; The obtaining of the first feature of the multimedia information to be processed includes: Obtaining input information of the third convolutional layer, where the input information is obtained according to a representation vector of the multimedia information; Normalizing the output information of the third convolutional layer according to the second batch normalization layer to obtain the output information of the second batch normalization layer; The output information of the second batch of normalization layers is processed by the preset activation function to obtain the first feature of the multimedia information to be processed.
3. The method according to claim 1, characterized in that The step of adding the first convolution result and the second convolution result element by element to obtain a second feature includes: The first convolution result, the second convolution result, and the first feature are added element by element to obtain the second feature.
4. The method according to claim 1, characterized in that The performing preset processing on the normalized second feature and the representation vector of the multimedia information to obtain the target feature of the multimedia information includes: Mapping the normalized second feature through a preset activation function to obtain a third feature; A target feature of the multimedia information is obtained according to the third feature and the representation vector of the multimedia information.
5. The method according to claim 4, characterized in that The obtaining, according to the third feature and the representation vector of the multimedia information, a target feature of the multimedia information includes: The third feature and the representation vector of the multimedia information are added element by element to obtain a target feature of the multimedia information.
6. The method according to claim 4, characterized in that The obtaining, according to the third feature and the representation vector of the multimedia information, a target feature of the multimedia information includes: Performing feature extraction on the representation vector of the multimedia information to obtain a fourth feature; The third feature and the fourth feature are added element by element to obtain a target feature of the multimedia information.
7. The method according to claim 2 or 4, characterized in that: The preset activation function is the ReLU activation function.
8. A multimedia information processing device, characterized in that: include: A first acquisition module, used to acquire a first feature of the multimedia information to be processed; A convolution processing module, used for inputting the first feature into the first convolution layer to obtain a first convolution result, and inputting the first feature into the second convolution layer to obtain a second convolution result, wherein the first convolution layer and the second convolution layer are respectively used for performing convolution processing on the first feature; A second acquisition module, configured to add the first convolution result and the second convolution result element by element to obtain a second feature; A normalization processing module, used for inputting the second feature into the first batch of normalization layers for normalization processing to obtain the second feature after normalization processing, and the first batch of normalization layers is used for normalizing the second feature; The third acquisition module is used to perform preset processing on the normalized second feature and the representation vector of the multimedia information to obtain the target feature of the multimedia information.
9. An electronic device, characterized in that: include: Memory; processor; as well as Computer programs; The computer program is stored in the memory and is configured to be executed by the processor to implement the method according to any one of claims 1 to 7.
10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.
