A method for training a neural network and related apparatuses

Abstract

The application discloses a training method of a neural network based on artificial intelligence technology and related devices. Pre-training data is obtained, the pre-training data including multiple pictures and label information of the multiple pictures, a pre-training model is subjected to self-supervised training according to the pre-training data, and a picture neural network is obtained. Since the label information of the picture is attribute information naturally possessed by the picture source, the label information is easy to obtain. Therefore, a large amount of pre-training data can be easily obtained. The pre-training model is trained based on the pre-training data to obtain the picture neural network, and the training efficiency and training precision of the picture neural network can be effectively improved. The pre-training model is subjected to self-supervised training, and the training efficiency and training precision can be effectively improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of artificial intelligence, and in particular to a neural network training method and related apparatus. BACKGROUND

[0002] With the development of the Internet, the amount of various image-text information is growing exponentially, and the sources and quality levels of these image-text information are uneven. Therefore, before the image-text information is distributed to users, manual auditing and marking of specific content is performed, and images with quality problems are filtered.

[0003] The current method is to identify the content by using a machine learning auxiliary algorithm. The iterative process of a neural network model is generally as follows: demand determination, data collection, model training, model testing and online. Since the types of image-text information are very diverse, in order to ensure the training accuracy of the model, training data with a wide variety of types need to be used during model training.

[0004] The label information of these training data currently needs to be manually marked. A large amount of training data results in high training cost of the model, and thus the training efficiency of the model is low. SUMMARY

[0005] Embodiments of the present application provide a neural network training method and related apparatus. The pre-training data includes multiple images and label information of the multiple images. Since the label information of the images is the attribute information naturally possessed by the image source, it is easy to obtain. Therefore, a large amount of pre-training data can be easily obtained. Based on the pre-training data, a pre-training model is trained to obtain an image neural network, which can effectively improve the training efficiency and training accuracy of the neural network model. The pre-training model uses self-supervised training, which can effectively improve the training efficiency and training accuracy.

[0006] Therefore, the present application provides a neural network training method, which comprises:

[0007] obtaining pre-training data, the pre-training data including multiple images and label information corresponding to the multiple images, and the multiple images including one or more of the following:

[0008] a frame-extracted image in a video file, wherein the label information corresponding to the frame-extracted image is the label information of the video file,

[0009] or an image of image-text information, wherein the label information corresponding to the image of image-text information is title or abstract information of the image-text information,

[0010] or a public image searched based on target label information, wherein the target label information is taken as the label information corresponding to the public image;

[0011] The pre-trained model is self-supervised trained based on the pre-trained data to obtain an image neural network, where the image neural network is used to obtain the embedding features of the image.

[0012] Another aspect of the present application provides a neural network training device, comprising:

[0013] The transceiver module is used to obtain pre-training data. The pre-training data includes multiple images and label information corresponding to the multiple images. The multiple images include one or more of the following:

[0014] The frame extraction image in the video file, wherein the label information corresponding to the frame extraction image is the label information of the video file,

[0015] Or, a picture of graphic information, wherein the tag information corresponding to the picture of graphic information is the title or summary information of the graphic information,

[0016] Alternatively, a public image is searched based on target tag information, wherein the target tag information is used as tag information corresponding to the public image;

[0017] The processing module is used to perform self-supervised training on the pre-trained model based on the pre-trained data to obtain an image neural network, wherein the image neural network is used to obtain the embedding features of the image.

[0018] In another implementation of another aspect of the embodiment of the present application, the neural network training apparatus further includes:

[0019] The processing module is also used to perform self-supervisory training on the pre-training model using a comparative learning method based on the pre-training data to obtain an image neural network, wherein the identification information corresponding to multiple images in the pre-training data is used as a supervisory signal in the training.

[0020] In another implementation of another aspect of the embodiment of the present application, the frame extraction graph in the video file includes: frame extraction graphs corresponding to a plurality of adjacent video frames in the same video file, and / or frame extraction graphs corresponding to video frames in different video files;

[0021] The images of the graphic information include: a cover image of the graphic information, a body image of the graphic information, and / or a summary image of the graphic information.

[0022] In another implementation of another aspect of the embodiment of the present application, the neural network training apparatus further includes:

[0023] The transceiver module is also used to obtain multiple video files;

[0024] The processing module is further used to perform deduplication processing on the multiple video files to obtain multiple deduplicated video files;

[0025] The processing module is also used to extract frame images from the multiple video files after deduplication.

[0026] In another implementation of another aspect of the embodiment of the present application, the neural network training apparatus further includes:

[0027] The processing module is also used to perform knowledge distillation on the pre-trained model to obtain an image neural network.

[0028] In another implementation of another aspect of the embodiment of the present application, the neural network training apparatus further includes:

[0029] The transceiver module is also used to obtain business samples, which include one or more pictures and identification information corresponding to the pictures;

[0030] The processing module is also used to fine-tune the pre-trained model using business samples to obtain a fine-tuned pre-trained model.

[0031] In another implementation of another aspect of the embodiment of the present application, the neural network training apparatus further includes:

[0032] The transceiver module is further configured to obtain a first tag information set, where the first tag information set includes one or more of the following: tag information of one or more video files, or tag information of one or more graphic and text information;

[0033] The processing module is further configured to perform deduplication processing on the first tag information set to obtain target tag information;

[0034] The processing module is further used to obtain multiple public images to be screened through a search engine based on target tag information;

[0035] The processing module is further configured to eliminate pictures with a resolution lower than a first threshold from the plurality of public pictures to be screened, and obtain public pictures.

[0036] Another aspect of the present application provides a computer device, comprising: a memory, a processor, and a bus system;

[0037] Wherein, the memory is used to store programs;

[0038] The processor is used to execute the program in the memory, and the processor is used to perform the above-mentioned methods according to the instructions in the program code;

[0039] The bus system is used to connect the memory and the processor so that the memory and the processor can communicate with each other.

[0040] Another aspect of the present application provides a computer-readable storage medium, wherein instructions are stored in the computer-readable storage medium. When the computer-readable storage medium is run on a computer, the computer is enabled to execute the above-mentioned methods.

[0041] Another aspect of the present application provides a computer program product or computer program, which includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the methods provided in the above aspects.

[0042] It can be seen from the above technical solutions that the embodiments of the present application have the following advantages:

[0043] First, pre-training data is obtained, where the pre-training data includes multiple images and label information corresponding to the multiple images, where the multiple images include one or more of the following: frame-drawing images in a video file, where the label information corresponding to the frame-drawing images is the label information of the video file, or images of graphic information, where the label information corresponding to the images of graphic information is the title or summary information of the graphic information, or public images obtained by searching based on target label information, where the target label information is used as the label information corresponding to the public images; then, self-supervised training is performed on the pre-training model based on the pre-training data to obtain an image neural network, where the backbone network of the pre-training model includes a Swin Transformer network, and the image neural network is used to obtain embedding features of the image.

[0044] In this application, the pre-training data includes multiple pictures and label information of multiple pictures. Since the label information of the picture is the attribute information naturally possessed by the source of the picture, it is easy to obtain. Therefore, a large amount of pre-training data can be easily obtained. Based on the pre-training data, the pre-training model is trained to obtain a picture neural network, which can effectively improve the training efficiency and training accuracy of the neural network model. The pre-training model adopts self-supervised training, which can effectively improve the training efficiency and training accuracy. The backbone network of the pre-training model includes a Swin Transformer network, which can ensure high training accuracy while saving training time. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] Figure 1 Schematic diagram of the application architecture of the neural network training method in the embodiment of the present application;

[0046] Figure 2 This is a schematic diagram of the structure of the neural network training system proposed in this application;

[0047] Figure 3A schematic diagram of an embodiment of a neural network training method proposed in an embodiment of the present application;

[0048] Figure 4 This is a schematic diagram of comparative learning proposed in an embodiment of the present application;

[0049] Figure 5 This is a schematic diagram of the process of training a pre-trained model in an embodiment of the present application;

[0050] Figure 6 A schematic diagram of a neural network training device proposed in an embodiment of the present application;

[0051] Figure 7 This is a schematic diagram of a server structure provided by an embodiment of the present application;

[0052] Figure 8 This is a schematic diagram of the terminal device structure provided in an embodiment of the present application. DETAILED DESCRIPTION

[0053] The embodiment of the present application provides a method for training a neural network, wherein the pre-training data includes multiple images and label information of the multiple images. Since the label information of the images is the attribute information naturally possessed by the image source, it is easy to obtain. Therefore, a large amount of pre-training data can be easily obtained. Based on the pre-training data, a pre-training model is trained to obtain an image neural network, which can effectively improve the training efficiency and training accuracy of the neural network model. The pre-training model adopts self-supervised training, which can effectively improve training efficiency and training accuracy.

[0054] The terms "first", "second", "third", "fourth", etc. (if any) in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequential order. It should be understood that the numbers used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can, for example, be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "corresponding to" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0055] Exemplary embodiments will be described in detail herein, with examples illustrated in the accompanying drawings. In the following description, when referring to the drawings, identical numbers in different figures represent identical or similar elements, unless otherwise indicated. The embodiments described in the following exemplary embodiments are not intended to represent all possible embodiments consistent with the present application. Rather, they are merely examples of methods consistent with certain aspects of the present application, as detailed in the appended claims.

[0056] Artificial Intelligence (AI) refers to the theories, methods, techniques, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, to perceive the environment, acquire knowledge, and use that knowledge to achieve optimal results. In other words, AI is a comprehensive technology within computer science that seeks to understand the essence of intelligence and produce new intelligent machines that can respond in a manner similar to human intelligence. AI also studies the design principles and implementation methods of various intelligent machines, enabling them to possess the capabilities of perception, reasoning, and decision-making.

[0057] Artificial intelligence (AI) technology is a comprehensive discipline encompassing a wide range of fields, encompassing both hardware and software technologies. Foundational AI technologies generally include sensors, specialized AI chips, cloud computing, distributed storage, big data processing, operating / interaction systems, and mechatronics. AI software technologies primarily encompass computer vision, speech processing, natural language processing, and machine learning / deep learning.

[0058] Natural language processing (NLP) is a key area of ​​research in computer science and artificial intelligence. It studies the theories and methods that enable effective communication between humans and computers using natural language. Natural language processing (NLP) integrates linguistics, computer science, and mathematics. Therefore, research in this field involves natural language—the language we use in everyday life—and is closely linked to the study of linguistics. Natural language processing technologies typically include text processing, semantic understanding, machine translation, robotic question answering, and knowledge graphs.

[0059] Machine learning (ML) is a multidisciplinary field that encompasses probability theory, statistics, approximation theory, convex analysis, and algorithmic complexity theory. It specifically studies how computers can simulate or implement human learning behaviors to acquire new knowledge or skills and reorganize existing knowledge structures to continuously improve their performance. Machine learning is at the core of artificial intelligence and the fundamental way to make computers intelligent. Its applications span all areas of AI. Machine learning and deep learning typically include techniques such as artificial neural networks, belief networks, reinforcement learning, transfer learning, inductive learning, and learning through demonstration.

[0060] With the research and advancement of artificial intelligence technology, artificial intelligence technology has been studied and applied in many fields, such as common smart homes, smart wearable devices, virtual assistants, smart speakers, smart marketing, unmanned driving, autonomous driving, drones, robots, smart medical care, smart customer service, etc. It is believed that with the development of technology, artificial intelligence technology will be applied in more fields and play an increasingly important role.

[0061] See Figure 1 As shown, Figure 1 Schematic diagram of the application architecture of the neural network training method in an embodiment of the present application, including a server 100 and a terminal device 200.

[0062] Terminal device 200 can be at least one of a smartphone, a computer, a game console, a desktop computer, a tablet computer, an intelligent voice interaction device, a smart home appliance, an in-vehicle terminal, an aircraft, an e-book reader, an MP3 (Moving Picture Experts Group Audio Layer III) player, an MP4 (Moving Picture Experts Group Audio Layer IV) player, or a laptop computer. Terminal device 200 installs and runs an application program that implements a neural network training method. For example, the application program can be a language processing application such as a compiler or assembler.

[0063] The terminal device 200 and the server 100 can be connected via a wireless network or a wired network to achieve mutual communication. Optionally, the above-mentioned wireless network or wired network uses standard communication technology and / or protocols. The wireless network or wired network is typically the Internet, but can also be any network, including but not limited to any combination of a local area network (LAN), a metropolitan area network (MAN), a wide area network (WAN), a mobile, wired or wireless network, a private network or a virtual private network. In some embodiments, technologies and / or formats including Hyper Text Mark-up Language (HTML), Extensible Markup Language (XML), etc. are used to represent data exchanged over the network. In addition, conventional encryption technologies such as Secure Socket Layer (SSL), Transport Layer Security (TLS), Virtual Private Network (VPN), Internet Protocol Security (IPsec), etc. can also be used to encrypt all or some links. In other embodiments, customized and / or dedicated data communication technologies may be used to replace or supplement the above-mentioned data communication technologies.

[0064] The server 100 can provide various network services for the terminal device 200, wherein the server 100 can be a single server, a server cluster consisting of several servers, or a cloud computing center.

[0065] Specifically, the server 100 may include a processor 110 (Center Processing Unit, CPU), a memory 120, an input device 130 and an output device 140, etc. The input device 130 may include a keyboard, a mouse, a touch screen, etc., and the output device 140 may include a display device, such as a liquid crystal display (LCD), a cathode ray tube (CRT), etc.

[0066] The memory 120 may include a read-only memory (ROM) and a random access memory (RAM), and provides the program instructions and data stored in the memory 120 to the processor 110. In the embodiment of the present application, the memory 120 may be used to store the program of the neural network training method in the embodiment of the present application.

[0067] The processor 110 calls the program instructions stored in the memory 120, and the processor 110 is used to execute the steps of any neural network training method in the embodiments of the present application according to the obtained program instructions.

[0068] It should be noted that, in the embodiment of the present application, the training method of the neural network is mainly executed by the server 100 side. For example, the terminal device 200 can send the collected multiple pictures to the server 100, and the server 100 processes the multiple pictures to obtain pre-training data. Then the server 100 trains the pre-training model according to the pre-training data to obtain the picture neural network. The server 100 can use the picture neural network to distribute and filter the pictures in the information flow, and can return the distribution and filtering results to the terminal device 200. Figure 1 The application architecture shown is explained using the server 100 as an example. Of course, the neural network training method in the embodiment of the present application can also be executed by the terminal device 200. For example, the terminal device 200 can obtain a pre-trained model from the server 100, and then train the pre-trained model based on the pre-training data to obtain an image neural network. The terminal device 200 can use the image neural network to distribute and filter images in the information flow, and then display the distribution and filtering results to the user of the terminal device 200. This is not limited in the embodiment of the present application.

[0069] In addition, the application architecture diagram in the embodiment of the present application is intended to more clearly illustrate the technical solutions in the embodiment of the present application, and does not constitute a limitation on the technical solutions provided in the embodiment of the present application. Of course, it is not limited to neural network training applications. For other application architectures and business applications, the technical solutions provided in the embodiment of the present application are also applicable to similar problems.

[0070] (1) Neural Network

[0071] A neural network can be composed of neural units. A neural unit can refer to an operation unit with xs and intercept 1 as input. The output of the operation unit can be:

[0072] Where s = 1, 2, ... n, n is a natural number greater than 1, Ws is the weight of xs, and b is the bias of the neural unit. f is the activation function of the neural unit, which is used to introduce nonlinear characteristics into the neural network to convert the input signal of the neural unit into the output signal. The output signal of the activation function can be used as the input of the next convolutional layer, and the activation function can be a sigmoid function. A neural network is a network formed by connecting multiple single neural units mentioned above, that is, the output of one neural unit can be the input of another neural unit. The input of each neural unit can be connected to the local receptive field of the previous layer to extract the features of the local receptive field. The local receptive field can be an area composed of several neural units.

[0073] (2) Transformer layer

[0074] The neural network may include an embedding layer and at least one transformer layer, wherein the at least one transformer layer may be N transformer layers (N is an integer greater than 0), wherein each transformer layer includes an attention layer, an add&norm layer, a feedforward layer, and an add&norm layer that are adjacent in sequence. In the embedding layer, the current input is embedded to obtain multiple feature vectors; in the attention layer, P input vectors are obtained from the previous layer of the first transformer layer, and with any first input vector among the P input vectors as the center, based on the correlation between each input vector within a preset attention window and the first input vector, an intermediate vector corresponding to the first input vector is obtained, thereby determining the P intermediate vectors corresponding to the P input vectors; in the pooling layer, the P intermediate vectors are merged into Q output vectors, wherein the multiple output vectors obtained by the last transformer layer in the transformer layer are used as feature representations of the current input.

[0075] Next, the above steps are introduced in detail with reference to specific examples.

[0076] First, in the embedding layer, the current input is embedded to obtain multiple feature vectors.

[0077] The embedding layer can be called the input embedding layer. The current input can be a text input, for example, a paragraph of text or a sentence. The text can be Chinese text, English text, or text in other languages. After obtaining the current input, the embedding layer can embed each word in the current input to obtain the feature vector of each word. In some embodiments, such as Figure 1As shown, the embedding layer includes an input embedding layer and a positional encoding layer. In the input embedding layer, word embedding processing can be performed on each word in the current input to obtain a word embedding vector for each word. In the positional encoding layer, the position of each word in the current input can be obtained, and a position vector can be generated for each word's position. In some examples, the position of each word can be its absolute position in the current input. Taking the current input "What number should I return Huabei?" as an example, the position of "what" can be represented as the first position, the position of "number" can be represented as the second position, and so on. In some examples, the position of each word can be the relative position of each word. Still taking the current input "What number should I return Huabei?" as an example, the position of "what" can be represented as before "number", the position of "number" can be represented as after "what" and before "should", and so on. Once the word embedding vector and position vector for each word in the current input are obtained, the position vector and the corresponding word embedding vector can be combined to obtain a feature vector for each word, that is, multiple feature vectors corresponding to the current input. The multiple feature vectors can be represented as an embedding matrix with a preset dimension. The number of eigenvectors in the multiple eigenvectors may be set to M, and the preset dimension may be H, so that the multiple eigenvectors may be represented as an M×H embedding matrix.

[0078] Secondly, P input vectors can be obtained from the previous layer of the transformer layer. With any input vector among the P input vectors as the center, the intermediate vector corresponding to the input vector is obtained based on the correlation between each input vector within the preset attention window range and the input vector. In this way, the P intermediate vectors corresponding to the P input vectors are determined. The attention layer can also be called a multi-head attention layer. In one example, the attention layer can be a fixed window multi-head attention layer.

[0079] In the embodiment of the present application, the architecture of the transformer layer is redesigned based on neural network search.

[0080] (3) Attention mechanism

[0081] The attention mechanism mimics the internal process of biological observation behavior, namely, a mechanism that aligns internal experience and external sensations to increase the observation precision of certain areas. It can quickly filter out high-value information from a large amount of information using limited attention resources. The attention mechanism can quickly extract important features from sparse data and is therefore widely used in natural language processing tasks, especially machine translation. The self-attention mechanism is an improvement on the attention mechanism, which reduces dependence on external information and is better at capturing the internal correlation of data or features. The essential idea of ​​the attention mechanism can be rewritten as the following formula:

[0082] Here, Lx = ||Source|| represents the length of the Source. This formula implies that the elements in the Source are imagined to consist of a series of data pairs. Given a Query element in the target, the similarity or correlation between the Query and each Key is calculated to obtain the weight coefficient for each Key's corresponding Value. The weighted sum of the Values ​​is then taken to obtain the final Attention value. Essentially, the Attention mechanism performs a weighted sum of the Values ​​of the Source elements, with the Query and Key used to calculate the weight coefficient for the corresponding Value. Conceptually, Attention can be understood as selectively filtering out a small amount of important information from a large amount of information and focusing on this important information, while ignoring the majority of less important information. This focusing process is reflected in the calculation of the weight coefficients: the larger the weight, the more focus is placed on the corresponding Value. In other words, the weight represents the importance of the information, while the Value represents the corresponding information. The self-attention mechanism can be understood as internal attention. The attention mechanism occurs between the Query element of the Target and all elements of the Source. The self-attention mechanism refers to the attention mechanism that occurs between the internal elements of the Source or the internal elements of the Target. It can also be understood as the attention calculation mechanism in the special case of Target = Source. The specific calculation process is the same, only the calculation object has changed.

[0083] (4) Neural Network Compression

[0084] Compression may include at least one of the following methods: model pruning, weight sharing, kernel sparsity (regularization), quantization, binarization, low-rank decomposition, etc.

[0085] Among them, quantization, also known as neural network quantization, is a model compression technology that converts floating-point storage (operation) into integer storage (operation). For example, the model parameters of a model were originally represented by float32 (32-bit floating point), and after quantization, the model parameters of the model are represented by int8 (8-bit fixed point). Through the quantization operation of the model, the operation speed of the model is improved at the cost of a small loss of accuracy.

[0086] The essence of model quantization is the conversion / mapping between two data types. In one implementation of converting floating-point data (data of floating-point data type) to fixed-point data (data of fixed-point data type), the following formula can be used:

[0087] Among them, R is the input floating-point data, Q is the fixed-point data after the floating-point data R is quantized, Z represents the zero point value (ZeroPoint), and S represents the ratio. It can be seen that after determining S and Z, the conversion between the two data can be performed. There are many ways to determine S and Z, for example:

[0088] Z=Qmax-Rmax / S;

[0089] Among them, Rmax represents the maximum value of the input floating-point data, Rmin represents the minimum value of the input floating-point data, Qmax represents the maximum value of the fixed-point data, and Rmin represents the minimum value of the fixed-point data.

[0090] Among them, the conversion between fixed-point data with different bit numbers (bits, 1 bit = 1 position) can refer to the conversion method between floating-point data and fixed-point data mentioned above, or it can be other conversion methods in the prior art, which will not be repeated here.

[0091] In one implementation, 4-bit and 8-bit conversions can be performed using the above-mentioned conversion method, while floating-point data and 2-bit (1-bit) conversions can be performed using the following formula:

[0092] The 2-bit data can be represented by three numbers: -1, 0, and 1. T is the threshold. When the floating-point data is greater than or equal to T, the converted 2-bit fixed-point data is 1. When the floating-point data is less than -T, its value is converted to -1. For other floating-point values, its value is converted to 0. The conversion method for 1-bit data is similar to that for 2-bit data, but its fixed-point values ​​are only -1 and 1, and the value of T is 0.

[0093] Among them, pruning can be understood as removing part of the network structure in the network.

[0094] After the model is compressed, the storage resources required to store the compressed model will be reduced. In order to ensure the accuracy of the compressed model, knowledge distillation can be performed on the compressed model. Specifically, the model before compression can be used as the teacher model, and the model after compression can be used as the student model. Knowledge distillation (or simply distillation) can be performed on the student model.

[0095] Among them, the teacher model can also be called a teacher model, a guidance model, etc., which is not limited here.

[0096] When performing knowledge distillation, the pre-compression model (a pre-trained complex network) can be used to train the post-compression model (another simple network), allowing the simple network to have the same or similar data processing capabilities as the complex network. Knowledge distillation involves transferring the "knowledge" of a trained complex network to a simpler network. This simpler network can have fewer parameters than the complex network.

[0097] Self-distillation refers to distilling a model into itself. The Teacher Model is an integrated version of the Student Model, known as a self-ensemble. Distillation improves model generalization, reduces model size, and improves post-deployment performance, which is crucial for project implementation.

[0098] It should be noted that the above-mentioned same or similar data processing capabilities can be understood as that when processing the same data to be processed, the processing results obtained by the student model and the teacher model after knowledge distillation are the same or similar.

[0099] (5) Loss function

[0100] During the training of a deep neural network, because we want the output of the deep neural network to be as close as possible to the desired predicted value, we can compare the current network's predicted value with the desired target value and then update the weight vector of each layer of the neural network based on the difference between the two. (Of course, before the first update, there is usually an initialization process, which pre-configures the parameters for each layer in the deep neural network.) For example, if the network's predicted value is too high, the weight vector is adjusted to make it predict a lower value. This adjustment is continued until the deep neural network can predict the desired target value or a value very close to the desired target value. Therefore, it is necessary to predefine "how to compare the difference between the predicted value and the target value." This is the loss function (or objective function), which is an important equation used to measure the difference between the predicted value and the target value. For example, the loss function output value (loss) indicates a greater difference, so training a deep neural network becomes a process of minimizing this loss.

[0101] When performing knowledge distillation, it is necessary to construct the loss based on the output of the teacher model and the output of the student model. The model output used to construct the loss can be the output of the output layer of the model, the intermediate feature map output of the intermediate network layer, or the result of processing the output of the output layer and / or the intermediate feature map output of the intermediate network layer.

[0102] Next, a neural network training method proposed in an embodiment of the present application is introduced in conjunction with the accompanying drawings. The system to which the neural network training method is applied is called a neural network training system. First, the neural network training system is introduced in conjunction with the accompanying drawings. Figure 2 , Figure 2 This is a schematic diagram of the structure of the neural network training system proposed in this application. The neural network training system proposed in this application includes: a pre-training database, a content production end, a content database, a content consumption end, an image service unit, a neural network training unit, and the Internet.

[0103] A pre-training database can obtain relevant data for training neural networks from content production, content database, and content consumption.

[0104] Specifically, the content production end may include: Professionally-generated Content (PGC) or User-generated Content (UGC), Multi-Channel Network (MCN) or Professional Generated Content + User Generated Content (PUGC) content producers. The above-mentioned content production end can provide graphic information and / or video files through terminal devices or back-end application programming interfaces (APIs). The graphic information includes but is not limited to self-media articles, self-media atlases, or files including pictures and texts, web pages including pictures and texts, etc. The graphic information can also be news subscriptions, web feeds, push notifications, feeds, blogs, microblogs or summaries, etc.

[0105] The text and image information and / or video files provided (or produced) by the content production end can be transmitted to the content database through the uplink and downlink content interface services. Specifically, the content production end first obtains the interface address of the content database, and then uploads the text and image information and / or video files to the content database.

[0106] Optionally, the content producer can also upload behavioral data of the content consumer to the content database. This behavioral data includes but is not limited to: reading speed, playback completion rate, reading time, pause time, loading time, or number of clicks.

[0107] The content database is used to store meta-information published by various content producers. This meta-information includes, but is not limited to: video files, graphic files, cover image links, video file bitrate, video file format, graphic information file format, title, release time, author, video file size, graphic information size, whether it is an original mark, whether it is a first release mark, or content classification information during the review process. Exemplarily, the classification information of content during the review process includes: multi-level classification information and tag information. Taking the graphic information of mobile phone B produced by manufacturer A as an example, the first-level classification information is: technology, the second-level classification information is: smart phone, the third-level classification information is: domestic mobile phone, and the tag information is: manufacturer A, mobile phone B.

[0108] The content consumption end can be a user who watches / plays / downloads / reads graphic information (and / or video). Exemplarily, the content consumption end can obtain graphic information through a content delivery network (CDN) deployed at the edge. The content consumption end can obtain videos through a video server. Typically, the content consumption end browses graphic information and / or videos through information streams (feeds). Therefore, the content consumption end can upload graphic information and / or videos to a pre-training database, and the graphic information and / or videos are used as training samples for subsequent training of the image neural network.

[0109] In addition to storing the training samples in the database, the neural network training system can also obtain public images from the Internet, and the public images are used as training samples.

[0110] The neural network training system also includes: an image service unit and a neural network training unit.

[0111] The neural network training unit is used to obtain pre-training data from various channels and then use the pre-training data to train the pre-training model to obtain an image neural network. For a detailed description, please refer to the subsequent embodiments.

[0112] The image service unit processes various image services based on the image neural network trained by the neural network training unit.

[0113] Specifically, the image business unit uses an image neural network to extract image features from the image, and the image features include but are not limited to: clarity score, aesthetic score, inappropriate image content subcategory, mosaic mark, watermark mark, or integrity mark, etc. An atomic model is constructed based on the image features and the image neural network, and the atomic model includes clarity, aesthetics, mosaic image, inappropriate image, image fingerprint, image integrity, etc. The image business unit provides image processing services based on the above-mentioned atomic model or the image features extracted by the atomic model. The image processing service includes but is not limited to: image review, image selection, image intelligent cropping, or image deduplication, etc. Exemplarily, image review includes: reviewing images in image and text information, or reviewing cover images or frame-extracted images of video files. The image business unit distributes image and text information or videos to the content consumption end based on the image review results.

[0114] It is understandable that, in addition to Figure 2In addition to the units or components shown in the figure, the neural network training system can also include other units or components, such as: a dispatch center service component, which is responsible for the dispatch flow process of video and graphic information. Specifically: obtain the stored video files and graphic information from the database, and obtain the metadata of the video files and the metadata of the graphic information from the content database. Then dispatch the image processing service to perform relevant processing on the video files or graphic information, such as: image quality review, image selection, screenshot or deduplication, etc. The dispatch center service component can also transmit the video files and graphic information to the manual review service component for manual review and processing.

[0115] The manual review service component, typically implemented as a system, first uses an image neural network to perform preliminary filtering of text and video files. This is then manually verified and reviewed, with the results recorded as metadata for the text or video files in the content database. Furthermore, the review results can be used to evaluate the effectiveness of the image neural network training.

[0116] The file download component can download text, image, and video files from the internet, content databases, content production, or content consumption. It then extracts images from the text, or extracts video frames from video files as frame images. These extracted images are stored as training samples in the pre-training database.

[0117] See also Figure 3 , Figure 3 This is a schematic diagram of an embodiment of a neural network training method proposed in an embodiment of the present application. A neural network training method proposed in an embodiment of the present application includes:

[0118] 301. Obtain pre-training data.

[0119] In this embodiment, the neural network training system first obtains pre-training data, which includes multiple images and label information corresponding to the multiple images. The multiple images include one or more of the following:

[0120] A frame-extracted image in a video file, wherein the label information corresponding to the frame-extracted image is the label information of the video file, or a picture of graphic information, wherein the label information corresponding to the picture of graphic information is the title or summary information of the graphic information, or a public picture obtained based on a search for target label information, wherein the target label information is used as the label information corresponding to the public picture.

[0121] Specifically, after the neural network training system obtains the video file, it extracts the frames of the video file to obtain an extracted frame graph. The label information corresponding to the extracted frame graph is the label information of the video file. Exemplarily, the label information of the video file includes but is not limited to: the title of the video file, the classification information of the video file, or the label information set for different moments on the timeline in the video file. For example: for video file A, the title of video file A is "Introduction video of mobile phone A", the classification information of video file A is "Technology", the moment 1 minute 30 seconds in video file A is "Main film", the moment 3 minutes 00 seconds is "Appearance", the moment 5 minutes 05 seconds is "Shooting", the moment 8 minutes 00 seconds is "Audio", and the moment 10 minutes 00 seconds is "Ending".

[0122] For pictures in graphic information, the label information corresponding to the picture in the graphic information is the title or summary information of the graphic information. The graphic information includes but is not limited to self-media articles, self-media atlases, or files including pictures and texts, web pages including pictures and texts, etc. For example, when the graphic information is a self-media article, the label information corresponding to the picture in the graphic information can be the title of the self-media article or the summary of the self-media article. For example: if the graphic information is "Introduction to Mobile Phone A", then the label information corresponding to the picture in the graphic information is "Mobile Phone".

[0123] Alternatively, public pictures are obtained by searching based on the target tag information, wherein the target tag information is used as the tag information corresponding to the public pictures. Specifically, the neural network training system determines the type of picture that needs to be obtained, and then determines the target tag information based on the type of the picture. Secondly, based on the target tag information, the neural network training system searches for public pictures among numerous pictures on the Internet through a search engine. Exemplarily, if the type of picture that the neural network training system needs to obtain is "mobile phone", the neural network training system searches for "mobile phone" as the target tag information in the search engine and obtains multiple pictures. The pictures obtained by the search are called public pictures, and the target tag information "mobile phone" is used as the tag information corresponding to these public pictures.

[0124] Furthermore, when the neural network training system uses multiple target label information to search for public images in a search engine, the multiple target label information can be merged and deduplicated first. The one or more target label information obtained after merging and deduplicating is called a target label information set. The neural network training system then performs image retrieval in a search engine based on the target label information set to obtain one or more public images. Exemplarily, the multiple target label information include: "mobile phone", "smartphone", "smartwatch" and "mobile phone". First, the neural network training system merges and deduplicates the multiple target label information, and the target label information obtained after merging and deduplicating includes: "smartphone", "smartwatch" and "mobile phone". Then, the neural network training system uses "smartphone", "smartwatch" and "mobile phone" as target label information sets to perform image retrieval in a search engine to obtain one or more public images.

[0125] Optionally, the pre-training data may also include images in a dataset, including but not limited to: "ImageNet" dataset, "ISLVRC" dataset, or "Places2" dataset.

[0126] 302. Perform self-supervised training on the pre-trained model based on the pre-trained data to obtain an image neural network.

[0127] In this embodiment, after the neural network training system obtains the pre-training data, the neural network training system uses the pre-training data to perform self-supervised training on the pre-training model to obtain an image neural network.

[0128] Specifically, self-supervised learning (SSL) mainly uses auxiliary tasks (pretext) to mine its own supervisory information from large-scale unsupervised data, and trains the network through this constructed supervisory information, so that it can learn representations that are valuable for downstream tasks. That is to say, the supervisory information of self-supervised learning is not manually labeled, but the algorithm automatically constructs supervisory information in large-scale unsupervised data to perform supervised learning or training. In this embodiment, since the label information of the pre-training data has the characteristic of being easy to obtain, for example: for the frame-by-frame image of the video file, the label information corresponding to the frame-by-frame image is the label information of the video file; for the picture in the graphic information, the label information corresponding to the picture of the graphic information is the title or summary information of the graphic information; for public pictures, the public pictures are obtained by the neural network training system based on the target label information search, so the label information of the public pictures is the target label information. Through the above method, the training cost can be effectively reduced and the training accuracy of the neural network model can be improved.

[0129] There are many possible implementations of the pre-trained model, such as the Swin Transformer network model, the ResNet50 network model, or the EfficientNet-B4 network model, etc., which are not limited in this embodiment of the present application. For ease of understanding, the embodiment of the present application takes the Swin Transformer network model as the backbone of the pre-trained model as an example for explanation.

[0130] In the Swin Transformer network model, the self-attention mechanism learns the relationships between different image embedding features, encoding their contextual information. This information is then passed through a feed-forward neural network (FNN) through nonlinear transformations to output vector representations of each image feature that incorporates contextual features. Each Transformer network layer primarily consists of two sublayers: a multi-head self-attention layer and a feed-forward network layer. The multi-head self-attention layer executes multiple self-attentions with different parameters in parallel and concatenates the results of each self-attention as input to the subsequent network.

[0131] In the embodiment of the present application, the pre-training data includes multiple pictures and label information of multiple pictures. Since the label information of the picture is the attribute information naturally possessed by the source of the picture, it is easy to obtain. Therefore, a large amount of pre-training data can be easily obtained. Based on the pre-training data, the pre-training model is trained to obtain a picture neural network, which can effectively improve the training efficiency and training accuracy of the neural network model and reduce labor costs. For some types of pictures that are difficult to collect, using a pre-training model for training can effectively reduce the demand for the number of training samples and speed up the model training.

[0132] In combination with the foregoing embodiments, a neural network training method proposed in an embodiment of the present application further includes: based on pre-training data, self-supervising training is performed on the pre-training model using a comparative learning method to obtain an image neural network, wherein the identification information corresponding to multiple images in the pre-training data is used as a supervisory signal in training.

[0133] Specifically, contrastive learning is achieved by constructing positive samples and negative samples, and then measuring the distance between positive and negative samples to achieve self-supervised learning. For easier understanding, please refer to Figure 4 , Figure 4This is a schematic diagram of contrastive learning proposed in an embodiment of the present application. For training samples (i.e., pre-training data), dimensionality reduction (i.e., feature extraction) is performed using an encoder or momentum encoder. Similarities between different training samples are then calculated and compared, and finally, contrastive loss is calculated.

[0134] The expression of contrastive loss is as follows:

[0135]

[0136] d=||a n -b n ||2d represents the Euclidean distance between the two sample features, y is the label of whether the two samples match, y=1 means the two samples are similar or matched, y=0 means they do not match, α n For samples a, b n is sample b, n is an integer greater than or equal to 1, N is an integer greater than or equal to 1, and margin is the set threshold.

[0137] Furthermore, since the pre-training data includes label information for multiple images, the label information for multiple images can be used as supervisory signals in self-supervised training. One possible implementation method is to use the classification information of multiple images as the supervisory signal in self-supervised training. For example, multiple images include: Image 1 "Mobile Phone A", Image 2 "Mobile Phone B", Image 3 "Watch A", and Image 4 "Watch B". The classification information includes: the classification information of Category 1 is "Mobile Phone", Category 1 includes Images 1 and 2; the classification information of Category 2 is "Watch", Category 2 includes Images 3 and 4. The classification information of Category 1 and Category 2 are used as supervisory signals in self-supervised training. By fusing modalities to predict classification results, different modalities can be better combined to learn better results.

[0138] In combination with the foregoing embodiments, a neural network training method proposed in an embodiment of the present application further includes: the frame extraction image in the video file includes one or more of the following: the frame extraction image corresponding to multiple adjacent video frames in the same video file, or the frame extraction image corresponding to the video frames in different video files; the picture of the graphic information includes one or more of the following: the cover picture of the graphic information, the text picture of the graphic information, or the summary picture of the graphic information.

[0139] The frame map can be a frame map corresponding to multiple adjacent video frames on the timeline in the same video file, or it can be a frame map corresponding to different video frames in multiple different video files. For example: after the neural network training system obtains video file A, it extracts 5 video frames at the 5th second, 10th second, 15th second, 20th second, and 25th second of video file A, respectively, to obtain the frame maps corresponding to these 5 video frames. For another example: after the neural network training system obtains video file A and video file B, it extracts 5 video frames at the 5th second, 10th second, 15th second, 20th second, and 25th second of video file A, respectively, and extracts 5 video frames at the 15th second, 20th second, 25th second, 30th second, and 35th second of video file B, respectively, to obtain the frame maps corresponding to the above 10 video frames.

[0140] It is understandable that the time for extracting video frames from a video file can be set according to actual business needs or can be random, and the embodiments of the present application do not limit this.

[0141] Adjacent video frames extracted from the same video file can be considered similar, and the corresponding frame images can be used as positive samples for image neural network training. For example, if three video frames are extracted at the 3rd, 4th, and 5th seconds of video file A, these three frames can be considered adjacent, and the corresponding frame images can be used as positive samples for image neural network training.

[0142] Video frames extracted from different video files or non-adjacent video frames extracted from the same video file can be regarded as dissimilar video frames. The extracted frame images corresponding to dissimilar video frames can be used as negative samples for image neural network training.

[0143] It is understandable that the frame extraction images in the video file may also include images set by the content producer, for example, the cover image of the video file set by the content producer, etc., which is not limited here.

[0144] The picture of the graphic information can be: the cover picture of the graphic information, the main picture of the graphic information, or the summary picture of the graphic information. The graphic information includes but is not limited to self-media articles, self-media atlases, or files including pictures and texts, web pages including pictures and texts, etc. Exemplarily, taking the graphic information as a self-media article as an example, the content producer can manually set a cover picture for the self-media article, and the cover picture can also be a new picture generated based on the graphic information according to a preset algorithm. When the graphic information includes multiple pictures, the main picture of the graphic information can be a picture manually set by the content producer, or it can be one or more pictures randomly selected from the graphic information in the neural network training system, and there is no restriction here. The summary picture of the graphic information is similar to the cover picture of the graphic information, that is, the summary picture of the graphic information can be a picture manually set by the content producer, or it can be a new picture generated based on the graphic information according to a preset algorithm, or it can be a picture selected from the graphic information by the neural network training system, and there is no restriction here.

[0145] In combination with the foregoing embodiments, a neural network training method proposed in an embodiment of the present application further includes: obtaining multiple video files; performing deduplication processing on the multiple video files to obtain multiple deduplicated video files; and extracting frame images from the video files from the multiple deduplicated video files.

[0146] Specifically, after acquiring multiple video files, the neural network training system performs deduplication processing based on the video file fingerprint information. This process results in multiple distinct video files. The system then extracts frame images from these deduplicated video files to ensure the diversity of the pre-training data.

[0147] In conjunction with the aforementioned embodiments, the present application proposes a neural network training method that constructs an image neural network based on a pre-trained model, including performing knowledge distillation on the pre-trained model to obtain an image neural network. This distillation improves the generalization of the neural network model, reduces its size, and improves its performance after deployment.

[0148] In combination with the foregoing embodiments, a neural network training method proposed in an embodiment of the present application, after training the pre-training model using a comparative learning method based on pre-training data, the method also includes: obtaining business samples, the business samples including one or more pictures, and identification information corresponding to the pictures; using the business samples to fine-tune the pre-training model (Fintune) to obtain a fine-tuned pre-training model. Specifically, a shallow network such as an LR model or an SVM model is added after the pre-training model, and then a small amount of business sample data is used to retrain and update some parameters of the model. The business sample data can be multiple pictures and label information corresponding to the multiple pictures.

[0149] For easier understanding, see Figure 5 , Figure 5 Schematic diagram of the process of training the pre-training model in the embodiment of the present application. The first step is to obtain pre-training data, which includes one or more of the following: frame extraction images in video files, images of graphic information, public images and images included in the data set (and label information corresponding to the above images). Then, the image is enhanced, and the image enhancement processing includes but is not limited to: cropping and resizing, color distortion or Gaussian blur to increase the generalization of the pre-training model. Then, the neural network training system uses the pre-training data after image enhancement processing to perform self-supervisory training on the pre-training model using contrast learning. The pre-training model obtained by self-supervised training is then fine-tuned using business samples with label information. The pre-training model obtained after fine-tuning can be distilled using business samples (including business samples with label information and business samples without label information) to obtain a picture neural network.

[0150] In combination with the foregoing embodiments, an embodiment of the present application proposes a neural network training method, wherein the multiple pictures include public pictures, and the method further includes: obtaining a first label information set, the first label information set including one or more of the following: label information of one or more video files, or label information of one or more graphic information; performing deduplication processing on the first label information set to obtain target label information; obtaining multiple public pictures to be screened through a search engine based on the target label information; and eliminating pictures with a resolution lower than a first threshold from the multiple public pictures to be screened to obtain public pictures.

[0151] For example, if the first threshold is 200*200 (pixels), then public images to be screened that are below the first threshold are eliminated. The remaining public images to be screened are used as the final public images, saving image storage costs.

[0152] The neural network training device in this application is described in detail below. Figure 6 , Figure 6 This is a schematic diagram of a neural network training device proposed in an embodiment of the present application. The neural network training device includes:

[0153] The transceiver module 601 is used to obtain pre-training data. The pre-training data includes multiple images and label information corresponding to the multiple images. The multiple images include one or more of the following:

[0154] The frame extraction image in the video file, wherein the label information corresponding to the frame extraction image is the label information of the video file,

[0155] Or, a picture of graphic information, wherein the tag information corresponding to the picture of graphic information is the title or summary information of the graphic information,

[0156] Alternatively, a public image is searched based on target tag information, wherein the target tag information is used as tag information corresponding to the public image;

[0157] The processing module 602 is used to perform self-supervised training on the pre-trained model based on the pre-trained data to obtain an image neural network, wherein the image neural network is used to obtain embedding features of the image.

[0158] In another implementation of another aspect of the embodiment of the present application, the neural network training apparatus further includes:

[0159] The processing module 602 is further used to perform self-supervisory training on the pre-training model using a comparative learning method based on the pre-training data to obtain an image neural network, wherein the identification information corresponding to multiple images in the pre-training data is used as a supervisory signal in the training.

[0160] In another implementation of another aspect of the embodiment of the present application, the frame extraction graph in the video file includes one or more of the following: frame extraction graphs corresponding to a plurality of adjacent video frames in the same video file, or frame extraction graphs corresponding to video frames in different video files;

[0161] The images of the graphic information include one or more of the following: a cover image of the graphic information, a body image of the graphic information, or a summary image of the graphic information.

[0162] In another implementation of another aspect of the embodiment of the present application, the neural network training apparatus further includes:

[0163] The transceiver module 601 is also used to obtain multiple video files;

[0164] The processing module 602 is further configured to perform deduplication processing on the multiple video files to obtain multiple deduplicated video files;

[0165] The processing module 602 is further configured to extract frame images from the multiple video files after deduplication.

[0166] In another implementation of another aspect of the embodiment of the present application, the neural network training apparatus further includes:

[0167] The processing module 602 is also used to perform knowledge distillation on the pre-trained model to obtain an image neural network.

[0168] In another implementation of another aspect of the embodiment of the present application, the neural network training apparatus further includes:

[0169] The transceiver module 601 is further configured to obtain a service sample, where the service sample includes one or more pictures and identification information corresponding to the pictures;

[0170] The processing module 602 is further configured to use the business sample to fine-tune the pre-trained model to obtain a fine-tuned pre-trained model.

[0171] In another implementation of another aspect of the embodiment of the present application, the neural network training apparatus further includes:

[0172] The transceiver module 601 is further configured to obtain a first tag information set, where the first tag information set includes one or more of the following: tag information of one or more video files, or tag information of one or more graphic and text information;

[0173] The processing module 602 is further configured to perform deduplication processing on the first tag information set to obtain target tag information;

[0174] The processing module 602 is further configured to obtain a plurality of public images to be screened through a search engine based on the target tag information;

[0175] The processing module 602 is further configured to eliminate pictures with a resolution lower than a first threshold from the plurality of public pictures to be screened, to obtain public pictures.

[0176] Figure 77 is a schematic diagram of a server structure provided in an embodiment of the present application. The server 700 may have relatively large differences due to different configurations or performances, and may include one or more central processing units (CPUs) 722 (for example, one or more processors) and memories 732, and one or more storage media 730 (for example, one or more massive storage devices) for storing application programs 742 or data 744. Among them, the memories 732 and the storage media 730 may be temporary storage or permanent storage. The program stored in the storage medium 730 may include one or more modules (not shown in the figure), and each module may include a series of instruction operations on the server. Furthermore, the central processing unit 722 may be configured to communicate with the storage medium 730 to execute a series of instruction operations in the storage medium 730 on the server 700.

[0177] The server 700 may further include one or more power supplies 726, one or more wired or wireless network interfaces 750, one or more input and output interfaces 758, or one or more operating systems 741, such as Windows Server 2003 or Windows Server 2003R. TM , Mac OS X TM , Unix TM ,Linux TM , FreeBSD TM etc.

[0178] The steps performed by the server in the above embodiment can be based on the Figure 7 The server structure shown.

[0179] Figure 8 This is a schematic diagram of a terminal device structure provided by an embodiment of the present application. Figure 8 For ease of explanation, only the parts related to the embodiments of the present application are shown. For specific technical details not disclosed, please refer to the method part of the embodiments of the present application. The terminal device can be any terminal device including a mobile phone, tablet computer, personal digital assistant (PDA), point of sales (POS), car computer, etc., taking the mobile phone as an example:

[0180] Figure 8 The block diagram shows a partial structure of a mobile phone related to the terminal device provided in the embodiment of the present application. Figure 8The mobile phone includes components such as a radio frequency (RF) circuit 810, a memory 820, an input unit 830, a display unit 840, a sensor 850, an audio circuit 860, a wireless fidelity (WiFi) module 870, a processor 880, and a power supply 890. Those skilled in the art will understand that Figure 8 The mobile phone structure shown in the figure does not constitute a limitation to the mobile phone, and may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently.

[0181] The following combination Figure 8 A detailed introduction to the various components of a mobile phone:

[0182] The RF circuit 810 can be used to receive and send signals during information transmission or calls. In particular, after receiving the downlink information from the base station, it is sent to the processor 880 for processing; in addition, the designed uplink data is sent to the base station. Generally, the RF circuit 810 includes but is not limited to an antenna, at least one amplifier, a transceiver, a coupler, a low noise amplifier (LNA), a duplexer, etc. In addition, the RF circuit 810 can also communicate with the network and other devices through wireless communication. The above-mentioned wireless communication can use any communication standard or protocol, including but not limited to Global System of Mobile communication (GSM), General Packet Radio Service (GPRS), Code Division Multiple Access (CDMA), Wideband Code Division Multiple Access (WCDMA), Long Term Evolution (LTE), email, Short Messaging Service (SMS), etc.

[0183] The memory 820 can be used to store software programs and modules. The processor 880 executes the various functional applications and data processing of the mobile phone by running the software programs and modules stored in the memory 820. The memory 820 may mainly include a program storage area and a data storage area. The program storage area may store an operating system and at least one application required for a function (such as a sound playback function, an image playback function, etc.); the data storage area may store data created based on the use of the mobile phone (such as audio data, a phone book, etc.). In addition, the memory 820 may include high-speed random access memory and may also include non-volatile memory, such as at least one disk storage device, a flash memory device, or other volatile solid-state storage device.

[0184] The input unit 830 can be used to receive input digital or character information, and to generate key signal input related to the user settings and function control of the mobile phone. Specifically, the input unit 830 may include a touch panel 831 and other input devices 832. The touch panel 831, also known as a touch screen, can collect user touch operations on or near it (such as operations performed by the user using any suitable object or accessory such as a finger, stylus, etc. on or near the touch panel 831) and drive the corresponding connection device according to a pre-set program. Optionally, the touch panel 831 may include two parts: a touch detection device and a touch controller. Among them, the touch detection device detects the user's touch direction, detects the signal caused by the touch operation, and transmits the signal to the touch controller; the touch controller receives the touch information from the touch detection device and converts it into touch point coordinates, which are then sent to the processor 880, and can receive commands sent by the processor 880 and execute them. In addition, the touch panel 831 can be implemented using various types such as resistive, capacitive, infrared, and surface acoustic wave. In addition to the touch panel 831, the input unit 830 may further include other input devices 832. Specifically, the other input devices 832 may include, but are not limited to, one or more of a physical keyboard, function keys (such as volume control keys, switch keys, etc.), a trackball, a mouse, and a joystick.

[0185] The display unit 840 can be used to display information input by the user or information provided to the user and various menus of the mobile phone. The display unit 840 may include a display panel 841. Optionally, the display panel 841 may be configured in the form of a liquid crystal display (LCD), an organic light-emitting diode (OLED), etc. Further, the touch panel 831 may cover the display panel 841. When the touch panel 831 detects a touch operation on or near it, it is transmitted to the processor 880 to determine the type of touch event. Subsequently, the processor 880 provides corresponding visual output on the display panel 841 according to the type of touch event. Although in Figure 8 In the embodiment, the touch panel 831 and the display panel 841 are used as two independent components to realize the input and output functions of the mobile phone, but in some embodiments, the touch panel 831 and the display panel 841 can be integrated to realize the input and output functions of the mobile phone.

[0186] The mobile phone may also include at least one sensor 850, such as a light sensor, a motion sensor, and other sensors. Specifically, the light sensor may include an ambient light sensor and a proximity sensor, wherein the ambient light sensor may adjust the brightness of the display panel 841 according to the brightness of the ambient light, and the proximity sensor may turn off the display panel 841 or the backlight when the mobile phone is moved to the ear. As a type of motion sensor, the accelerometer sensor can detect the magnitude of acceleration in all directions (generally three axes), and can detect the magnitude and direction of gravity when stationary. It can be used for applications that identify the posture of the mobile phone (such as horizontal and vertical screen switching, related games, magnetometer posture calibration), vibration recognition related functions (such as pedometer, tapping), etc.; as for other sensors that the mobile phone can also be configured with, such as gyroscopes, barometers, hygrometers, thermometers, infrared sensors, etc., they will not be described here.

[0187] Audio circuit 860, speaker 861, and microphone 862 provide an audio interface between the user and the phone. Audio circuit 860 converts received audio data into electrical signals and transmits them to speaker 861, which then converts them into sound signals for output. Microphone 862, on the other hand, converts collected sound signals into electrical signals, which are then received by audio circuit 860 and converted into audio data. The audio data is then processed by processor 880 and transmitted via RF circuit 810 to, for example, another phone. Alternatively, the audio data can be output to memory 820 for further processing.

[0188] WiFi is a short-range wireless transmission technology. The mobile phone can help users send and receive emails, browse graphic information and access streaming media through the WiFi module 870. It provides users with wireless broadband Internet access. Figure 8A WiFi module 870 is shown, but it is understandable that it is not an essential component of the mobile phone and can be omitted as needed without changing the essence of the invention.

[0189] The processor 880 is the control center of the mobile phone, connecting all parts of the mobile phone using various interfaces and circuits. It executes software programs or modules stored in the memory 820 and accesses data stored in the memory 820 to perform various functions and process data. Optionally, the processor 880 may include one or more processing units. Alternatively, the processor 880 may integrate an application processor and a modem processor. The application processor primarily handles the operating system, user interface, and application programs, while the modem processor primarily handles wireless communications. It is understood that the modem processor may not be integrated into the processor 880.

[0190] The mobile phone also includes a power supply 890 (such as a battery) for supplying power to various components. Optionally, the power supply can be logically connected to the processor 880 through a power management system, thereby managing functions such as charging, discharging, and power consumption through the power management system.

[0191] Although not shown, the mobile phone may also include a camera, a Bluetooth module, etc., which will not be described in detail here.

[0192] The steps performed by the terminal device in the above embodiment can be based on the Figure 8 The terminal device structure shown.

[0193] A computer-readable storage medium is also provided in an embodiment of the present application. The computer-readable storage medium stores a computer program, which, when executed on a computer, enables the computer to execute the methods described in the aforementioned embodiments.

[0194] An embodiment of the present application also provides a computer program product including a program, which, when executed on a computer, enables the computer to execute the methods described in the aforementioned embodiments.

[0195] Those skilled in the art will clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0196] In the several embodiments provided in this application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are merely schematic. For example, the division of units is only a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interface, device or unit, which can be electrical, mechanical or other forms.

[0197] Units described as separate components may or may not be physically separate, and components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected to achieve the purpose of this embodiment according to actual needs.

[0198] In addition, the functional units in the various embodiments of the present application may be integrated into a single processing unit, or each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or software functional units.

[0199] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the various embodiments of the present application. The aforementioned storage medium includes various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0200] The above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the above embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the above embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A neural network training method, characterized in that: include: Obtain pre-training data, where the pre-training data includes multiple images and label information corresponding to the multiple images, where the multiple images include one or more of the following: A frame extraction diagram in a video file, wherein the label information corresponding to the frame extraction diagram is the label information of the video file, Alternatively, a public image obtained by searching based on target tag information, wherein the target tag information is used as tag information corresponding to the public image; Performing self-supervised training on the pre-trained model based on the pre-trained data to obtain an image neural network, wherein the image neural network is used to obtain embedding features of the image; If the plurality of pictures include the public picture, the method further includes: Obtaining a first tag information set, the first tag information set including one or more of the following: tag information of one or more video files, or tag information of one or more graphic information; the tag corresponding to the picture of the graphic information is the title or summary information of the graphic information; Performing deduplication processing on the first tag information set to obtain the target tag information; Based on the target tag information, a plurality of public images to be screened are obtained through a search engine; The public pictures are obtained by eliminating pictures with a resolution lower than a first threshold from the plurality of public pictures to be screened.

2. The method according to claim 1, characterized in that Performing self-supervisory training on the pre-training model according to the pre-training data to obtain the image neural network includes: Based on the pre-training data, the pre-training model is self-supervised trained using a comparative learning method to obtain the image neural network, wherein the identification information corresponding to the multiple images in the pre-training data is used as a supervisory signal in training.

3. The method according to any one of claims 1 to 2, characterized in that The frame extraction graph in the video file includes one or more of the following: frame extraction graphs corresponding to a plurality of adjacent video frames in the same video file, or frame extraction graphs corresponding to video frames in different video files; The pictures of the graphic and text information include one or more of the following: a cover picture of the graphic and text information, a body picture of the graphic and text information, or a summary picture of the graphic and text information.

4. The method according to any one of claims 1 to 2, characterized in that The method further comprises: Get multiple video files; Performing deduplication processing on the multiple video files to obtain multiple deduplicated video files; Extracting frame images from the multiple video files after deduplication.

5. The method according to claim 2, characterized in that After training the pre-training model using a contrastive learning method based on the pre-training data, the method further includes: Perform knowledge distillation on the pre-trained model to obtain the image neural network.

6. The method according to claim 2, characterized in that After training the pre-training model using a contrastive learning method based on the pre-training data, the method further includes: Obtaining a business sample, where the business sample includes one or more pictures and identification information corresponding to the pictures; The pre-trained model is fine-tuned using the business sample to obtain the fine-tuned pre-trained model.

7. A neural network training device, characterized in that: include: The transceiver module is used to obtain pre-training data, where the pre-training data includes multiple images and label information corresponding to the multiple images, where the multiple images include one or more of the following: A frame extraction diagram in a video file, wherein the label information corresponding to the frame extraction diagram is the label information of the video file, Alternatively, a public image obtained by searching based on target tag information, wherein the target tag information is used as tag information corresponding to the public image; A processing module, configured to perform self-supervised training on the pre-trained model based on the pre-trained data to obtain an image neural network, wherein the image neural network is configured to obtain embedding features of the image; The neural network training device further includes: The transceiver module is further configured to obtain a first tag information set, wherein the first tag information set includes one or more of the following: tag information of one or more video files, or tag information of one or more graphic information; the tag corresponding to the picture of the graphic information is the title or summary information of the graphic information; The processing module is further configured to perform deduplication processing on the first tag information set to obtain the target tag information; The processing module is further configured to obtain a plurality of public images to be screened through a search engine based on the target tag information; The processing module is further configured to eliminate pictures with a resolution lower than a first threshold from the plurality of public pictures to be screened, to obtain the public pictures.

8. The device according to claim 7, characterized in that The neural network training device further includes: The processing module is further used to perform self-supervised training on the pre-training model using a comparative learning method based on the pre-training data to obtain the image neural network, wherein the identification information corresponding to the multiple images in the pre-training data is used as a supervisory signal in training.

9. The device according to any one of claims 7-8, characterized in that The frame extraction graph in the video file includes one or more of the following: frame extraction graphs corresponding to a plurality of adjacent video frames in the same video file, or frame extraction graphs corresponding to video frames in different video files; The pictures of the graphic and text information include one or more of the following: a cover picture of the graphic and text information, a body picture of the graphic and text information, or a summary picture of the graphic and text information.

10. The device according to any one of claims 7-8, characterized in that The neural network training device further includes: The transceiver module is further used to obtain multiple video files; The processing module is further configured to perform deduplication processing on the plurality of video files to obtain a plurality of deduplicated video files; The processing module is further configured to extract frame extraction images from the multiple video files after deduplication.

11. The device according to claim 8, characterized in that The neural network training device further includes: The processing module is also used to perform knowledge distillation processing on the pre-trained model to obtain an image neural network.

12. The device according to claim 8, characterized in that The neural network training device further includes: The transceiver module is further configured to obtain a service sample, wherein the service sample includes one or more pictures and identification information corresponding to the pictures; The processing module is further configured to use the business sample to fine-tune the pre-training model to obtain the fine-tuned pre-training model.

13. A computer device, characterized in that: include: Memory, processor, and bus system; Wherein, the memory is used to store programs; The processor is configured to execute the program in the memory, and the processor is configured to execute the method according to any one of claims 1 to 6 according to instructions in the program code; The bus system is used to connect the memory and the processor so that the memory and the processor can communicate with each other.

14. A computer-readable storage medium comprising instructions, which, when executed on a computer, causes the computer to perform the method according to any one of claims 1 to 6.

15. A computer program product comprising a computer program and instructions, characterized in that When the computer program / instructions are executed by a processor, the method according to any one of claims 1 to 6 is implemented.

Images