Content processing method and device, electronic equipment, chip and storage medium

Abstract

The invention provides a content processing method and device, electronic equipment, a chip and a storage medium, and relates to the field of artificial intelligence, and the method comprises the steps: carrying out the feature extraction of an input feature map through at least one convolution layer of serial processing in a content processing model, so as to obtain an output feature map; wherein a plurality of convolution kernels in any convolution layer adopt corresponding convolution algorithms to perform convolution operation on the input feature map or a middle feature map obtained by feature extraction of the previous convolution layer in parallel, and the convolution algorithms corresponding to at least two convolution kernels are different; and performing prediction processing on the output feature map through a prediction network in the content processing model to obtain a prediction result. Therefore, each convolution kernel can select the most suitable convolution algorithm according to own characteristics, and the advantages of different convolution algorithms in calculation efficiency and calculation precision are brought into full play, so that the overall calculation efficiency and calculation precision are improved, the calculation time and resource consumption of the content processing model are reduced, and the power consumption of equipment is reduced.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular to a content processing method, apparatus, electronic device, chip, and storage medium. Background Technology

[0002] With the continuous development of artificial intelligence technology, deep learning has become a core driving force for progress in many fields, and is widely used in key areas such as computer vision and natural language processing. Taking computer vision as an example, tasks such as image classification, object detection, and semantic segmentation are crucial for practical applications such as intelligent security, autonomous driving, and medical image analysis. In these tasks, deep learning models extract and analyze features from input data through multiple convolutional layers, thereby achieving accurate identification and classification of targets. Summary of the Invention

[0003] This application proposes a content processing method, apparatus, electronic device, chip, and storage medium to at least partially solve one of the technical problems in the related art.

[0004] One embodiment of this application proposes a content processing method. The convolutional network in the content processing model includes at least one convolutional layer for sequential processing. Each convolutional layer includes multiple convolutional kernels. The method includes: extracting features from an input feature map through at least one convolutional layer in the content processing model to obtain an output feature map; wherein multiple convolutional kernels in any convolutional layer perform convolution operations in parallel on the input feature map or an intermediate feature map obtained from feature extraction in the previous convolutional layer using corresponding convolutional algorithms, and at least two convolutional kernels correspond to different convolutional algorithms; the input feature map is determined based on first input data; the first input data includes text or an image; and the output feature map is predicted through a prediction network in the content processing model to obtain a prediction result.

[0005] Another embodiment of this application proposes a content processing apparatus. The convolutional network in the content processing model includes at least one convolutional layer that processes sequentially. Each convolutional layer includes multiple convolutional kernels. The apparatus includes: a feature extraction module, used to extract features from an input feature map through at least one convolutional layer in the content processing model to obtain an output feature map; wherein multiple convolutional kernels in any convolutional layer use corresponding convolutional algorithms to perform convolution operations in parallel on the input feature map or an intermediate feature map obtained from feature extraction in the previous convolutional layer, and at least two convolutional kernels correspond to different convolutional algorithms; the input feature map is determined based on first input data; the first input data includes text or an image; and a prediction module, used to perform prediction processing on the output feature map through a prediction network in the content processing model to obtain a prediction result.

[0006] Another embodiment of this application provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, it implements the content processing method as described in the foregoing aspect.

[0007] Another aspect of this application provides a chip including an interface circuit and a processing circuit coupled to each other, the interface circuit being used to input or output signals, and the processing circuit being configured to perform the content processing method as described in the foregoing aspect.

[0008] In another aspect of this application, a non-transitory computer-readable storage medium is provided, on which computer program instructions are stored, which, when executed by a processor, implement the content processing method as described in the foregoing aspect.

[0009] Another aspect of this application provides a computer program product having a computer program stored thereon, which, when executed by a processor, implements the content processing method as described in the foregoing aspect.

[0010] The content processing method, apparatus, electronic device, chip, and storage medium proposed in this application include a content processing model with a convolutional network comprising at least one convolutional layer for sequential processing. Multiple convolutional kernels in each convolutional layer employ corresponding convolutional algorithms to perform convolution operations in parallel on the input feature map or an intermediate feature map extracted from the previous convolutional layer, thereby extracting an output feature map. At least two convolutional kernels employ different convolutional algorithms, allowing each kernel to select the most suitable algorithm based on its own characteristics. This fully leverages the advantages of different convolutional algorithms in terms of computational efficiency and accuracy, thereby improving overall computational efficiency and accuracy, reducing the computation time and resource consumption of the content processing model, and reducing device power consumption. Furthermore, because different convolutional algorithms can extract and analyze features from the input feature map from different perspectives, the final extracted output feature map has stronger representational capabilities and contains richer feature information. This enables the prediction network in the content processing model to generate more accurate prediction results based on the output feature map, better meeting the diverse performance and experience requirements of practical application scenarios.

[0011] Additional aspects and advantages of this application will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of this application. Attached Figure Description

[0012] The above and / or additional aspects and advantages of this application will become apparent and readily understood from the following description of the embodiments taken in conjunction with the accompanying drawings, wherein: Figure 1 A schematic flowchart illustrating a content processing method provided for an exemplary embodiment of this application; Figure 2 A flowchart illustrating another content processing method provided for an exemplary embodiment of this application; Figure 3 A flowchart illustrating yet another content processing method provided for an exemplary embodiment of this application; Figure 4 A schematic flowchart illustrating another content processing method provided for an exemplary embodiment of this application; Figure 5 A schematic diagram of a standard convolution process provided for an exemplary embodiment of this application; Figure 6 A schematic diagram of a hybrid inference process using Direct conv and Winograd conv provided for an exemplary embodiment of this application; Figure 7 A schematic diagram of a single operator computation process provided for an exemplary embodiment of this application; Figure 8 A schematic diagram of a hardware implementation architecture provided for an exemplary embodiment of this application; Figure 9 A schematic diagram of the structure of a content processing apparatus provided for an exemplary embodiment of this application; Figure 10 A schematic diagram of the structure of an electronic device provided for an exemplary embodiment of this application; Figure 11 This is a schematic diagram of the structure of a chip proposed in an exemplary embodiment of this application. Detailed Implementation

[0013] The embodiments of this application are described in detail below. Examples of these embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and intended to explain this application, and should not be construed as limiting this application.

[0014] Deep learning models face numerous challenges when performing forward inference on hardware, particularly in terms of power consumption and performance. Deep learning models typically consist of a large number of neurons and complex connections, and their computation involves massive amounts of floating-point operations. In hardware implementation, floating-point operations require complex circuit structures, which not only consume significant chip area but also lead to high power consumption. Taking convolutional neural networks, a common deep learning technique, as an example, the numerous floating-point multiplication and addition operations during convolution operations drastically increase hardware power consumption. Furthermore, due to the complexity of floating-point operations, the computation speed is relatively slow, making it difficult to meet the demands of applications with high real-time requirements.

[0015] To address these issues, quantization techniques emerged. Quantization converts weights and activation values ​​in deep learning models from high-precision floating-point numbers (such as fp32) to low-bit integers (such as int8), effectively reducing the model's storage requirements and data transfer volume. In terms of hardware implementation, low-bit integer operations can utilize simpler circuit structures, such as integer arithmetic units, significantly reducing chip area and power consumption. Simultaneously, the reduced data bit width accelerates data transfer speeds within the hardware, contributing to improved overall system performance.

[0016] While the precision of the data is reduced when performing convolution calculations in the quantized model, the speed and efficiency of convolution calculations can be significantly improved while maintaining model precision through reasonable quantization strategies and dequantization operations. Specifically, the quantization process follows the path of "fp32->quant->int8_conv->int32->dequant->fp32". First, the deep learning model is trained in floating-point (fp32) format to obtain baseline precision. Then, the weights and activation values ​​in the model are quantized (quantized) to determine the quantization bit width, quantization parameters (such as scaling factor, zero point), etc., and converted to low-bit integers (such as int8). Next, convolution calculations are performed in the int8 quantized model. Due to the use of low-bit integer operations, the amount of computation and power consumption are greatly reduced. The low-bit integer result (int32) obtained from the convolution calculation is then dequantized (dequantized) to convert it back to floating-point format for subsequent network layer calculations or output.

[0017] In related technologies, each convolutional layer in the convolutional network of deep learning models contains multiple convolutional kernels. When these convolutional kernels perform convolution calculations on the input feature map, they all use the same convolution algorithm, such as the ordinary convolution (Direct conv) algorithm. The Direct conv algorithm is a relatively basic and general convolution calculation method. It obtains the output feature map by sliding the convolutional kernel to perform element-wise multiplication and addition operations on the input feature map.

[0018] Although this unified convolution algorithm design simplifies the model implementation and computation process to some extent, enabling the model to have a relatively stable computation mode during training and inference, it cannot balance computational accuracy and computational performance.

[0019] Therefore, in view of at least one of the problems existing in the above-mentioned related technologies, this application proposes a content processing method, apparatus, electronic device, chip and storage medium.

[0020] The following description, with reference to the accompanying drawings, describes content processing methods, apparatuses, electronic devices, chips, and storage media according to embodiments of this application.

[0021] Figure 1 This is a schematic flowchart of a content processing method provided for an exemplary embodiment of this application.

[0022] It should be noted that the content processing method of this application embodiment can be applied to a content processing device. In some possible embodiments, the content processing device can be configured in an electronic device or a chip so that the electronic device or chip can perform content processing functions. Additionally, in some possible embodiments, the content processing device can also be software in an electronic device.

[0023] In any embodiment of this application, the chip can be integrated into an electronic device. The chip includes a Central Processing Unit (CPU), an Image Signal Processing (ISP), an Application-Specific Integrated Circuit (ASIC), a Digital Signal Processor (DSP), a Field-Programmable Gate Array (FPGA), a System-on-Chip (SOC), a Reduced Instruction Set Computer (RISC), etc., which will not be listed here.

[0024] The electronic devices mentioned include, but are not limited to, terminals, personal computers, and servers. A terminal is a user-side entity used to receive or transmit signals, such as a mobile phone. Terminals can also be called terminal equipment (UE), user equipment (UE), mobile station (MS), mobile terminal equipment (MT), etc. Terminals can be communication-enabled vehicles, smart cars, mobile phones, wearable devices, tablets, computers with wireless transceiver capabilities, virtual reality (VR) terminals, augmented reality (AR) terminals, wireless terminals in industrial control, wireless terminals in self-driving, wireless terminals in remote medical surgery, wireless terminals in smart grids, wireless terminals in transportation safety, wireless terminals in smart cities, wireless terminals in smart homes, etc. The embodiments of this application do not limit the specific technology or device form used in the terminal.

[0025] like Figure 1 As shown, the content processing method may include the following steps S101 to S102: Step S101: Extract features from the input feature map using at least one convolutional layer in the content processing model to obtain an output feature map; wherein, multiple convolutional kernels in any convolutional layer perform convolution operations on the input feature map or the intermediate feature map obtained by feature extraction from the previous convolutional layer in parallel using corresponding convolutional algorithms, and at least two convolutional kernels have different convolutional algorithms; the input feature map is determined based on the first input data; the first input data includes text or images.

[0026] The content processing model includes a convolutional network, which includes at least one convolutional layer for serial processing. Each convolutional layer includes multiple convolutional kernels, which are used to process the feature maps input to each kernel in parallel.

[0027] Each convolutional layer contains multiple convolutional kernels, each with its own convolutional algorithm. At least two kernels in each convolutional layer may use different convolutional algorithms.

[0028] Convolution algorithms include, but are not limited to, Direct conv and Winograd conv. Among them, the Winograd conv algorithm is an optimized convolution algorithm that reduces the amount of computation and improves the computation speed by converting the convolution operation into a series of smaller matrix multiplication operations.

[0029] The input feature map refers to the feature map input to the first convolutional layer in the convolutional network. This input feature map is determined based on the input data of the content processing model (referred to as the first input data in this application).

[0030] The first input data can be data content input or provided by the user, including but not limited to: text and images. For example, taking an image as the first input data, the content processing model can be an image processing model, which includes, but is not limited to: object detection models, image segmentation models (or semantic segmentation models), image classification models, and Optical Character Recognition (OCR) models. Taking text as the first input data, the content processing model can be a text processing model, which includes, but is not limited to: text classification and annotation models (such as sentiment classification models, topic classification models, intent recognition models, and named entity recognition models), sequence generation and conversion models (such as machine translation models, text summarization models, text generation models, and text error correction models), and semantic understanding and reasoning models (such as question answering models and information extraction models).

[0031] In this embodiment of the application, features can be extracted from the input feature map through at least one convolutional layer in the content processing model to obtain an output feature map. Specifically, for the first convolutional layer in the at least one convolutional layer, multiple convolutional kernels in the first convolutional layer perform convolution operations on the input feature map in parallel using corresponding convolution algorithms to obtain an intermediate feature map extracted by the first convolutional layer; for non-first convolutional layers in the at least one convolutional layer, multiple convolutional kernels in the non-first convolutional layer perform convolution operations on the intermediate feature map extracted by the previous convolutional layer in parallel using corresponding convolution algorithms to obtain an intermediate feature map extracted by the non-first convolutional layer.

[0032] The output feature map is determined based on the intermediate feature map extracted from the last convolutional layer in at least one convolutional layer.

[0033] Step S102: The output feature map is processed by the prediction network in the content processing model to obtain the prediction result.

[0034] In addition to convolutional networks, the content processing model can also include prediction networks.

[0035] In this embodiment of the application, the output feature map can be processed by a prediction network to obtain the prediction result, which is the final output result of the content processing model.

[0036] As an application scenario, taking an image as the first input data and a target detection model as the content processing model as an example, the prediction network can perform classification prediction and regression prediction on the output feature map. The prediction results include: the location information of at least one detection box, and the category to which the target in the detection box belongs.

[0037] As another application scenario, taking an image as the first input data and an image segmentation model as the content processing model as an example, the prediction network can perform semantic segmentation on the output feature map. The prediction results include: pixel-level category labels (the category label of each pixel in the image, such as "person", "vehicle", "road", "sky", etc., indicating the object to which the pixel belongs) and semantic mask (a binary or multi-valued matrix with the same size as the original image, where the value of each pixel is used to indicate the category to which the corresponding pixel belongs in the original image).

[0038] As another application scenario, taking an image as the first input data and an image classification model as the content processing model, the prediction network can perform semantic classification on the output feature map. The prediction results include the category to which each object in the image belongs, such as "person", "vehicle", "road", "sky" etc.

[0039] As another application scenario, taking an image as the first input data and an OCR model as the content processing model, the prediction network can perform OCR recognition on the output feature map, and the prediction results include: each character in the image.

[0040] As another application scenario, taking text as the first input data and sentiment classification model as the content processing model, the prediction network can perform sentiment classification on the output feature map. The prediction results include: text sentiment tendency (such as positive, negative, neutral) or refined emotions (anger, joy, sadness, etc.).

[0041] As another application scenario, taking text as the first input data and intent recognition model as the content processing model as an example, the prediction network can perform intent recognition on the output feature map, and the prediction results include: user intent.

[0042] As another application scenario, taking text as the first input data and text correction model as the content processing model, the prediction network can perform text correction on the output feature map, and the prediction results include: text correction results.

[0043] As another application scenario, taking text as the first input data and information extraction model as the content processing model, the prediction network can extract information from the output feature map, and the prediction results include: structured information in the text (such as events and relationships).

[0044] The content processing method in this embodiment includes a convolutional network of a content processing model comprising at least one convolutional layer processed sequentially. Multiple convolutional kernels in each convolutional layer employ corresponding convolutional algorithms to perform convolution operations in parallel on the input feature map or an intermediate feature map extracted from the previous convolutional layer, thereby extracting an output feature map. At least two convolutional kernels use different convolutional algorithms, allowing each kernel to select the most suitable algorithm based on its own characteristics. This fully leverages the advantages of different convolutional algorithms in terms of computational efficiency and accuracy, thereby improving overall computational efficiency and accuracy, reducing the computation time and resource consumption of the content processing model, and decreasing device power consumption. Furthermore, because different convolutional algorithms can extract and analyze features from the input feature map from different perspectives, the final extracted output feature map has stronger representational capabilities and contains richer feature information. This enables the prediction network in the content processing model to generate more accurate prediction results based on the output feature map, better meeting the diverse performance and experience requirements of real-world application scenarios.

[0045] As one possible implementation method, Figure 2 This is a flowchart illustrating another content processing method provided for an exemplary embodiment of this application. It should be noted that this content processing method can be executed alone, or it can be executed together with any embodiment or possible implementation thereof in this application, or it can be executed together with any technical solution in related technologies. This application does not impose any limitations on this.

[0046] like Figure 2 As shown, the content processing method may include the following steps S201 to S203: Step S201: Determine a content processing model that matches the user's content processing needs. The convolutional network in the content processing model includes at least one convolutional layer for sequential processing, and each convolutional layer includes multiple convolutional kernels.

[0047] Content processing requirements are used to indicate the user's personalized processing needs, including but not limited to: intent recognition, information extraction, object detection, and semantic segmentation. Each content processing requirement can have a matching content processing model, and different requirements can use different content processing models.

[0048] In any embodiment of this application, the content processing requirement is determined, for example, by determining the content processing requirement associated with the target interface element in response to a triggering operation on a target interface element in a graphical user interface (GUI). The triggering operation includes, but is not limited to, click operations, drag operations, etc. The target interface element includes interactive interface elements in the GUI.

[0049] For example, the GUI interface may include interface elements such as translation and OCR recognition. If the user clicks "Translate", the content processing requirement can be determined as "machine translation", and the content processing model that matches the user's content processing requirement is the machine translation model.

[0050] In any embodiment of this application, the content processing requirement is determined, for example, by: in response to input operations on interactive controls in the GUI, acquiring the input task description information, and performing intent recognition on the task description information to determine the content processing requirement. The interactive controls include, but are not limited to: voice input controls, text input controls, dialog windows, text fields, rich text editors, drop-down lists (allowing users to select one or more options from a predefined list of options as task description information in natural language), etc.

[0051] It should be understood that the above-mentioned methods for determining content processing requirements are merely illustrative examples. Other methods can also be used to determine content processing requirements. For example, users can indicate content processing requirements by dragging and dropping controllable controls in the GUI. This application embodiment does not limit this.

[0052] In summary, different methods can be used to determine the user's actual content processing needs, thus improving the flexibility and applicability of this method.

[0053] Step S202: Extract features from the input feature map using at least one convolutional layer in the content processing model to obtain an output feature map; wherein, multiple convolutional kernels in any convolutional layer perform convolution operations on the input feature map or the intermediate feature map obtained by feature extraction from the previous convolutional layer in parallel using corresponding convolutional algorithms, and at least two convolutional kernels have different convolutional algorithms; the input feature map is determined based on the first input data; the first input data includes text or images.

[0054] Step S203: The output feature map is processed by the prediction network in the content processing model to obtain the prediction result.

[0055] It should be noted that the explanations of steps S202 to S203 can be found in the relevant descriptions in any embodiment of this application, and will not be repeated here.

[0056] The content processing method in this application adopts a content processing model that matches the user's personalized content processing needs to process the first input data. This can accurately meet the user's personalized needs, improve the output quality of the model, and ensure that the processing results are more in line with the actual application scenario and user expectations.

[0057] As one possible implementation method, Figure 3 This is a flowchart illustrating another content processing method provided as an exemplary embodiment of this application. It should be noted that this content processing method can be executed alone, or it can be executed together with any embodiment or possible implementation thereof in this application, or it can be executed together with any technical solution in related technologies. This application does not impose any limitations on this.

[0058] like Figure 3 As shown, based on any embodiment of this application, any convolutional layer in at least one convolutional layer of a convolutional network includes at least one first convolutional kernel and at least one second convolutional kernel arranged in sequence, wherein the convolutional layer is used to perform the following steps S301 to S303: Step S301: Using the first convolution kernel and the first convolution algorithm, perform the first convolution operation on the feature map input to any convolutional layer to obtain the first output feature; wherein, the feature map input to any convolutional layer includes the input feature map or the intermediate feature map obtained by feature extraction from the previous convolutional layer.

[0059] Among them, the first convolution algorithm has relatively high computational accuracy, but relatively low computational performance (such as computational efficiency). For example, the first convolution algorithm includes, but is not limited to, Direct conv.

[0060] In this embodiment, a first convolution operation can be performed on the feature map input to the convolutional layer to which the first convolutional kernel belongs, using a corresponding first convolution algorithm, through each first convolutional kernel in any convolutional layer, to obtain the first output feature output by the first convolutional kernel. Wherein, if any convolutional layer is the first convolutional layer, the feature map input to the first convolutional layer is the input feature map in the above embodiment; if any convolutional layer is not the first convolutional layer, the feature map input to the non-first convolutional layer is the intermediate feature map obtained by feature extraction from the previous convolutional layer.

[0061] Step S302: Using the second convolution kernel and the second convolution algorithm, perform the second convolution operation on the feature map input to any convolutional layer to obtain the second output feature; wherein, the computational accuracy of the first convolution algorithm is higher than that of the second convolution algorithm, and the computational performance of the first convolution algorithm is lower than that of the second convolution algorithm.

[0062] The second convolution algorithm has relatively low computational accuracy but relatively high computational performance. Specifically, in this application, the first convolution algorithm has higher computational accuracy than the second convolution algorithm, and its computational performance is lower than that of the second convolution algorithm. For example, the second convolution algorithm includes, but is not limited to, Winogradconv.

[0063] In this embodiment of the application, a second convolution operation can be performed on the feature map input to the convolutional layer to which the second convolution kernel belongs by using the corresponding second convolution algorithm for each second convolution kernel in any convolutional layer, so as to obtain the second output feature output by the second convolution kernel.

[0064] Step S303: Based on the rearrangement order of multiple convolutional kernels in any convolutional layer, the first output feature and the second output feature are concatenated to obtain an intermediate feature map of the output of any convolutional layer.

[0065] In this embodiment of the application, the first output features output by each first convolutional kernel and the second output features output by each second convolutional kernel can be concatenated based on the rearranged order of multiple convolutional kernels in any convolutional layer to obtain an intermediate feature map obtained by feature extraction from any convolutional layer.

[0066] In any embodiment of this application, when any of the above convolutional layers is a non-first convolutional layer among at least one convolutional layer, considering that the intermediate feature map output by the previous convolutional layer has been spliced ​​according to the rearrangement order of multiple convolutional kernels, that is, spliced ​​according to the grouping of the output channels (oc) of multiple convolutional kernels (the first convolutional kernel is divided into one group, and the second convolutional kernel is divided into another group), this means that the channel order of the intermediate feature map output by the previous convolutional layer may have changed. If the convolutional kernel of the next convolutional layer is not adjusted accordingly, the convolutional kernel of the next convolutional layer will not be able to correctly match the channels of the input feature map.

[0067] To address the aforementioned issues, in order to maintain computational consistency, the input channels (ic) of the convolutional kernels of non-first convolutional layers need to be mapped one-to-one with the channels of the intermediate feature maps output by the previous convolutional layer. That is, before step S301, the indices of the input channels of the multiple convolutional kernels of non-first convolutional layers can be updated according to the rearrangement order of the multiple convolutional kernels of the previous convolutional layer.

[0068] For example, the input channels (ic) of non-first convolutional layers are actually the output channels (oc) of the previous convolutional layer. In the previous convolutional layer, quantization error analysis has determined which ocs use Winograd convolution and which use Direct convolution, and the indices of these ocs have been recorded. In this application, the ic index of the next convolutional kernel can be adjusted based on the oc indices recorded in the previous convolutional layer. For example, if a certain oc in the previous convolutional layer uses Winograd convolution, then in the next convolutional layer, the corresponding ic should also be marked as the output from Winograd convolution.

[0069] In any embodiment of this application, when any of the above convolutional layers is a non-first convolutional layer in at least one convolutional layer, before step S301, the indexes of the bias terms corresponding to the multiple convolutional kernels of the non-first convolutional layer can be adjusted according to the rearrangement order of the multiple convolutional kernels of the previous convolutional layer; wherein, the bias term is used to adjust the output features of the corresponding convolutional kernel.

[0070] Therefore, when using bias terms in convolutional networks, synchronously adjusting the index of the bias terms can ensure that the bias terms are correctly matched with the corresponding output channels, thereby improving the output quality of the convolutional network.

[0071] In any embodiment of this application, where any convolutional layer is the last convolutional layer in at least one convolutional layer, and the index of the feature map output by the output channel of each convolutional kernel matches the index of the output channel of the convolutional kernel, the position of the feature maps of multiple output channels in the intermediate feature map output by the last convolutional layer can be adjusted based on the original index of multiple convolutional kernels in the first convolutional layer in at least one convolutional layer, so as to obtain the output feature map. Thus, index restoration can be achieved, the normal OC order of the original convolutional network can be maintained, and the output quality of the convolutional network can be improved.

[0072] The content processing method in this application embodiment uses a first convolution algorithm with relatively high computational accuracy and a second convolution algorithm with relatively high computational performance in each convolutional layer to perform convolution operations on the feature map input to that convolutional layer. This can balance the computational accuracy and computational performance of the convolutional network and further reduce device power consumption.

[0073] As one possible implementation method, Figure 4This is a flowchart illustrating another content processing method provided as an exemplary embodiment of this application. It should be noted that this content processing method can be executed alone, or it can be executed together with any embodiment or possible implementation thereof in this application, or it can be executed together with any technical solution in related technologies. This application does not impose any limitations on this.

[0074] like Figure 4 As shown, based on any embodiment of this application, the first and second convolutional kernels in any convolutional layer of a convolutional network can be determined (or divided) and rearranged using the following steps S401 to S403: Step S401: Obtain the first quantization error of multiple convolution kernels in any convolutional layer using the first convolution algorithm, and the second quantization error of multiple convolution kernels in any convolutional layer using the second convolution algorithm.

[0075] In the embodiments of this application, the first quantization error of multiple convolution kernels in any convolutional layer using the first convolution algorithm can be obtained, and the second quantization error of multiple convolution kernels in any convolutional layer using the second convolution algorithm can be obtained.

[0076] In any embodiment of this application, for any convolution kernel in any convolutional layer, the model weights of the content processing model can be quantized using the first convolution algorithm to obtain the first quantized weights, and the first quantization error can be determined based on the difference between the model weights and the first quantized weights.

[0077] The model weights can be floating-point model weights (model weights after fp32 and before quantization), and the first quantization error can be the error between the floating-point model weights (model weights after fp32 and before quantization) and the quantized model weights (i.e., model weights after quantization and before int8_conv).

[0078] Similarly, the model weights can be quantized using the second convolution algorithm through this convolution kernel to obtain the second quantized weights, and the second quantization error can be determined based on the difference between the model weights and the second quantized weights.

[0079] In any embodiment of this application, a content processing model can be used to perform forward propagation calculation on the second input data to obtain a first feature map of multiple convolution kernels input to any convolutional layer. For any convolution kernel in any convolutional layer, the first feature map can be quantized by using the first convolution algorithm to obtain a second feature map, and a first quantization error between the first feature map and the second feature map can be determined.

[0080] The data modality of the second input data matches the data modality of the first input data, such as both being text or both being images. The first feature map can be a floating-point feature map (the feature map after fp32 and before quantization), and the first quantization error can be the error between the floating-point feature map (the feature map after fp32 and before quantization) and the quantized feature map (i.e., the feature map after quantization and before int8_conv).

[0081] Similarly, the first feature map can be quantized using the second convolution algorithm through the same convolution kernel to obtain the third feature map, and the second quantization error between the first and third feature maps can be determined.

[0082] In summary, different methods can be used to calculate the first quantization error of each convolution kernel using the first convolution algorithm and the second quantization error using the second convolution algorithm, thereby improving the flexibility and applicability of this method.

[0083] Step S402: Based on the first quantization error and the second quantization error corresponding to the multiple convolutional kernels in any convolutional layer, determine the first convolutional kernel and the second convolutional kernel from the multiple convolutional kernels in any convolutional layer; wherein, the first quantization error corresponding to the first convolutional kernel is less than or equal to the second quantization error, and the first quantization error corresponding to the second convolutional kernel is greater than the second quantization error.

[0084] In the embodiments of this application, for any convolutional layer with multiple convolutional kernels, a convolutional kernel with a first quantization error less than or equal to a second quantization error can be used as a first convolutional kernel, and a convolutional kernel with a first quantization error greater than a second quantization error can be used as a second convolutional kernel.

[0085] Step S403: Based on the indices of the output channels corresponding to the first and second convolution kernels, the order of the first and second convolution kernels is rearranged.

[0086] In this embodiment, the first and second convolutional kernels can be rearranged based on the indices of their respective output channels. For example, if a convolutional layer contains eight kernels, k0, k1, k2, k3, k4, k5, k6, and k7, and the indices of the output channels of the first convolutional kernels using the first convolution algorithm are 0, 2, 3, 5, and 7, and the indices of the output channels of the second convolutional kernels using the second convolution algorithm are 1, 4, and 6, then the rearranged convolutional kernels are: k0, k2, k3, k5, k7, k1, k4, and k6.

[0087] The content processing method of this application embodiment divides multiple convolution kernels based on the quantization error of the first convolution algorithm and the second convolution algorithm, to obtain a first convolution kernel using the first convolution algorithm and a second convolution kernel using the second convolution algorithm, which can improve the rationality and reliability of the division result.

[0088] In any embodiment of this application, the first convolution algorithm is Direct conv and the second convolution algorithm is Winograd conv, as examples. Considering that the computational performance of Winograd conv is 2.25 times that of Direct conv, introducing Winograd conv computation in hardware can greatly improve the overall performance. However, quantization or replacement with Winograd conv will result in a loss of precision. Therefore, when quantizing the same number of bits (this application accepts any number of bits), the quantization precision of Winograd conv is lower than that of Direct conv. That is, Winograd conv has lower computational precision but higher performance, while using Direct conv and quantization results in higher precision but lower performance. Therefore, in this application, Direct conv and Winograd conv can be used in combination to balance computational precision and performance.

[0089] Convolutional networks include, for example Figure 5 Taking the two convolutional layers shown as an example, the convolutional kernels of the first convolutional layer are denoted as layer_0_kernels (abbreviated as k0). k0_0 to k0_7 represent the 8 convolutional kernels of the first layer, meaning each kernel has 8 output channels and 3 input channels. The shape of k0 is (8, 3, 3, 3), which are (oc, ic, kh, kw), where oc is the output channel, ic is the input channel, kh is the kernel height, and kw is the kernel width. Winograd conv only accepts conv replacements with kernel_size=3 and stride=1. After convolution calculation, a feature map with 8 channels is generated. The feature map output by the first convolutional layer is denoted as layers_0_featuremaps (abbreviated as f0), and k1 and f1 are similarly denoted. Figure 5 The convolution process shown is a standard convolution process.

[0090] The hybrid inference process combining Direct conv and Winograd conv can be as follows: Figure 6As shown, before convolution, the kernel is analyzed, and the errors of Direct Convolution quantization and Winograd Convolution quantization with the floating-point parameters before quantization are compared for each objective (referred to as quantization error in this application). For each objective, the convolution algorithm with the smaller error is selected. That is, for each objective, if the quantization error of Direct Convolution is less than that of Winograd Convolution, then Direct Convolution is selected, and if the quantization error of Direct Convolution is greater than that of Winograd Convolution, then Winograd Convolution is selected.

[0091] The quantization error is calculated in two ways: First, the model weights are calculated directly in the oc direction. That is, the error between the floating-point model weights (after fp32 and before quantization) and the quantized model weights (after quantization and before int8_conv) is calculated. Second, a batch of input data is used to perform forward-backward calculations on the feature map in the oc direction, and the kernel oc corresponding to the oc of that feature is found (the two indices are consistent, i.e., feature oc equals kernel oc). That is, for each oc, the error between the floating-point feature (after fp32 and before quantization) and the quantized feature (after quantization and before int8_conv) is calculated.

[0092] For each objective (OC), a convolution algorithm with smaller quantization error is selected. For example, if an OC using Winograd convolution has a smaller quantization error, its index is recorded. After calculating the quantization error of all OCs, the OCs with smaller quantization errors using Winograd convolution are extracted and merged into one group (denoted as group 1), while the remaining OCs are in another group (group 2). After calculating the quantization errors of the two groups separately, they are concatenated before proceeding to the next layer of computation.

[0093] As an example, group 1 can adopt the following approach: Figure 7 (b) shows the Winograd conv, which performs convolution calculations on the input feature map. Group 2 can be implemented as follows: Figure 7 (b) shows the Direct conv, which performs convolution computation on the input feature map. The Winograd conv has a computational performance 2.25 times that of the Direct conv, and using the Winograd conv kernel can accelerate the computation to less than half the original time. Figure 7 (a) is a schematic diagram of the computation process of Direct conv in related technologies.

[0094] After the kernel rearrangement described above, the input of the next convolutional layer will be affected. Therefore, the ic index of the kernel in the next convolutional layer needs to be modified according to the OC index of the previous convolutional layer. The calculation method of the modified next convolutional layer is the same as the above process: first, the error is calculated in the OC direction, then the OC index of the Winograd convolution is selected, then grouped and calculated, and then concatenated, and so on. Until the output of the last convolutional layer, the index is restored to maintain the normal OC order of the original network.

[0095] It is important to note that when rearranging the convolutional kernels of intermediate convolutional layers, the index of the corresponding bias term must also be modified to correspond with the feature.

[0096] As an example, the hardware implementation principle can be as follows: Figure 8 As shown, each computing unit (CU) includes an artificial intelligence engine (AIE) and a single-point data processing module (SDP). The AIE is used to implement the quantization calculation of the neural network, and the SDP is used to implement the dequantization calculation of the neural network. A single CU means that one CU sequentially executes the quantization and dequantization calculations of Winograd conv and Direct conv. Multiple CUs means that one CU is responsible for executing the quantization and dequantization calculations of Winograd conv, while another CU executes the quantization and dequantization calculations of Direct conv in parallel.

[0097] To implement the above embodiments, this application also proposes a content processing apparatus. Figure 9 This is a schematic diagram of the structure of a content processing apparatus provided for an exemplary embodiment of this application.

[0098] like Figure 9 As shown, the convolutional network in the content processing model includes at least one convolutional layer that is processed sequentially, and each convolutional layer includes multiple convolutional kernels. The content processing device 900 may include a feature extraction module 910 and a prediction module 920.

[0099] The feature extraction module 910 is used to extract features from the input feature map through at least one convolutional layer in the content processing model to obtain an output feature map. Multiple convolutional kernels in any convolutional layer perform convolution operations in parallel on the input feature map or an intermediate feature map obtained from the feature extraction of the previous convolutional layer using corresponding convolutional algorithms, and at least two convolutional kernels use different convolutional algorithms. The input feature map is determined based on first input data, which includes text or an image. The prediction module 920 is used to perform prediction processing on the output feature map through a prediction network in the content processing model to obtain a prediction result.

[0100] In one implementation of this application, the content processing model is matched with the user's content processing needs, which are determined by the following modules: a processing module, configured to perform any of the following: in response to a trigger operation on a target interface element in a graphical user interface (GUI), determine the content processing needs associated with the target interface element; in response to an input operation on an interactive control in the GUI, obtain the input task description information and perform intent recognition on the task description information to determine the content processing needs.

[0101] In one implementation of this application, the interactive control includes at least one of the following: a voice input control, a text input control, a dialog window, a text field, a rich text editor, and a drop-down list.

[0102] In one implementation of this application, any convolutional layer includes at least one first convolutional kernel and at least one second convolutional kernel after sequential rearrangement; any convolutional layer is configured to perform the following operations: using the first convolutional kernel and a first convolutional algorithm, perform a first convolution operation on the feature map input to any convolutional layer to obtain a first output feature; wherein, the feature map input to any convolutional layer includes an input feature map or an intermediate feature map obtained from feature extraction of the previous convolutional layer; using the second convolutional kernel and a second convolutional algorithm, perform a second convolution operation on the feature map input to any convolutional layer to obtain a second output feature; wherein, the computational accuracy of the first convolutional algorithm is higher than that of the second convolutional algorithm, and the computational performance of the first convolutional algorithm is lower than that of the second convolutional algorithm; based on the rearrangement order of multiple convolutional kernels in any convolutional layer, concatenate the first output feature and the second output feature to obtain an intermediate feature map output by any convolutional layer.

[0103] In one implementation of this application, the first convolutional kernel and the second convolutional kernel are determined and rearranged using the following modules: an acquisition module, used to acquire the first quantization error of multiple convolutional kernels in any convolutional layer using the first convolution algorithm, and the second quantization error of multiple convolutional kernels in any convolutional layer using the second convolution algorithm; a determination module, used to determine the first convolutional kernel and the second convolutional kernel from multiple convolutional kernels in any convolutional layer based on the first quantization error and the second quantization error corresponding to the multiple convolutional kernels in any convolutional layer; wherein the first quantization error corresponding to the first convolutional kernel is less than or equal to the second quantization error, and the first quantization error corresponding to the second convolutional kernel is greater than the second quantization error; and a rearrangement module, used to rearrange the first convolutional kernel and the second convolutional kernel in order based on the index of the output channel corresponding to the first convolutional kernel and the second convolutional kernel respectively.

[0104] In one implementation of this application, the acquisition module is configured to: quantize the model weights of the content processing model using a first convolution algorithm on any convolution kernel in any convolutional layer to obtain a first quantized weight, and determine a first quantization error based on the difference between the model weights and the first quantized weight; quantize the model weights using a second convolution algorithm on any convolution kernel to obtain a second quantized weight, and determine a second quantization error based on the difference between the model weights and the second quantized weight.

[0105] In one implementation of this application, the acquisition module is configured to: perform forward propagation calculation on the second input data using a content processing model to obtain a first feature map input to multiple convolutional kernels in any convolutional layer; wherein the data modality of the second input data matches the data modality of the first input data; for any convolutional kernel in any convolutional layer, quantize the first feature map using a first convolution algorithm to obtain a second feature map, and determine a first quantization error between the first feature map and the second feature map; quantize the first feature map using a second convolution algorithm to obtain a third feature map, and determine a second quantization error between the first feature map and the third feature map.

[0106] In one implementation of this application, in response to any convolutional layer being a non-first convolutional layer among at least one convolutional layer, the content processing apparatus 900 may further include: an update module, configured to update the index of the input channels of the multiple convolutional kernels of the non-first convolutional layer according to the rearrangement order of the multiple convolutional kernels of the previous convolutional layer.

[0107] In one implementation of this application, the content processing device 900 may further include: a first adjustment module, configured to adjust the indexes of the bias terms corresponding to the multiple convolutional kernels of the non-first convolutional layer according to the rearrangement order of the multiple convolutional kernels of the previous convolutional layer; wherein the bias term is used to adjust the output features of the corresponding convolutional kernel.

[0108] In one implementation of this application, in response to any convolutional layer being the last convolutional layer in at least one convolutional layer, the index of the feature map output by the output channel of each convolutional kernel matches the index of the output channel of the convolutional kernel; the content processing device 900 may further include: a second adjustment module, used to adjust the position of the feature maps of multiple output channels in the intermediate feature map output by the last convolutional layer based on the original index of multiple convolutional kernels of the first convolutional layer in at least one convolutional layer, so as to obtain the output feature map.

[0109] It should be noted that the foregoing explanation of any content processing method embodiment also applies to the content processing device of that embodiment, and will not be repeated here.

[0110] In the content processing apparatus of this application embodiment, the convolutional network of the content processing model includes at least one convolutional layer processed sequentially. Multiple convolutional kernels in each convolutional layer employ corresponding convolutional algorithms to perform convolution operations in parallel on the input feature map or the intermediate feature map obtained from the feature extraction of the previous convolutional layer, thereby extracting the output feature map. At least two convolutional kernels employ different convolutional algorithms, allowing each kernel to select the most suitable algorithm based on its own characteristics. This fully leverages the advantages of different convolutional algorithms in terms of computational efficiency and accuracy, thereby improving overall computational efficiency and accuracy, reducing the computation time and resource consumption of the content processing model, and reducing device power consumption. Simultaneously, because different convolutional algorithms can extract and analyze features from the input feature map from different perspectives, the final extracted output feature map has stronger representational capabilities and contains richer feature information. This enables the prediction network in the content processing model to generate more accurate prediction results based on the output feature map, better meeting the diverse performance and experience requirements in practical application scenarios.

[0111] To implement the above embodiments, this application also proposes an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, it implements the content processing method as described in any of the foregoing embodiments.

[0112] Figure 10This is a schematic diagram of the structure of an electronic device provided for an exemplary embodiment of this application. For example, the electronic device 1000 may be a mobile phone, computer, digital broadcasting terminal, messaging device, game console, tablet device, medical device, fitness equipment, personal digital assistant, etc.

[0113] Reference Figure 10 The electronic device 1000 may include one or more of the following components: a processing component 1002, a memory 1004, a power component 1006, a multimedia component 1008, an audio component 1010, an input / output (I / O) interface 1012, a sensor component 1014, and a communication component 1016.

[0114] Processing component 1002 typically controls the overall operation of electronic device 1000, such as operations associated with display, telephone calls, data communication, camera operation, and recording operations. Processing component 1002 may include one or more processors 1020 to execute instructions to perform all or part of the steps of the methods described above. Furthermore, processing component 1002 may include one or more modules to facilitate interaction between processing component 1002 and other components. For example, processing component 1002 may include a multimedia module to facilitate interaction between multimedia component 1008 and processing component 1002.

[0115] Memory 1004 is configured to store various types of data to support the operation of electronic device 1000. Examples of this data include instructions for any application or method operating on electronic device 1000, contact data, phonebook data, messages, pictures, videos, etc. Memory 1004 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as Static Random Access Memory (SRAM), Electrically Erasable Programmable Read-Only Memory (EEPROM), Erasable Programmable Read-Only Memory (EPROM), Programmable Read-Only Memory (PROM), Read-Only Memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk.

[0116] Power component 1006 provides power to various components of electronic device 1000. Power component 1006 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to electronic device 1000.

[0117] Multimedia component 1008 includes a screen that provides an output interface between the electronic device 1000 and the user. In some embodiments, the screen may include a Liquid Crystal Display (LCD) and a Touch Panel (TP). If the screen includes a Touch Panel, the screen may be implemented as a touchscreen to receive input signals from the user. The Touch Panel includes one or more touch sensors to sense touches, swipes, and gestures on the Touch Panel. The touch sensors may sense not only the boundaries of the touch or swipe action but also the duration and pressure associated with the touch or swipe operation. In some embodiments, multimedia component 1008 includes a front-facing camera and / or a rear-facing camera. When the electronic device 1000 is in an operating mode, such as a shooting mode or a video mode, the front-facing camera and / or the rear-facing camera may receive external multimedia data. Each front-facing camera and rear-facing camera may be a fixed optical lens system or have focal length and optical zoom capabilities.

[0118] Audio component 1010 is configured to output and / or input audio signals. For example, audio component 1010 includes a microphone (MIC) configured to receive external audio signals when electronic device 1000 is in an operating mode, such as call mode, recording mode, and voice recognition mode. The received audio signals may be further stored in memory 1004 or transmitted via communication component 1016. In some embodiments, audio component 1010 also includes a speaker for outputting audio signals.

[0119] I / O interface 1012 provides an interface between processing component 1002 and peripheral interface modules, which may be keyboards, click wheels, buttons, etc. These buttons may include, but are not limited to, home buttons, volume buttons, start buttons, and lock buttons.

[0120] Sensor assembly 1014 includes one or more sensors for providing state assessments of various aspects of electronic device 1000. For example, sensor assembly 1014 may detect the on / off state of electronic device 1000, the relative positioning of components such as the display and keypad of electronic device 1000, changes in position of electronic device 1000 or a component of electronic device 1000, the presence or absence of user contact with electronic device 1000, the orientation or acceleration / deceleration of electronic device 1000, and temperature changes of electronic device 1000. Sensor assembly 1014 may include a proximity sensor configured to detect the presence of nearby objects without any physical contact. Sensor assembly 1014 may also include an optical sensor, such as a complementary metal-oxide-semiconductor (CMOS) or charge-coupled device (CCD) image sensor, for use in imaging applications. In some embodiments, sensor assembly 1014 may also include an accelerometer, gyroscope, magnetometer, pressure sensor, or temperature sensor.

[0121] Communication component 1016 is configured to facilitate wired or wireless communication between electronic device 1000 and other devices. Electronic device 1000 can access wireless networks based on communication standards, such as WiFi, 4G, or 5G, or combinations thereof. In one exemplary embodiment, communication component 1016 receives broadcast signals or broadcast-related information from an external broadcast management system via a broadcast channel. In one exemplary embodiment, communication component 1016 further includes a Near Field Communication (NFC) module to facilitate short-range communication. For example, the NFC module may be implemented based on Radio Frequency Identification (RFID), Infrared Data Association (IrDA), Ultra-Wideband (UWB), Bluetooth, and other technologies.

[0122] In an exemplary embodiment, the electronic device 1000 may be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field-programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic components to perform the methods described above.

[0123] In an exemplary embodiment, a non-transitory computer-readable storage medium including instructions is also provided, such as a memory 1004 including instructions, which can be executed by a processor 1020 of an electronic device 1000 to perform the above-described method. For example, the non-transitory computer-readable storage medium may be a read-only memory (ROM), a random access memory (RAM), a compact disc read-only memory (CD-ROM), magnetic tape, floppy disk, and optical data storage device, etc.

[0124] To implement the above embodiments, this application also proposes a chip, wherein the chip includes an interface circuit and a processing circuit coupled to each other. The interface circuit is used to input or output signals, and the processing circuit is configured to perform the content processing method as provided in any of the foregoing embodiments.

[0125] Figure 11 This is a schematic diagram of the structure of a chip proposed in an exemplary embodiment of this application. See also... Figure 11 The diagram shown is a schematic representation of the structure of chip 1100, but it is not limited to this.

[0126] Chip 1100 includes processing circuit 1101, which is configured to perform any of the above processing methods.

[0127] In some embodiments, chip 1100 further includes one or more interface circuits 1102. Optionally, the interface circuit 1102 is connected to memory 1103, and the interface circuit 1102 can be used to receive signals from memory 1103 or other devices, and the interface circuit 1102 can be used to send signals to memory 1103 or other devices. For example, the interface circuit 1102 can read instructions stored in memory 1103 and send the instructions to processing circuit 1101.

[0128] In some embodiments, the interface circuit 1102 performs at least one of the communication steps such as sending and / or receiving in the above method, while the processing circuit 1101 performs other steps.

[0129] In some embodiments, the terms interface circuit, interface, transceiver pin, transceiver, etc., can be used interchangeably.

[0130] In some embodiments, chip 1100 further includes one or more memories 1103 for storing instructions. Optionally, all or part of the memories 1103 may be located outside of chip 1100.

[0131] To implement the above embodiments, this application also proposes a non-transitory computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the content processing method as described in any of the foregoing method embodiments.

[0132] To implement the above embodiments, this application also proposes a computer program product having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the content processing method as described in any of the foregoing method embodiments.

[0133] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., refer to specific features, structures, materials, or characteristics described in connection with that embodiment or example, which are included in at least one embodiment or example of this application. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of different embodiments or examples.

[0134] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. In the description of this application, "multiple" means at least two, such as two, three, etc., unless otherwise explicitly specified.

[0135] Any process or method description in the flowchart or otherwise herein can be understood as representing a module, segment, or portion of code comprising one or more executable instructions for implementing custom logic functions or processes, and the scope of the preferred embodiments of this application includes additional implementations in which functions may be performed not in the order shown or discussed, including substantially simultaneously or in reverse order depending on the functions involved, as should be understood by those skilled in the art to which embodiments of this application pertain.

[0136] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as a sequenced list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus, or device (such as a computer-based system, a processor-including system, or other system that can fetch and execute instructions from, an instruction execution system, apparatus, or device). For the purposes of this specification, "computer-readable medium" can be any means that can contain, store, communicate, propagate, or transmit programs for use by, or in conjunction with, an instruction execution system, apparatus, or device. More specific examples (a non-exhaustive list) of computer-readable media include: an electrical connection having one or more wires (electronic device), a portable computer disk drive (magnetic device), random access memory (RAM), read-only memory (ROM), erasable and editable read-only memory (EPROM or flash memory), fiber optic devices, and compact disc read-only memory (CDROM). Alternatively, the computer-readable medium may be paper or other suitable media on which the program can be printed, since the program can be obtained electronically, for example, by optically scanning the paper or other medium, followed by editing, interpreting, or otherwise processing as necessary, and then stored in a computer memory.

[0137] It should be understood that various parts of this application can be implemented using hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented using software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, it can be implemented using any one or a combination of the following techniques known in the art: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (PGAs), FPGAs, etc.

[0138] Those skilled in the art will understand that all or part of the steps of the methods in the above embodiments can be implemented by a program instructing related hardware. The program can be stored in a computer-readable storage medium, and when executed, the program includes one or a combination of the steps of the method embodiments.

[0139] Furthermore, the functional units in the various embodiments of this application can be integrated into a processing module, or each unit can exist physically separately, or two or more units can be integrated into a module. The integrated module can be implemented in hardware or as a software functional module. If the integrated module is implemented as a software functional module and sold or used as an independent product, it can also be stored in a computer-readable storage medium.

[0140] The storage medium mentioned above can be a read-only memory, a disk, or an optical disk, etc. Although embodiments of this application have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting this application. Those skilled in the art can make changes, modifications, substitutions, and variations to the above embodiments within the scope of this application.

Claims

1. A content processing method, characterized in that, The convolutional network in the content processing model includes at least one convolutional layer for sequential processing, each of the convolutional layers including multiple convolutional kernels, and the method includes: The content processing model extracts features from the input feature map using at least one convolutional layer to obtain an output feature map. In any convolutional layer, multiple convolutional kernels employ corresponding convolutional algorithms to perform convolution operations in parallel on the input feature map or an intermediate feature map obtained from the feature extraction of the previous convolutional layer, and at least two convolutional kernels use different convolutional algorithms. The input feature map is determined based on first input data, which includes text or an image. The output feature map is predicted by the prediction network in the content processing model to obtain the prediction result.

2. The method according to claim 1, characterized in that, The content processing model is matched with the user's content processing needs, which are determined using any of the following methods: In response to a triggering operation on a target interface element in a graphical user interface (GUI), determine the content processing requirements associated with the target interface element. In response to input operations to interactive controls in the GUI, the system obtains the input task description information and performs intent recognition on the task description information to determine the content processing requirements.

3. The method according to claim 2, characterized in that, The interactive controls include at least one of the following: voice input control, text input control, dialog window, text field, rich text editor, drop-down list.

4. The method according to claim 1, characterized in that, Each convolutional layer includes at least one first convolutional kernel rearranged in sequence and at least one second convolutional kernel; each convolutional layer is used to perform the following steps: The first convolutional kernel employs a first convolutional algorithm to perform a first convolution operation on the feature map input to any convolutional layer, thereby obtaining a first output feature; wherein, the feature map input to any convolutional layer includes the input feature map or an intermediate feature map obtained from feature extraction of the previous convolutional layer; The second convolutional algorithm is used to perform a second convolution operation on the feature map input to any convolutional layer using the second convolutional kernel to obtain a second output feature; wherein, the computational accuracy of the first convolutional algorithm is higher than that of the second convolutional algorithm, and the computational performance of the first convolutional algorithm is lower than that of the second convolutional algorithm. Based on the rearrangement order of multiple convolutional kernels in any convolutional layer, the first output feature and the second output feature are concatenated to obtain an intermediate feature map output by any convolutional layer.

5. The method according to claim 4, characterized in that, The first convolutional kernel and the second convolutional kernel are determined and rearranged using the following steps: Obtain the first quantization error of multiple convolution kernels in any convolutional layer using the first convolution algorithm, and the second quantization error of multiple convolution kernels in any convolutional layer using the second convolution algorithm; Based on the first quantization error and the second quantization error corresponding to multiple convolutional kernels in any convolutional layer, a first convolutional kernel and a second convolutional kernel are determined from multiple convolutional kernels in any convolutional layer; wherein, the first quantization error corresponding to the first convolutional kernel is less than or equal to the second quantization error, and the first quantization error corresponding to the second convolutional kernel is greater than the second quantization error; Based on the indices of the output channels corresponding to the first and second convolutional kernels, the order of the first and second convolutional kernels is rearranged.

6. The method according to claim 5, characterized in that, The step of obtaining the first quantization error of the multiple convolution kernels in any convolutional layer using the first convolution algorithm, and the second quantization error of the multiple convolution kernels in any convolutional layer using the second convolution algorithm, includes: For any convolution kernel in any convolutional layer, the model weights of the content processing model are quantized using the first convolution algorithm through the convolution kernel to obtain the first quantized weights, and the first quantization error is determined based on the difference between the model weights and the first quantized weights. The model weights are quantized using the second convolution algorithm with any one of the convolution kernels to obtain second quantized weights, and a second quantization error is determined based on the difference between the model weights and the second quantized weights.

7. The method according to claim 5, characterized in that, The step of obtaining the first quantization error of the multiple convolution kernels in any convolutional layer using the first convolution algorithm, and the second quantization error of the multiple convolution kernels in any convolutional layer using the second convolution algorithm, includes: The content processing model is used to perform forward propagation calculation on the second input data to obtain a first feature map of multiple convolutional kernels input to any one of the convolutional layers; wherein the data modality of the second input data matches the data modality of the first input data; For any convolution kernel in any convolutional layer, the first convolution algorithm is used to quantize the first feature map through the first convolution kernel to obtain the second feature map, and a first quantization error between the first feature map and the second feature map is determined. The first feature map is quantized using the second convolution algorithm with any one of the convolution kernels to obtain a third feature map, and a second quantization error between the first feature map and the third feature map is determined.

8. The method according to any one of claims 4-7, characterized in that, In response to any of the convolutional layers being a non-first convolutional layer in the at least one convolutional layer, Before performing the first convolution operation on the input feature map using the first convolution kernel and the first convolution algorithm to obtain the first output feature, the method further includes: The indices of the input channels of the multiple convolutional kernels in the non-first convolutional layer are updated based on the rearranged order of the multiple convolutional kernels in the previous convolutional layer.

9. The method according to claim 8, characterized in that, The method further includes: Based on the rearrangement order of the multiple convolutional kernels of the previous convolutional layer, the indices of the bias terms corresponding to the multiple convolutional kernels of the non-first convolutional layer are adjusted respectively. The bias term is used to adjust the output features of the corresponding convolution kernel.

10. The method according to any one of claims 4-7, characterized in that, In response to any convolutional layer being the last convolutional layer in the at least one convolutional layer, the index of the feature map output by the output channel of each convolutional kernel matches the index of the output channel of the convolutional kernel; After concatenating the first output feature and the second output feature based on the rearrangement order of multiple convolutional kernels in any convolutional layer to obtain an intermediate feature map output by any convolutional layer, the method further includes: Based on the original indices of multiple convolutional kernels in the first convolutional layer of the at least one convolutional layer, the positions of the feature maps of multiple output channels in the intermediate feature map output by the last convolutional layer are adjusted to obtain the output feature map.

11. A content processing apparatus, characterized in that, The convolutional network in the content processing model includes at least one convolutional layer that processes sequentially, each of the convolutional layers including multiple convolutional kernels, and the apparatus includes: The feature extraction module is used to extract features from the input feature map through at least one convolutional layer in the content processing model to obtain an output feature map; wherein, multiple convolutional kernels in any convolutional layer perform convolution operations in parallel on the input feature map or the intermediate feature map obtained by feature extraction from the previous convolutional layer using corresponding convolutional algorithms, and at least two convolutional kernels correspond to different convolutional algorithms; the input feature map is determined based on first input data; the first input data includes text or images; The prediction module is used to perform prediction processing on the output feature map through the prediction network in the content processing model to obtain the prediction result.

12. The apparatus according to claim 11, characterized in that, Each convolutional layer includes at least one first convolutional kernel rearranged in sequence and at least one second convolutional kernel; each convolutional layer is used to perform the following operations: The first convolutional kernel employs a first convolutional algorithm to perform a first convolution operation on the feature map input to any convolutional layer, thereby obtaining a first output feature; wherein, the feature map input to any convolutional layer includes the input feature map or an intermediate feature map obtained from feature extraction of the previous convolutional layer; The second convolutional algorithm is used to perform a second convolution operation on the feature map input to any convolutional layer using the second convolutional kernel to obtain a second output feature; wherein, the computational accuracy of the first convolutional algorithm is higher than that of the second convolutional algorithm, and the computational performance of the first convolutional algorithm is lower than that of the second convolutional algorithm. Based on the rearrangement order of multiple convolutional kernels in any convolutional layer, the first output feature and the second output feature are concatenated to obtain an intermediate feature map output by any convolutional layer.

13. The apparatus according to claim 12, characterized in that, The first convolutional kernel and the second convolutional kernel are determined and rearranged using the following module: The acquisition module is used to acquire the first quantization error of multiple convolution kernels in any convolutional layer using the first convolution algorithm, and the second quantization error of multiple convolution kernels in any convolutional layer using the second convolution algorithm; The determining module is configured to determine a first convolutional kernel and a second convolutional kernel from multiple convolutional kernels in any convolutional layer based on a first quantization error and a second quantization error corresponding to multiple convolutional kernels in any convolutional layer; wherein the first quantization error corresponding to the first convolutional kernel is less than or equal to the second quantization error, and the first quantization error corresponding to the second convolutional kernel is greater than the second quantization error; The rearrangement module is used to rearrange the first convolutional kernel and the second convolutional kernel in order based on the indices of the output channels corresponding to the first convolutional kernel and the second convolutional kernel, respectively.

14. An electronic device, characterized in that, It includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, it implements the steps of the method as described in any one of claims 1 to 10.

15. A non-transitory computer-readable storage medium having computer program instructions stored thereon, characterized in that, When executed by a processor, the program instructions implement the steps of the method according to any one of claims 1 to 10.

16. A chip, characterized in that, The chip includes an interface circuit and a processing circuit that are coupled to each other. The interface circuit is used to input or output signals, and the processing circuit is used to implement the method of any one of claims 1 to 10.

17. A computer program product, characterized in that, It includes a computer program, which, when executed by a processor, implements the steps of the method according to any one of claims 1 to 10.

Images