A lightweight neural network model generation method for short play processing
By constructing a tag dataset and optimizing a lightweight neural network model through neural architecture search, the efficiency and effectiveness issues of real-time processing of short dramas on mobile devices were solved, achieving low-latency and efficient emotional recognition and stylization processing of short dramas.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- BEIJING MUST READ TECHNOLOGY CO LTD
- Filing Date
- 2026-03-18
- Publication Date
- 2026-06-12
Smart Images
Figure CN122197989A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of artificial intelligence technology, and in particular to a lightweight neural network model generation method for short drama processing. Background Technology
[0002] In recent years, short video content, represented by short dramas, has rapidly gained popularity on mobile internet platforms. These videos are typically short in length, have tight plots, strong emotional expression, and unique visual styles, making them very popular with users. Users on mobile devices (such as smartphones and tablets) not only consume this type of content but also have a need for real-time processing and creation, such as recognizing emotions in videos in real time during filming and matching them with corresponding stylized effects.
[0003] While various neural network models exist for emotion recognition and image style transfer, directly applying them to real-time processing scenarios for short dramas on mobile devices faces numerous challenges. On one hand, traditional deep learning models are typically complex and have a large number of parameters. When running on mobile devices with limited computing resources and power consumption, they exhibit high inference latency, failing to meet real-time processing requirements and rapidly consuming device battery power, severely impacting user experience. On the other hand, general-purpose models are not optimized for the unique emotional expression patterns (such as "domineering CEO's cold smile" or "resisting adversity") and visual styles (such as "Hong Kong retro style" or "cyberpunk style") of short dramas, resulting in poor processing performance. Furthermore, manually designing and optimizing a lightweight, dedicated model that balances performance and effectiveness for different mobile hardware models is a time-consuming, labor-intensive task requiring a high level of technical expertise.
[0004] Therefore, there is a lack of existing technologies that can automatically generate neural network models. These models need to be lightweight (low latency, low power consumption) to adapt to mobile deployments, and their structure should be specifically optimized for short drama content to achieve high-precision real-time emotion recognition and stylization processing. Summary of the Invention
[0005] The main objective of this invention is to provide a lightweight neural network model generation method, apparatus, electronic device, and computer-readable storage medium for short drama processing, aiming to solve the problem in the prior art of automatically generating efficient neural network models that take into account both mobile hardware performance and the specificity of short drama processing tasks.
[0006] To achieve the above objectives, this invention provides a lightweight neural network model generation method for short drama processing, comprising: constructing a training dataset containing short drama video data and its corresponding labels, wherein the labels include sentiment categories for content understanding and visual style categories for visual effects processing; constructing a search space for neural architecture search, wherein the search space defines a set of domain-specific operations directly corresponding to the labels, the set including: a temporal expression analysis operation for identifying the sentiment category from the video data, and a visual style transfer operation for applying the visual style category to the video data; searching in the search space based on a differentiable search algorithm to obtain an optimal architecture configuration, wherein the optimal architecture configuration is determined by co-optimizing the recognition accuracy of the sentiment category and visual style category on the validation set of the training dataset and the model inference latency estimated based on a pre-built hardware latency lookup table; and generating a final lightweight neural network model based on the optimal architecture configuration obtained by the search, for performing real-time sentiment recognition and stylized effects processing on short drama video content.
[0007] Optionally, the temporal expression analysis operation is configured as a searchable component in the search space, and its searchable parameters include: the insertion depth of the operation in the network, the kernel size used to calculate the inter-frame attention weights, and the number of attention heads.
[0008] Optionally, the visual style transfer operation includes: extracting style feature vectors of the style image through a style encoding network, generating affine transformation parameters based on the style feature vectors, and adjusting the amplitude of the affine transformation parameters through a globally learnable style intensity scalar λ.
[0009] Furthermore, in the model deployment phase: a mobile application integrating the lightweight neural network model is published to an application distribution platform; the application provides a graphical user interface to the end user, which includes a slider control for real-time control of the stylization level; the real-time position parameters of the slider control are mapped to a numerical value of the style intensity scalar λ through the application logic layer; during inference calculation, the lightweight neural network model calls the assigned style intensity scalar λ, thereby achieving dynamic and continuous adjustment of the stylization level of the output image without reloading the model.
[0010] Optionally, the construction process of the pre-built hardware latency lookup table includes: on each model of chip in the target mobile device chip pool, in an isolated computing environment, measuring the inference latency of each basic operator and domain-specific operation in the search space under different input tensor shapes, and establishing a multi-dimensional mapping relationship database containing hardware identifiers, operator types, input shapes and measured latency values.
[0011] Optionally, the objectives of the collaborative optimization also include power consumption estimated based on theoretical computational FLOPs and memory accesses (DRAMAccesses), and the recognition accuracy, inference latency, and power consumption are fused into a single multi-objective loss function through a weighted formula to jointly guide the search direction of the optimal architecture configuration.
[0012] Optionally, the search space also includes a set of basic operators optimized for mobile devices, including at least: depthwise separable convolution, inverse residual module MBConv, and lightweight channel attention mechanism ECA-Net.
[0013] Optionally, the lightweight neural network model generated by the method is an image style transfer model, which has an encoder-bottleneck layer-decoder architecture; wherein the encoder and decoder are composed of a combination of searched basic operators, and the bottleneck layer integrates the structure that implements the visual style transfer operation.
[0014] Furthermore, the decoder uses pixel shuffling for upsampling, and the subsequent convolutional modules are lightweight convolutional modules determined through search. This combination of structures is obtained through the neural architecture search process for collaborative optimization of the short drama style transfer task.
[0015] Optionally, after the model generation step, a model quantization step is also included: perceptual quantization training is performed on the generated final model, wherein an asymmetric quantization scheme based on the characteristics of mobile terminal chips is adopted for the output activation values generated by the structure that implements the temporal expression analysis operation, so as to reduce the quantization error caused by the large dynamic range of activation values in the short drama video data.
[0016] Compared with existing technologies, this invention has the following advantages: By constructing a hardware latency lookup table and taking the estimated inference latency of the model on the target hardware as one of the core optimization objectives during the search process, it ensures that the final generated model is "tailor-made" for the performance of mobile devices. This enables the model to achieve low-latency real-time processing during actual deployment, while reducing power consumption and improving user experience. The use of neural architecture search technology automates the manual model design process, which previously required significant manpower and time. Through algorithm-driven collaborative optimization, it can efficiently find a balance between accuracy and hardware efficiency within a vast architectural space, greatly shortening the development cycle of high-performance dedicated models and lowering the technical threshold. Attached Figure Description
[0017] To more clearly illustrate the technical solutions of the embodiments of this application, the drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0018] Figure 1 This is a data preparation flowchart according to an embodiment of the present invention.
[0019] Figure 2 This is a search space structure diagram according to an embodiment of the present invention.
[0020] Figure 3 This is a block diagram of a hardware-aware search system according to an embodiment of the present invention.
[0021] Figure 4 This is a schematic diagram of a lightweight neural network model architecture according to an embodiment of the present invention.
[0022] Figure 5 This is a schematic diagram of a mobile application GUI interface according to an embodiment of the present invention.
[0023] Figure 6 This is a schematic diagram of the structure of an electronic device according to an embodiment of the present invention. Detailed Implementation
[0024] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.
[0025] The terminology used in the embodiments of this invention is for the purpose of describing particular embodiments only and is not intended to limit the invention. The singular forms “a,” “the,” and “the” used in the embodiments of this invention and the appended claims are also intended to include the plural forms, unless the context clearly indicates otherwise. “Multiple” generally includes at least two, but does not exclude the inclusion of at least one.
[0026] It should be understood that the term "and / or" used in this article is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, and B existing alone. Additionally, the character " / " in this article generally indicates that the preceding and following related objects have an "or" relationship.
[0027] Depending on the context, the words “if” or “suppose” as used here can be interpreted as “when” or “in response to determination” or “in response to detection.” Similarly, depending on the context, the phrases “if determination” or “if detection (of the stated condition or event)” can be interpreted as “when determination” or “in response to determination” or “when detection (of the stated condition or event)” or “in response to detection (of the stated condition or event).”
[0028] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a product or system comprising a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a product or system. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the product or system that includes said element.
[0029] Furthermore, the timing of the steps in the following method embodiments is merely an example and not a strict limitation.
[0030] The lightweight neural network model generation method for short drama processing provided in this invention can be executed by an electronic device, such as a PC, laptop, or smartphone, or a server. The server can be a physical server containing an independent host, a virtual server, a cloud server, or a server cluster.
[0031] Before providing a further detailed description of the embodiments of this application, some of the nouns and terms involved in the embodiments of this application will be explained. The nouns and terms involved in the embodiments of this application are subject to the following interpretations.
[0032] (1) Short dramas: These are video content in a vertical field that typically lasts 1-10 minutes, with a tight plot and strong emotional expression. They have significant characteristics that distinguish them from traditional film and television content in terms of facial expressions, scene transitions, and visual style, such as more exaggerated emotional expression, faster pace, and more obvious stylization.
[0033] (2) Neural Architecture Search (NAS): refers to a technology for automatically designing neural network structures. It aims to automatically find the optimal or suboptimal network architecture in a predefined search space containing various computational operations and connection methods, based on specific performance indicators (such as accuracy, inference latency, power consumption, etc.) and through specific search strategies (such as reinforcement learning, evolutionary algorithms or gradient-based algorithms).
[0034] (3) Domain-Specific Operation: refers to a non-general computational operation or module specifically designed to solve a problem in a specific domain. In the context of this application, it specifically refers to the "temporal expression analysis operation" and "visual style transfer operation" designed for processing short drama videos. These operations can capture the core features of short drama content more effectively than general operators.
[0035] (4) Hardware Latency Look-Up Table: This refers to a pre-built database or multidimensional mapping table that stores the precise computation time required to execute each basic operator or domain-specific operation in the search space on a specific hardware platform (such as a specific mobile chip) under different input sizes (such as different image resolutions and number of channels). It enables the rapid and accurate estimation of the total inference latency of any candidate model during the architecture search process without the need for time-consuming deployment tests on real hardware.
[0036] (5) Style intensity scalar λ: This refers to a global, learnable, or user-controllable numerical parameter. In visual style transfer operations, this scalar is used to uniformly regulate the magnitude of style transformation applied to content features. By adjusting the value of λ, the degree of stylization in the final output image can be controlled, achieving a smooth transition from no style to full stylization.
[0037] (6) Pixel Shuffle: This is an efficient upsampling technique commonly used in the decoder portion of image generation and super-resolution tasks. It improves resolution by rearranging a low-resolution feature map with C×r² channels into a high-resolution feature map with C channels and a spatial size increased by r times. Compared to upsampling methods such as deconvolution, pixel shuffle requires less computation and effectively avoids checkerboard artifacts.
[0038] In one embodiment of the present invention, a lightweight neural network model generation method for short drama processing is provided. This method aims to address the challenge that existing general-purpose models, when processing specific video content such as short dramas, cannot simultaneously achieve depth of emotional understanding, visual style matching, and real-time operating efficiency on mobile devices. Through a hardware-aware neural architecture search process, a highly efficient neural network model deeply customized for short drama scenarios is automatically generated.
[0039] Reference Figure 1This method first requires constructing a training dataset specifically for model training. The core of this step lies in the targeted nature of the data sources and the professionalism of the labeling system. Specifically, a large amount of representative short drama video data is collected, such as video clips from legitimate channels of short drama platforms, covering various themes (e.g., urban, historical, campus). Subsequently, this raw data is cleaned and preprocessed, removing invalid or low-quality clips and standardizing resolution and frame rate. The key is to assign these video data dual labels directly related to the short drama processing task: one is an emotion category label for content understanding, and the other is a visual style category label for visual effects processing. The emotion category label reflects the exaggerated emotional expressions of the short drama characters, while the visual style category label corresponds to specific visual aesthetics common in the short drama content.
[0040] Next, a search space for Neural Architecture Search (NAS) needs to be constructed, which serves as a "blueprint" for model innovation. For example... Figure 2 As shown, this search space not only includes a series of basic computational operations, but more importantly, it embeds a set of domain-specific operations directly corresponding to the aforementioned labeling system. This set is the key difference between this method and general NAS, as it encodes the domain knowledge of short drama processing into the design source of the model architecture. Specifically, this set includes a temporal expression analysis operation for identifying emotion categories from video data, and a visual style transfer operation for applying visual style categories to video data. By including these highly customized operations as options in the search space, the final generated model inherently possesses the ability to handle the core tasks of short drama processing.
[0041] After preparing the dataset and search space, the method enters the core search phase. For example... Figure 3 As shown, this stage is based on a differentiable search algorithm that automatically searches within the constructed search space to obtain an optimal architecture configuration. Here, "optimal" is not a single-dimensional concept, but rather a result of multi-objective collaborative optimization. Specifically, the optimization objectives include at least two aspects: first, the model's combined recognition accuracy for sentiment and visual style categories on the validation set of the training dataset; and second, the inference latency of the model on the target mobile device estimated through a pre-built hardware latency lookup table. This collaborative optimization ensures that the search process not only pursues high model performance but also strictly constrains its operational efficiency to meet the real-time processing requirements of mobile devices.
[0042] Finally, based on the optimal architecture configuration obtained from the search, a lightweight neural network model is generated. This process typically involves discretizing the continuous, probabilistic architecture representation obtained in the search phase into a deterministic network structure, and then training this deterministic network to converge the weights. The resulting model is a lightweight, efficient, and highly targeted neural network that can be directly deployed in mobile applications to perform real-time sentiment recognition and stylized effects processing on input short video content, thereby achieving a deep integration of content understanding and visual enhancement.
[0043] In a preferred embodiment, to improve the efficiency of the generated model on mobile devices, the search space also includes a set of basic operators optimized for mobile hardware (such as ARM-based CPUs or mobile GPUs). These operators are designed with a balance between computational cost and parameter count. For example, the basic operators include at least: depthwise separable convolution, inverse residual module (MBConv), and lightweight channel attention mechanism (ECA-Net). Depthwise separable convolution reduces computation by several orders of magnitude by decomposing standard convolution into depthwise convolution and pointwise convolution, while maintaining the receptive field. Inverse residual module effectively improves the nonlinear expressiveness of the network with limited computational cost through an "expansion-depth convolution-compression" structure. Lightweight channel attention mechanism achieves inter-channel information interaction through efficient one-dimensional convolution, enabling the model to adaptively focus on more important feature channels. Using these efficient operators as basic building blocks ensures that the searched architecture has a lightweight foundation, laying a solid foundation for achieving low-latency inference.
[0044] Furthermore, the construction process of the pre-built hardware latency lookup table is crucial for achieving hardware-aware search. In a specific embodiment, the construction process of this lookup table includes the following steps: First, a target mobile device chip pool is identified, covering mainstream high-end, mid-range, and low-end chip models on the market. Then, in a strictly isolated computing environment (e.g., using specialized benchmarking tools with all unnecessary background processes closed), the actual inference latency of each basic operator and domain-specific operation defined in the search space is measured under different input tensor shapes (such as different resolutions and number of channels). This measurement process is exhaustive and systematic, ensuring the comprehensiveness of the data. Finally, all measurement results—including hardware identifiers, operator types, input shapes, and measured latency values—are established into a multi-dimensional mapping database. This database can be used to quickly query hardware performance data in subsequent search processes. The latency of any candidate architecture can be estimated by summing the results from the lookup table, thereby avoiding time-consuming real hardware deployment tests and greatly improving the efficiency of hardware-aware search.
[0045] In another preferred embodiment, to enable the model to more deeply understand the emotional dynamics within the short drama, the temporal expression analysis operation is configured as a highly flexible searchable component in the search space. Its searchable parameters include, but are not limited to: the insertion depth of the operation throughout the network (determining at which level of features it performs temporal analysis), the kernel size used to calculate the attention weights between video frames (determining the receptive field size in the temporal dimension), and the number of attention heads (determining how many different temporal dependency patterns the model can simultaneously attend to). By making these key hyperparameters searchable, the neural architecture search algorithm can automatically discover the module configuration best suited for capturing the unique emotional rhythms of the short drama (such as the instantaneous burst of a "domineering CEO's cold laugh" or the sustained state of "resisting a counterattack"), thereby improving the accuracy of emotion recognition.
[0046] In one specific embodiment, the internal implementation of the temporal expression analysis operation can employ a multi-head self-attention architecture. To reduce the high cost of performing global attention calculations on video sequences, a temporal-dimensional local attention window can be introduced. Its attention calculation can follow the scaling dot product attention formula:
[0047] Where Q, K, and V represent the query, key, and value matrices, respectively, which are obtained by different linear transformations of the input video frame feature sequence; is the dimension of the key vector, used as a scaling factor to prevent the dot product result from being too large when the dimension is too high, which would cause the gradient of the softmax function to vanish; the softmax function converts the calculated attention score into a probability distribution, which is used to perform a weighted summation of the value (V), thereby obtaining the output feature that incorporates temporal context information.
[0048] Furthermore, to achieve rich and controllable visual stylization effects, the visual style transfer operation is implemented as follows. In a preferred embodiment, the operation includes: First, extracting a style feature vector of the target style image through a style encoding network (e.g., a lightweight variant of VGG or MobileNet). Then, based on this feature vector condensed with style information, generating a set of affine transformation parameters (typically a scaling factor γ and a bias factor β) through a small parameter generator (e.g., a few fully connected layers). Unlike traditional instance normalization, this embodiment introduces a globally learnable style intensity scalar λ, which is used to adjust the magnitude of the affine transformation parameters; for example, the new parameters become γ' = λ·γ and β' = λ·β. Finally, the adjusted parameters are applied to the feature map of the content video, and style injection is completed through an adaptive instance normalization (AdaIN) operation. The technical advantage of this design is that the intensity of stylization can be easily adjusted by learning or controlling a single λ value, laying the foundation for subsequent real-time interactive control.
[0049] Reference Figure 4 In a more specific embodiment, the lightweight neural network model generated by the method of the present invention can be primarily an encoder-decoder architecture for image style transfer. Specifically, the model includes an encoder 402, a bottleneck layer 406, and a decoder 409. The encoder 402 is responsible for receiving input video frames and compressing them into deep feature representations through a series of downsampling and feature extraction modules (e.g., a combination of MBConv modules determined by search). The bottleneck layer 406 is the core of style and content fusion, integrating the aforementioned visual style transfer operation structure, such as an AdaIN module 407. This module receives the content features output by the encoder and the style feature vector 408 from the external input and performs a stylized affine transformation. The decoder 409 is responsible for progressively upsampling the stylized feature map and reconstructing it into a high-quality stylized video frame with the same resolution as the input. The advantage of this architecture is that the specific structures of the encoder and decoder (such as module type, number of layers, number of channels, etc.) are automatically optimized and determined by the neural architecture search process, while the core mechanism of style transfer is explicitly placed in the bottleneck layer, resulting in a clear and easily controllable structure.
[0050] Building upon the aforementioned encoder-decoder architecture, to further improve decoding efficiency and output image quality, the decoder 409 can employ a pixel shuffling operation 410 for upsampling. Pixel shuffling achieves resolution improvement through tensor rearrangement operations. Compared to traditional deconvolution, it requires less computation and effectively avoids checkerboard artifacts, making it suitable for lightweight deployment on mobile devices. Furthermore, the type and parameters (such as kernel size and number of channels) of the convolutional module 411 connected after the pixel shuffling operation 410 are also optimized collaboratively through a neural architecture search process to meet the specific needs of short drama style transfer tasks (such as texture detail restoration and color fidelity). This combination of "pixel shuffling + search-determined lightweight convolution" is a preferred solution for the decoder structure in this invention, minimizing computational overhead while ensuring high-quality upsampling.
[0051] Reference Figure 5 One important application scenario of this invention is to implement real-time, interactive video stylization in mobile applications. In a preferred embodiment, the application is implemented in the deployment phase as follows: First, the mobile application integrating the lightweight neural network model generated by this invention is published to an application distribution platform. This application provides a graphical user interface (GUI) to the end user, for example, on the screen of a smartphone 501. This interface, in addition to displaying a real-time camera preview 503, also includes a slider control 511 located in the control area 504 for real-time control of the stylization level. Figure 5 As shown, a text label "Style Intensity" (510) can be displayed near the slider control 511 to indicate the function of the control to the user. When the user drags the slider 507 on the screen, the application's logic layer captures the slider's real-time position parameter (e.g., a floating-point number from 0.0 to 1.0). Then, the logic layer maps this position parameter to a value for the style intensity scalar λ in the aforementioned visual style transfer operation in real time and without delay. Finally, the lightweight neural network model performing inference calculations immediately calls this newly assigned λ, thereby achieving dynamic and continuous adjustment of the output image's stylization level without reloading the model or interrupting the video stream. As the user drags the slider, the image style smoothly changes from none to some, from weak to strong, achieving a "what you see is what you get" interactive experience.
[0052] To further improve model performance when deployed on mobile devices, in a preferred implementation, the co-optimization objective, in addition to accuracy and inference latency, may also include power consumption estimated based on theoretical computational power consumption (FLOPs) and memory accesses (DRAM accesses). Power consumption is crucial for mobile devices as it directly impacts battery life and heat generation. In this case, the three interdependent objectives of recognition accuracy, inference latency, and power consumption can be fused into a single multi-objective loss function using a weighted formula, for example: in, It's an accuracy loss. It is a delayed loss. It is power consumption loss, and and It is a trade-off hyperparameter used to balance the importance of the three factors. This unified loss function will collectively guide the optimization direction of the neural architecture search, enabling it to automatically find Pareto optimal solutions in the three dimensions of accuracy, speed, and energy efficiency.
[0053] Furthermore, a model quantization step can be added in the final stage of model generation to further compress the model size and accelerate inference. In a specific embodiment, perceptual quantization training is performed on the generated final floating-point model. In particular, considering the dramatic changes in facial expressions in short drama videos, the output activation values of the network layers implementing temporal expression analysis may have a large dynamic range. If standard symmetric quantization is used, it may introduce large quantization errors, affecting the accuracy of emotion recognition. Therefore, an asymmetric quantization scheme based on the characteristics of mobile chips can be used for the output activation values of this part of the structure. For example, quantization can be performed using the following formula: Where x is the original floating-point activation value, and represents the minimum and maximum activation values of the data set, and b is the number of quantization bits (e.g., 8). This formula maps the minimum and maximum values of the original data precisely to the two endpoints of the quantization range by shifting and scaling the numerical range. This allows for more efficient use of a limited number of bits to represent data with a large dynamic range, reducing quantization errors. This targeted quantization strategy is a key means of ensuring that the model maintains high performance even after high compression.
[0054] This invention also provides a lightweight neural network model generation device for short drama processing. Logically, the device may include: a data preparation module, a search space construction module, a model search module, and a model generation module. The functions of these modules correspond one-to-one with the steps in the aforementioned method embodiments. The data preparation module is used to construct a training dataset containing short drama video data and their corresponding emotion and visual style labels. The search space construction module is used to construct a neural architecture search space containing basic operators and domain-specific operations (temporal expression analysis operation and visual style transfer operation). The model search module is used to obtain the optimal architecture configuration in the search space by co-optimizing accuracy and estimated inference latency based on a differentiable search algorithm. The model generation module then generates a lightweight neural network model for real-time emotion recognition and stylization processing based on this optimal architecture configuration.
[0055] In a preferred embodiment of the device, the temporal expression analysis operation in the model search module is configured as a searchable component. Its searchable parameters include the insertion depth of the operation in the network, the size of the convolutional kernel used to calculate inter-frame attention weights, and the number of attention heads, thereby enabling the device to automatically optimize its ability to capture short-form emotional dynamics. Further, the visual style transfer operation can be implemented through several functional units, including: a style encoding unit for extracting style image feature vectors; a parameter generation unit for generating affine transformation parameters based on the vectors; and an intensity control unit for adjusting the amplitude of the affine transformation parameters using a global style intensity scalar λ. In addition, the device may also include a deployment interaction module, which is responsible for providing a graphical user interface with a slider control in a mobile application and mapping the real-time position parameters of the slider to a numerical assignment of the style intensity scalar λ, enabling the neural network model to dynamically adjust the stylization level of the output image. The device may also include a delay lookup table construction module, used to establish a multi-dimensional mapping relationship database supporting hardware-aware search by measuring the delays of various operators on the target chip.
[0056] Reference Figure 6This invention also provides an electronic device, such as a smartphone, tablet computer, or server for model training. The electronic device may include a processor 22, a memory 21, and a communication interface 23 in its hardware structure, which can communicate with each other via a bus. The memory 21 stores computer programs, for example, including instruction code implementing any of the aforementioned methods. The processor 22, such as a central processing unit (CPU), a graphics processing unit (GPU), or a dedicated artificial intelligence chip (NPU), executes the computer program stored in the memory 21 to implement one or more steps in the aforementioned lightweight neural network model generation method for short drama processing. In a specific application scenario, the electronic device is a mobile terminal, such as a smartphone 501. In this case, the electronic device may also include a display unit (such as an OLED screen) for displaying... Figure 5 The graphical user interface shown includes an input detection unit (such as a touchscreen) for detecting user operations on slider 511 and generating position parameters, and an application logic unit (implemented by application code executed by processor 22) for mapping the position parameters to the value of style intensity scalar λ, thereby realizing complete real-time interactive functionality.
[0057] This invention also provides a computer-readable storage medium, such as a non-volatile USB flash drive, hard disk, or optical disk. The computer-readable storage medium stores a computer program. When the computer program is accessed by an electronic device (as described above)... Figure 6 When the processor 22 of the device shown executes, it is able to implement the method of any one of claims 1 to 10. In a preferred embodiment, when the computer program is executed by the processor, in addition to implementing the model generation method, it can also implement user interaction logic on the deployment end, specifically including: rendering a graphical user interface containing a slider in the mobile application, receiving the intensity adjustment command input by the user through the slider, mapping the command to a value of the style intensity scalar λ in real time, and controlling the lightweight neural network model to perform inference calculation based on the updated λ value, thereby completing the closed-loop control of dynamic style adjustment.
[0058] In a comprehensive embodiment, a scheme integrating multiple preferred technical features will be demonstrated. First, when constructing the search space, not only are temporal expression analysis and visual style transfer operations defined, but a set of efficient basic operators is also integrated, including depthwise separable convolution, the inverse residual module MBConv, and the lightweight channel attention mechanism ECA-Net. Next, before executing the search, a precise hardware latency lookup table is constructed through systematic measurements on multiple target mobile chips. Then, a search process based on a differentiable algorithm is initiated, with the optimization objective being a multi-objective loss function that integrates accuracy, latency estimated from the lookup table, and theoretically calculated power consumption. The search process automatically determines the core architecture of the network as an encoder-bottleneck layer-decoder structure, where the encoder and decoder are composed of a combination of MBConv and ECA-Net determined by the search. The decoder upsampling adopts pixel shuffling technology, and the optimal insertion depth and internal parameters of the temporal expression analysis module are also determined. After obtaining this optimal discrete architecture, it is fully retrained, and perceptual quantization training is introduced in the later stages of training, especially using an asymmetric quantization scheme for the activation values of the temporal analysis module. Finally, this highly optimized, quantified model was packaged into a mobile application. This application provides an interface with a slider, allowing users to control the style intensity scalar λ in real time. The model, on the mobile processor, improves frame rate stability, enabling interactive real-time emotion recognition and stylization processing of short video clips. This comprehensive embodiment, through the synergistic effect of various technical features, enhances the overall performance of the model, achieving the technical goals of high efficiency, accuracy, and low power consumption.
[0059] The technical solution of this invention is not limited to short drama processing. Its core idea—that is, to deeply integrate domain-specific knowledge and operations into model design through hardware-aware neural architecture search to generate lightweight, high-performance customized models—has broad application prospects. For example, this method can be applied to short video platforms to automate and deepen content understanding (such as the recognition of emotions, scenes, and atmosphere) and intelligent tag generation for massive amounts of user-uploaded content, thereby improving the accuracy of recommendation systems. It can also be applied to live streaming applications to develop smarter and more expressive real-time beautification and special effects filters that can dynamically change according to the host's emotions or the live streaming content. In addition, in personal video creation tools (such as Vlog editing software), this method can be used to implement intelligent editing functions, such as automatically matching background music or visual styles that match the emotional tone of video clips, greatly reducing the creative threshold for ordinary users and improving creation efficiency and work quality.
[0060] In addition, embodiments of the present invention provide a non-transitory machine-readable storage medium storing executable code. When the executable code is executed by a processor of an electronic device, the processor can at least implement the lightweight neural network model generation method for short drama processing provided in the foregoing embodiments.
[0061] The device embodiments described above are merely illustrative, and the units described as separate components may or may not be physically separate. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.
[0062] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of a necessary general-purpose hardware platform, or by a combination of hardware and software. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a computer product. The present invention can take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.
[0063] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.
Claims
1. A lightweight neural network model generation method for short drama processing, characterized in that, include: Construct a training dataset containing short drama video data and their corresponding labels, wherein the labels include sentiment categories for content understanding and visual style categories for visual effects processing; Construct a search space for neural architecture search, wherein the search space defines a set of domain-specific operations directly corresponding to the label, the set including: a temporal expression analysis operation for identifying the emotion category from video data, and a visual style transfer operation for applying the visual style category to the video data; Based on a differentiable search algorithm, an optimal architecture configuration is obtained by searching the search space. The optimal architecture configuration is determined by co-optimizing the recognition accuracy of the emotion category and visual style category on the validation set of the training dataset and the model inference latency estimated based on a pre-built hardware latency lookup table. According to the optimal architecture configuration obtained by the search, a final lightweight neural network model is generated for performing real-time emotion recognition and stylized effects processing on short drama video content.
2. The method according to claim 1, characterized in that, The temporal expression analysis operation is configured as a searchable component in the search space, and its searchable parameters include: the insertion depth of the operation in the network, the kernel size used to calculate the inter-frame attention weights, and the number of attention heads.
3. The method according to claim 1, characterized in that, The visual style transfer operation includes: extracting style feature vectors from a style image through a style encoding network, generating affine transformation parameters based on the style feature vectors, and adjusting the amplitude of the affine transformation parameters through a globally learnable style intensity scalar λ.
4. The method according to claim 3, characterized in that, During the model deployment phase: The mobile application integrating the lightweight neural network model is published to an application distribution platform; the application provides a graphical user interface to the end user, which includes a slider control for real-time control of the stylization level. The real-time position parameters of the slider control are mapped to a numerical value of the style intensity scalar λ through the application logic layer. During inference calculations, the lightweight neural network model calls the assigned style intensity scalar λ, thereby enabling dynamic and continuous adjustment of the stylization level of the output image without reloading the model.
5. The method according to claim 1, characterized in that, The construction process of the pre-built hardware latency lookup table includes: on each model of chip in the target mobile device chip pool, in an isolated computing environment, measuring the inference latency of each basic operator and domain-specific operation in the search space under different input tensor shapes, and establishing a multi-dimensional mapping database containing hardware identifiers, operator types, input shapes and measured latency values.
6. The method according to claim 1, characterized in that, The objectives of the collaborative optimization also include power consumption estimated based on theoretical computational FLOPs and DRAM accesses. Furthermore, the recognition accuracy, inference latency, and power consumption are fused into a single multi-objective loss function through a weighted formula to jointly guide the search direction of the optimal architecture configuration.
7. The method according to claim 1, characterized in that, The search space also includes a set of basic operators optimized for mobile devices, including at least: depthwise separable convolution, inverse residual module MBConv, and lightweight channel attention mechanism ECA-Net.
8. The method according to claim 3, characterized in that, The lightweight neural network model generated by the method is an image style transfer model, and its structure is an encoder-bottleneck layer-decoder architecture. The encoder and decoder are composed of a combination of searched basic operators, and the bottleneck layer integrates the structure that implements the visual style transfer operation.
9. The method according to claim 8, characterized in that, The decoder uses pixel shuffling for upsampling, and the convolutional modules connected to it are lightweight convolutional modules determined through search. This combination of structures is obtained through the neural architecture search process for collaborative optimization of the short drama style transfer task.
10. The method according to claim 1, characterized in that, Following the model generation step, a model quantization step is also included: perceptual quantization training is performed on the generated final model, wherein an asymmetric quantization scheme based on the characteristics of mobile chip is adopted for the output activation values generated by the structure that implements the temporal expression analysis operation, in order to reduce the quantization error caused by the large dynamic range of activation values in the short drama video data.