Video-based webpage generation method, system, equipment and medium

Abstract

The invention is applicable to the technical field of webpage front ends, and provides a video-based webpage generation method, which comprises the following steps of: obtaining a video file, and performing format conversion on the video file to obtain a standard video file; constructing a multi-task learning model, inputting the standard video file into the multi-task learning model, and outputting frame indexes of key frames and UI elements in each key frame; extracting feature information in the UI elements, classifying the UI elements according to the feature information, and determining webpage structure information of a webpage to be generated; inputting the webpage structure information into a code conversion model, and outputting a webpage code; and deploying the webpage code to a server to generate an accessible webpage. According to the method, the video content is automatically processed, the webpage code is generated, the time and energy of manual coding are greatly reduced, and the efficiency and convenience of webpage development can be remarkably improved.

Description

Technical Field

[0001] The present application belongs to the field of web front-end technology, and in particular relates to a video-based web page generation method, system, device and medium. Background Art

[0002] With the rapid development of the internet, web pages have become a common means of communication. Web pages contain a wealth of information and require rational organization and design to create a good user experience and attract users. Generally, web page design requires the collaboration of visual designers and R&D engineers. First, the visual designer draws the web design draft, which is then restored by the front-end engineer using HTML pages for easy maintenance and dissemination. However, manually writing and adjusting web page code remains a time-consuming and labor-intensive process, especially for projects that require rapid iteration and frequent updates. Therefore, the web design and development field has gradually developed many advanced tools and technologies that can help developers efficiently create complex web pages, such as technologies that automatically generate web pages based on image recognition.

[0003] However, with the rapid development of artificial intelligence (AI) and the influx of short videos, technologies for generating webpage code from videos are gradually emerging. By automatically generating corresponding webpage code using visual information from videos, these technologies can significantly improve the efficiency and convenience of webpage development. While some initial attempts have been made, such as generating static webpages through video frame extraction and simple element recognition, these technologies still have many shortcomings in terms of accuracy, intelligence, and practicality of the generated code. Summary of the Invention

[0004] The embodiments of the present application provide a video-based web page generation method, system, device, and medium, which can solve one of the above-mentioned problems in the prior art.

[0005] In a first aspect, an embodiment of the present application provides a method for generating a webpage based on a video, comprising:

[0006] Acquire a video file, and convert the format of the video file to obtain a standard video file;

[0007] Constructing a multi-task learning model, inputting the standard video file into the multi-task learning model, and outputting the frame index of the key frame and the UI element in each key frame;

[0008] Extracting characteristic information from UI elements, classifying each UI element according to the characteristic information, and determining webpage structure information of the webpage to be generated;

[0009] Inputting the webpage structure information into a code conversion model and outputting webpage code;

[0010] Deploy the web page code to a server to generate an accessible web page.

[0011] Furthermore, the multi-task learning model includes a keyframe extraction task and a UI element recognition task;

[0012] The constructing of a multi-task learning model, inputting the standard video file into the multi-task learning model, and outputting the frame index of the key frame and the UI element in each key frame includes:

[0013] For the key frame extraction task, a sparsity constraint method is used to determine whether the sparsity of the key frame meets the preset requirements, and an L1 regularization term is introduced to enable the multi-task learning model to learn a sparse key frame representation;

[0014] For the UI element recognition task, a comprehensive constraint method is adopted to identify all UI elements in the keyframe;

[0015] A dynamic weight adjustment strategy is adopted to dynamically adjust the weights of the key frame extraction task and the UI element recognition task according to their difficulty and task completion degree;

[0016] During the multi-task learning model training process, an alternating optimization strategy is adopted to train the key frame extraction task and the UI element recognition task in turn.

[0017] Furthermore, the constructing of a multi-task learning model, inputting the standard video file into the multi-task learning model, and outputting the frame index of the key frame and the UI element in each key frame also includes:

[0018] Constructing an encoder-decoder network structure, the encoder-decoder network structure includes an encoder part and a decoder part, the encoder part uses a convolutional neural network to extract the spatiotemporal features of the standard video file, and the decoder part uses a long short-term memory network to generate a key frame index sequence and a UI element information sequence;

[0019] Adding an attention mechanism module to the last convolutional layer of the encoder part to learn the importance weights of different time steps in the standard video file;

[0020] Introducing a pointer network structure into the decoder part, judging whether each time step is a key frame based on the spatiotemporal features, and if the time step is a key frame, outputting the frame index corresponding to the time step and the UI element corresponding to the key frame;

[0021] The key frame extraction task and the UI element recognition task are dynamically optimized and trained through a preset multi-task loss function, wherein the multi-task loss function includes binary cross entropy loss, multi-category cross entropy loss and regression loss.

[0022] Furthermore, extracting characteristic information from UI elements, classifying each UI element according to the characteristic information, and determining webpage structure information of the webpage to be generated includes:

[0023] Processing the image of the UI element using image processing technology to obtain image features, wherein the image features include edge features, color features, and texture features;

[0024] Processing the text of the UI element using natural language processing technology to obtain text features, wherein the text features include text keywords;

[0025] Using spatial feature extraction technology to extract spatial features of the UI element in the corresponding key frame, the spatial features include position features and size features;

[0026] Based on the image features and the text features, a machine learning model is used to classify each of the UI elements to obtain element categories, where the element categories include buttons, text boxes, labels, icons, menus, and pictures;

[0027] Based on the spatial features and the element categories, the layout relationship of each of the UI elements on the web page to be generated is analyzed to generate network structure information.

[0028] Furthermore, the analyzing the layout relationship of the UI elements on the to-be-generated web page based on the spatial features and the element categories to generate web page structure information includes:

[0029] Obtaining a feature vector representation of each UI element according to the element category of the UI element;

[0030] Construct element nodes based on the spatial characteristics of the UI elements and determine the edge connection relationships between different element nodes;

[0031] Constructing a web page structure graph based on the feature vector representation and the edge connection nodes;

[0032] The web page structure diagram is input into a pre-trained graph neural network model to obtain the layout relationship of each UI element on the web page to be generated, generate a web page hierarchy tree, and output the web page structure information.

[0033] Furthermore, the code conversion model includes a generator network and a discriminator network;

[0034] Before inputting the webpage structure information into the code conversion model and outputting the webpage code, the process includes:

[0035] Using a generative adversarial network method, the webpage structure information is evaluated by the discriminator network, random noise is input into the generator network, and a simulated webpage structure code is generated;

[0036] Based on the evaluation results, adversarial training is performed on the generator network and the discriminator network, so that the generator network generates high-quality web page code.

[0037] Furthermore, the step of inputting the webpage structure information into a code conversion model and outputting webpage code includes:

[0038] Using a deep learning model to encode the webpage structure information to generate a text sequence representation of a tree structure;

[0039] According to predefined grammatical rules, the tree-like text sequence is parsed to obtain structured information;

[0040] Using the attention mechanism to extract features from the structured information to obtain a semantic representation of the web page structure;

[0041] Inputting the semantic information into a code conversion model for decoding to generate a corresponding HTML tag sequence;

[0042] A rule matching method is used to supplement the style and script information for the HTML tag sequence to generate web page code.

[0043] In a second aspect, an embodiment of the present application provides a video-based webpage generation system, comprising:

[0044] The first processing module is used to obtain a video file, convert the format of the video file, and obtain a standard video file;

[0045] The second processing module is used to build a multi-task learning model, input the standard video file into the multi-task learning model, and output the frame index of the key frame and the UI element in each key frame;

[0046] The third processing module is used to extract characteristic information from UI elements, classify each UI element according to the characteristic information, and determine the webpage structure information of the webpage to be generated;

[0047] A fourth processing module is used to input the webpage structure information into a code conversion model and output webpage code;

[0048] The fifth processing module is used to deploy the web page code to the server to generate an accessible web page.

[0049] In a third aspect, an embodiment of the present application provides a terminal device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the above-mentioned video-based web page generation method when executing the computer program.

[0050] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, comprising a computer-readable storage medium storing a computer program, wherein the computer program implements the above-mentioned video-based web page generation method when executed by a processor.

[0051] Compared with the prior art, the embodiments of the present application have the following beneficial effects:

[0052] The present application provides a video-based web page generation method that adopts a multi-task learning model to obtain key frames in a video file and the corresponding UI elements in the key frames, and generates web page code based on the obtained UI elements, thereby realizing automatic processing of video content and generating web page code, greatly reducing the time and effort of manual coding, and significantly improving the efficiency and convenience of web page development. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings required for use in the embodiments or descriptions of the prior art. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0054] Figure 1 This is a flow chart of a method for generating a web page based on video provided by one embodiment of the present invention;

[0055] Figure 2 This is a structural diagram of a video-based webpage generation system provided by one embodiment of the present invention;

[0056] Figure 3 It is a structural diagram of the terminal device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0057] In the following description, specific details such as specific system structures and techniques are provided for purposes of illustration rather than limitation to facilitate a thorough understanding of the embodiments of the present application. However, it will be apparent to those skilled in the art that the present application may be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits, and methods are omitted to avoid obscuring the description of the present application with unnecessary detail.

[0058] It should be understood that when used in the present specification and the appended claims, the term "comprising" indicates the presence of described features, integers, steps, operations, elements and / or components, but does not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or collections thereof.

[0059] It will also be understood that the term "and / or" used in this specification and the appended claims refers to and includes any and all possible combinations of one or more of the associated listed items.

[0060] As used in this specification and the appended claims, the term "if" can be interpreted as "when" or "upon" or "in response to determining" or "in response to detecting," depending on the context. Similarly, the phrase "if it is determined" or "if [described condition or event] is detected" can be interpreted as meaning "upon determination" or "in response to determining" or "upon detection of [described condition or event]" or "in response to detecting [described condition or event]," depending on the context.

[0061] In addition, in the description of the present application specification and the appended claims, the terms "first", "second", "third", etc. are only used to distinguish the descriptions and cannot be understood as indicating or implying relative importance.

[0062] References to "one embodiment" or "some embodiments" in this specification mean that a particular feature, structure, or characteristic described in conjunction with that embodiment is included in one or more embodiments of the present application. Thus, phrases such as "in one embodiment," "in some embodiments," "in other embodiments," and "in other embodiments" appearing in various places in this specification do not necessarily refer to the same embodiment, but rather mean "one or more but not all embodiments," unless otherwise specifically emphasized. The terms "including," "comprising," "having," and variations thereof all mean "including but not limited to," unless otherwise specifically emphasized.

[0063] See also Figure 1 As shown, the present invention is a method for generating a web page based on a video, comprising the following steps:

[0064] S100: Acquire a video file, and convert the format of the video file to obtain a standard video file;

[0065] The video-based web page generation method of the present application provides a video upload page for users to upload video files that they want to convert into web pages. The above-mentioned video upload page can be a web application or desktop application, which supports multiple video formats, such as MP4, AVI, MOV, etc.

[0066] In the present embodiment, before the video file is carried out format conversion, it is necessary to determine whether the video format of the video file is a preset format, and then determine whether to carry out the format conversion operation. If the video file is already in the preset format, then just can directly carry out the subsequent web page generation operation, without carrying out unnecessary format conversion process, thereby saving computing resources and time. For the video file that needs to carry out format conversion, by using the video processing library, the video is converted into a standard video file, so that subsequent processing is convenient. In a preferred embodiment, the format conversion of the video file adopts the video processing library of ffmpeg, according to the video attributes and the target format requirement, corresponding transcoding parameters are set, and attributes such as the encoding format, resolution, bit rate of the video are controlled by the transcoding parameters to meet the requirement of the target format, while ensuring the quality of the video after the conversion. After conversion is complete, the standard video file obtained by conversion is passed to the next step for subsequent web page generation operation.

[0067] S200: Build a multi-task learning model, input the standard video file into the multi-task learning model, and output the frame index of the key frame and the UI element in each key frame;

[0068] In this embodiment, a multi-task learning model is constructed to simultaneously process multiple tasks in a standard video file. Specifically, it identifies key frames in a video file and extracts UI elements within those key frames. By simultaneously optimizing multiple related tasks, the multi-task learning model leverages information sharing and complementarity between tasks, thereby improving the prediction accuracy of each task. In video processing, this means more accurate identification of key frames in a video file and more precise extraction of UI elements within those key frames.

[0069] In some embodiments, the multi-task learning model includes a keyframe extraction task and a UI element recognition task; wherein the above step S200 includes:

[0070] For the key frame extraction task, a sparsity constraint method is used to determine whether the sparsity of the key frame meets the preset requirements, and an L1 regularization term is introduced to enable the multi-task learning model to learn a sparse key frame representation;

[0071] For the UI element recognition task, a comprehensive constraint method is adopted to identify all UI elements in the keyframe;

[0072] A dynamic weight adjustment strategy is adopted to dynamically adjust the weights of the key frame extraction task and the UI element recognition task according to their difficulty and task completion degree;

[0073] During the multi-task learning model training process, an alternating optimization strategy is adopted to train the key frame extraction task and the UI element recognition task in turn.

[0074] In this embodiment, for the key frame extraction task, a method based on sparsity constraints is adopted to ensure that the multi-task learning model can effectively filter out the most representative key frames from the long video sequence of the video file. Through the above method, the multi-task learning model can continuously try to reduce the number of non-key frames during the training process, while retaining those key frames that can best summarize the video content.

[0075] In this embodiment, the key frames extracted in the key frame extraction task are judged to see whether they meet the requirements based on the preset sparsity threshold. If not, the multi-task learning model is further constrained to learn a sparser key frame representation by increasing the weight of the L1 regularization term.

[0076] In this embodiment, for each extracted key frame, a comprehensive constraint method is adopted so that the multi-task learning model can recognize and distinguish all UI elements on the key frame. Specifically, a loss function is set for the UI recognition task, and the parameters in the loss function are adjusted to optimize the model.

[0077] In this embodiment, a dynamic weight adjustment strategy is adopted to balance the keyframe extraction task and the UI element recognition task. The weights of the relevant tasks are dynamically adjusted according to the difficulty of the two tasks in the multi-task learning model, so that the multi-task learning model can reduce the number of keyframes while ensuring the integrity of UI elements.

[0078] In this embodiment, the difficulty of the key frame extraction task and the difficulty of the UI element recognition task of the standard video file are judged according to preset rules to obtain the difficulty coefficients of the two tasks, wherein the preset rules are as follows: the difficulty evaluation criteria for the key frame extraction task include video length, dynamic complexity, lighting changes and blurriness. If the video is longer, the number of frames required to be processed for key frame extraction is greater, and the difficulty increases accordingly. In addition, the difficulty of key frame extraction is determined by judging the speed and frequency of scene changes in the video, the degree of change of lighting conditions in the video, and the clarity of the video content. The difficulty evaluation criteria for the UI element recognition task include interface complexity, color contrast, occlusion and resolution ratio. Specifically, the difficulty of UI element recognition is judged by judging the number, type and layout complexity of UI elements in the relevant video image of the key frame, as well as the color contrast between UI elements and the background, the degree of occlusion between adjacent UI elements, and the resolution of the video image of the key frame. Specifically, a basic difficulty value is set for each evaluation criterion of the key frame extraction task and the UI element recognition task, and adjusted according to the specific situation of the video file. All adjusted difficulty values are weighted and summed to obtain the key frame extraction difficulty coefficient and the UI element recognition difficulty coefficient, respectively. Based on the key frame extraction difficulty coefficient and the UI element recognition difficulty coefficient, a normalization algorithm is used to obtain the initial weight values of the two tasks.

[0079] In this embodiment, the weights of the keyframe extraction task and the UI element recognition task are dynamically adjusted based on their real-time completion levels. Specifically, task completion thresholds are set for each task. If the completion level of a task falls below the preset threshold, the weight of the task is increased, while the weight of the other task is decreased.

[0080] In this embodiment, a video sequence of a standard video file is segmented into individual frames, and preliminary feature extraction, such as color histogram and texture features, is performed on each frame to obtain an initial video frame feature representation. Each frame is then scored based on preset keyframe evaluation metrics, such as inter-frame differences and visual saliency. The top N frames with the highest scores are selected as candidate keyframes, and the candidate keyframes are input into a pre-trained UI recognition model to obtain UI elements. Based on the obtained UI elements, the candidate keyframes are re-scored, and the keyframe selection results are updated. The updated keyframes are used as training samples and input into a keyframe extraction model for training. The performance of keyframe extraction is improved by optimizing the loss function related to keyframe extraction. The UI element recognition results are used as training samples and input into a UI element recognition model for training. The performance of UI element recognition is improved by optimizing the loss function related to UI element recognition. By alternately optimizing the keyframe extraction task and the UI element recognition task, the two tasks promote each other, ultimately obtaining a joint keyframe extraction and UI element recognition model with excellent performance, i.e., a multi-task learning model.

[0081] In some embodiments, the above step S200 further includes:

[0082] Constructing an encoder-decoder network structure, the encoder-decoder network structure includes an encoder part and a decoder part, the encoder part uses a convolutional neural network to extract the spatiotemporal features of the standard video file, and the decoder part uses a long short-term memory network to generate a key frame index sequence and a UI element information sequence;

[0083] Adding an attention mechanism module to the last convolutional layer of the encoder part to learn the importance weights of different time steps in the standard video file;

[0084] Introducing a pointer network structure into the decoder part, judging whether each time step is a key frame based on the spatiotemporal features, and if the time step is a key frame, outputting the frame index corresponding to the time step and the UI element corresponding to the key frame;

[0085] The key frame extraction task and the UI element recognition task are dynamically optimized and trained through a preset multi-task loss function, wherein the multi-task loss function includes binary cross entropy loss, multi-category cross entropy loss and regression loss.

[0086] In the present embodiment, by constructing an encoder-decoder network structure, in which the encoder part adopts a convolutional neural network, it is possible to effectively extract rich spatiotemporal features from standard video files, capture motion information and scene changes in the video, and provide a strong feature basis for subsequent processing. At the same time, an attention mechanism module is added after the last convolution layer of the encoder part, which can learn and automatically assign different time steps (frames) of the video with different importance weights. This mechanism enables the network to be more focused on the key parts in the video, that is, those frames that contain important information or significant changes, thereby improving the accuracy and efficiency of key frame extraction. In addition, the decoder part adopts a long short-term memory network and introduces a pointer network structure, so that the network can accurately judge whether each time step is a key frame based on the spatiotemporal features extracted by the encoder. If it is determined to be a key frame, the index of the frame and its corresponding UI element are further output. This joint recognition method not only improves the accuracy of key frame detection, but also realizes the automatic recognition and extraction of UI elements in the video, enhancing the depth of video analysis and understanding.

[0087] In this example, a multi-task loss function, including binary cross-entropy loss, multi-class cross-entropy loss, and regression loss, is designed to dynamically optimize training for both the keyframe extraction task and the UI element recognition task. This multi-task learning strategy helps the network simultaneously focus on and optimize two related but not identical objectives, avoiding the overfitting or suboptimal solutions that can result from single-task training, thereby improving the generalization and performance of the overall model.

[0088] Overall, the multi-task learning model in this embodiment, employing an encoder-decoder network structure and its optimization strategy, significantly improves the efficiency and accuracy of extracting keyframes and identifying UI elements from videos. This not only reduces the cost and time of manual review, but also provides strong technical support for areas such as video content analysis, automatic summary generation, and user behavior analysis, enhancing the practicality and application value of video processing technology.

[0089] S300, extracting characteristic information from UI elements, classifying each UI element according to the characteristic information, and determining webpage structure information of a webpage to be generated;

[0090] In some embodiments, step S300 includes:

[0091] Processing the image of the UI element using image processing technology to obtain image features, wherein the image features include edge features, color features, and texture features;

[0092] Processing the text of the UI element using natural language processing technology to obtain text features, wherein the text features include text keywords;

[0093] Using spatial feature extraction technology to extract spatial features of the UI element in the corresponding key frame, the spatial features include position features and size features;

[0094] Based on the image features and the text features, a machine learning model is used to classify each of the UI elements to obtain element categories, where the element categories include buttons, text boxes, labels, icons, menus, and pictures;

[0095] Based on the spatial features and the element categories, the layout relationship of each of the UI elements on the web page to be generated is analyzed to generate web page structure information.

[0096] In this embodiment, the UI elements output by the multi-task learning model include the image, size, text and bounding box of the UI elements. The UI elements are processed by using image processing technology and natural language technology to obtain image features, text features and spatial features related to the UI elements. The extracted features are classified to determine the category of the UI elements.

[0097] In this embodiment, the Canny edge detection algorithm is used to extract edge features of UI elements and identify their shapes and outlines. The color distribution of UI elements is analyzed, such as using RGB histograms or HSV space, to identify elements such as buttons and labels. Gabor filters or local binary patterns are used to extract texture features of UI elements and distinguish between different types of background and content areas.

[0098] In this embodiment, for text features, models such as TF-IDF or BERT are used to extract keywords from the text to help determine the main content and functions of UI elements. Word embedding technologies such as Word2Vec and GloVe are used to convert the text into vector representation for subsequent classification and analysis.

[0099] In this embodiment, for spatial features, the position of the UI element in the page, such as relative coordinates, is recorded to determine the position and hierarchical relationship of the UI element in the page layout. By recording the width, height and area of the UI element, the layout relationship of the UI element on the web page to be generated is further determined.

[0100] In some embodiments, analyzing the layout relationship of the UI elements on the to-be-generated web page based on the spatial features and the element categories to generate web page structure information includes:

[0101] Obtaining a feature vector representation of each UI element according to the element category of the UI element;

[0102] Construct element nodes based on the spatial characteristics of the UI elements and determine the edge connection relationships between different element nodes;

[0103] Constructing a web page structure graph based on the feature vector representation and the edge connection nodes;

[0104] The web page structure diagram is input into a pre-trained graph neural network model to obtain the layout relationship of each UI element on the web page to be generated, generate a web page hierarchy tree, and output the web page structure information.

[0105] In this embodiment, each UI element represents a feature vector, which includes the image features, text features and spatial features of the feature vector. For example, a UI element classified as a button has a feature vector including feature information such as its element category, position, size, color, etc. Specifically, the above feature information is converted into vector form through One-Hot encoding.

[0106] In this embodiment, UI element nodes are constructed based on the spatial positional relationships between different UI elements. For example, if the center points of two UI elements are less than 100 pixels apart, an edge is established between them to indicate their spatial proximity. Based on the feature vectors and edge connectivity of the UI element nodes, an adjacency matrix and a feature matrix are used to construct a graph neural network, i.e., a webpage structure graph.

[0107] In this embodiment, the constructed graph neural network is input into a preset graph neural network model, such as a Graph Convolutional Network, for training. The cross entropy loss function and the Adam optimization algorithm are used, the learning rate is set to 01, and 1000 epochs are trained to obtain a trained GCN model. The trained GCN model is used to predict the category of each UI element node. For example, the probability of predicting a node as a button is 8, and the probability of predicting a node as a text box is 2. Based on the predicted UI element node category labels and the edge connection relationship between the nodes, a depth-first search algorithm is used to generate a hierarchical tree of the web page. For example, starting from the root node, the graph is traversed, and adjacent nodes of the same type are grouped into a subtree. Different subtrees are generated for nodes of different classes, ultimately resulting in a tree representing the structure of the web page. Finally, based on the generated web page hierarchical tree, the structural information of the web page is output. For example, the tree structure is converted into JSON format. Each node represents a UI element, including its category, location, child nodes, and other information, which facilitates subsequent analysis and application.

[0108] S400: Input the webpage structure information into a code conversion model and output the webpage code;

[0109] In some embodiments, the transcoding model includes a generator network and a discriminator network; wherein, before the above step S400, it includes:

[0110] Using a generative adversarial network method, the webpage structure information is evaluated by the discriminator network, random noise is input into the generator network, and a simulated webpage structure code is generated;

[0111] Based on the evaluation results, adversarial training is performed on the generator network and the discriminator network, so that the generator network generates high-quality web page code.

[0112] In this embodiment, in order to ensure the diversity of UI element layout, a generative adversarial network method is adopted to train the code conversion model. Specifically, a discriminator network is introduced to evaluate the generated web page structure information. Through adversarial training of the generator network and the discriminator network, the code transcoding model from the UI element layout to the web page code is continuously optimized to improve the quality and accuracy of the generated web page.

[0113] In this embodiment, a set of UI element layout diagrams are obtained as a training data set for the code conversion model. The UI element layout diagrams need to include different styles and layout methods to improve the generation diversity. A code conversion model is constructed, including a generator network and a discriminator network. The generator network uses random noise as input to generate simulated web page structure codes. The discriminator network uses real web page codes and simulated web page structure codes generated based on the UI element layout as input, extracts features through a convolutional neural network, and determines whether the discriminator network input is real web page code or simulated web page code. During the training process, the generator network and the discriminator network are optimized alternately. The generator network generates realistic web page structure codes as much as possible to deceive the discriminator, while the discriminator network distinguishes between real and generated web page codes as accurately as possible. The mapping model of UI layout to web page code is continuously optimized through adversarial training, so that the generator network can generate corresponding high-quality web page code according to the input UI layout diagram. When generating web page code, the diversity of the generated web page structure can be controlled by changing the distribution of input noise to adapt to different layout requirements. Evaluate the quality of the generated web page code, remove the generated results with chaotic structure and misplaced layout, ensure the usability of the generated web page, and ultimately output high-quality and diversified web page design solutions.

[0114] In some embodiments, step S400 includes:

[0115] Using a deep learning model to encode the webpage structure information to generate a text sequence representation of a tree structure;

[0116] According to predefined grammatical rules, the tree-like text sequence is parsed to obtain structured information;

[0117] Using the attention mechanism to extract features from the structured information to obtain a semantic representation of the web page structure;

[0118] Inputting the semantic information into a code conversion model for decoding to generate a corresponding HTML tag sequence;

[0119] A rule matching method is used to supplement the style and script information for the HTML tag sequence to generate web page code.

[0120] In this embodiment, based on the web page hierarchy tree of the web page structure information, a deep learning model is used to encode the web page structure information to generate a tree-like text sequence representation. Specifically, a Transformer-based encoder is used to encode the input web page structure information, convert the web page structure information into a vector representation, and then process the vector through a multi-layer self-attention mechanism and a feedforward neural network to generate a tree-like text sequence representation. When parsing the tree-like text sequence, an LSTM-based recursive neural network model is used to parse the text sequence in a bottom-up manner according to pre-defined grammatical rules to obtain structured information. The parsed structured information is subjected to feature extraction through an attention mechanism and encoded using a Bi-LSTM network to obtain a semantic representation vector of the web page structure. The semantic representation vector is used as input and the corresponding HTML tags are gradually generated through a decoder.

[0121] In this embodiment, a rule-based matching method is used to supplement style and script information for the generated tag sequence. For example, font size and color styles are added according to the hierarchical relationship of the title, and bullet points or numbers are added to the list. When performing syntax checking on the webpage code, an analysis method based on the abstract syntax tree can be used. The code is converted into a syntax tree through recursive descent parsing. The tree is then traversed to check for syntax errors or non-compliant parts, such as unclosed tags and missing attributes, and appropriate corrections are made.

[0122] S500: Deploy the webpage code to a server to generate an accessible webpage.

[0123] In this embodiment, a web page code file is obtained and the list of files and directory structure that need to be deployed is determined. The target server is connected, and based on the server type and operating system, an appropriate file transfer protocol is selected to upload the web page code file to the server's designated directory. Based on the programming language and framework of the web page code, the corresponding operating environment is configured on the server, such as installing a programming language interpreter, web server software, etc. The website's access rights and security policies are configured on the server, and appropriate user permissions and access control lists are set to prevent unauthorized access and malicious attacks. The website's domain name, port, virtual host and other parameters are configured in the web server software so that the server can correctly respond to user requests and map the requests to the corresponding web page files. The web service is started, listening to the designated port, and waiting for users to access it through the network. When a user request is received, the server parses the request parameters and calls the corresponding web page file for processing. The server dynamically generates web page content and returns the generated HTML, CSS, JavaScript and other codes to the user's browser. The browser parses the received web page code, renders a visual web page interface, and presents it to the user for interaction and browsing.

[0124] In some embodiments, before deploying the web page code to the server and generating an accessible web page, the web page code is also included in the syntax analysis to determine whether there are syntax errors in the web page code. If there are syntax errors, the syntax errors are corrected according to the syntax rules to obtain grammatically correct web page code. Thereafter, the grammatically correct web page code is formatted according to the preset web page code specifications, and the code indentation, spaces, and line breaks are adjusted to make it meet the specification requirements, and the final standardized web page code is determined.

[0125] In some embodiments, after the web page code is deployed to the server and an accessible web page is generated, it also includes implementing feedback information monitoring of the user's accessible web page, and the feedback information includes interactive behavior data and feedback evaluation. Specifically, web page embedding data is obtained to record the user's interactive behaviors such as clicking and scrolling on the web page; by setting up a feedback form in the web page, the user's subjective evaluation and suggestions on the web page are collected; the collected user interactive behavior data is cleaned and preprocessed to eliminate abnormal data; natural language processing is performed on the user feedback evaluation content to extract keywords and emotional tendencies; the user's browsing path and focus on the web page are analyzed based on the interactive behavior data; a collaborative filtering algorithm is used to cluster the feedback based on the user's similar behavior patterns; and structured user feedback information is generated by integrating the results of interactive behavior analysis and feedback evaluation mining.

[0126] See also Figure 2 As shown, the present invention also provides a video-based web page generation system, the system comprising:

[0127] The first processing module 201 is used to obtain a video file and convert the format of the video file to obtain a standard video file;

[0128] The second processing module 202 is used to build a multi-task learning model, input the standard video file into the multi-task learning model, and output the frame index of the key frame and the UI element in each key frame;

[0129] The third processing module 203 is used to extract characteristic information from UI elements, classify each UI element according to the characteristic information, and determine the webpage structure information of the webpage to be generated;

[0130] The fourth processing module 204 is used to input the webpage structure information into the code conversion model and output the webpage code;

[0131] The fifth processing module 205 is used to deploy the webpage code to the server to generate an accessible webpage.

[0132] It is understandable that if Figure 1 The contents of the video-based web page generation method embodiment shown are all applicable to the video-based web page generation system embodiment. The functions specifically implemented by the video-based web page generation system embodiment are similar to those in the embodiment shown in FIG. Figure 1 The embodiment of the video-based web page generation method shown is the same as that shown in FIG. Figure 1 The beneficial effects achieved by the embodiment of the video-based webpage generation method shown are also the same.

[0133] It should be noted that the information interaction, execution process and other contents between the above-mentioned systems are based on the same concept as the embodiment of the method of the present invention. Their specific functions and technical effects can be found in the method embodiment part and will not be repeated here.

[0134] Those skilled in the art can clearly understand that, for the convenience and brevity of description, only the division of the above-mentioned functional units and modules is used as an example for illustration. In actual applications, the above-mentioned functions can be distributed and completed by different functional units and modules as needed, that is, the internal structure of the system can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiment can be integrated into one processing unit, or each unit can exist physically alone, or two or more units can be integrated into one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of software functional units. In addition, the specific names of the functional units and modules are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of this application. The specific working process of the units and modules in the above-mentioned system can refer to the corresponding process in the aforementioned method embodiment, and will not be repeated here.

[0135] See also Figure 3 As shown, an embodiment of the present invention further provides a computer device 3, comprising: a memory 302 and a processor 301 and a computer program 303 stored on the memory 302. When the computer program 303 is executed on the processor 301, the video-based web page generation method as described in any one of the above methods is implemented.

[0136] The computer device 3 may be a desktop computer, a notebook computer, a PDA, a cloud server or other computing devices. The computer device 3 may include, but is not limited to, a processor 301 and a memory 302. Those skilled in the art will understand that Figure 3 This is merely an example of the computer device 3 and does not constitute a limitation on the computer device 3 . The computer device 3 may include more or fewer components than shown in the figure, or a combination of certain components, or different components. For example, the computer device 3 may also include input and output devices, network access devices, etc.

[0137] The processor 301 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. A general-purpose processor may be a microprocessor or any conventional processor.

[0138] In some embodiments, the memory 302 may be an internal storage unit of the computer device 3, such as a hard disk or memory of the computer device 3. In other embodiments, the memory 302 may also be an external storage device of the computer device 3, such as a plug-in hard disk, a Smart Media Card (SMC), a Secure Digital (SD) card, a Flash Card, etc. equipped on the computer device 3. Furthermore, the memory 302 may include both an internal storage unit of the computer device 3 and an external storage device. The memory 302 is used to store an operating system, application programs, a boot loader, data, and other programs, such as the program code of the computer program. The memory 302 may also be used to temporarily store data that has been output or is about to be output.

[0139] An embodiment of the present invention further provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the video-based webpage generation method as described in any one of the above methods is implemented.

[0140] In this embodiment, if the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present application implements all or part of the process of the above-mentioned method embodiment by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium. When the computer program is executed by a processor, it can implement the steps of each of the above-mentioned method embodiments. The computer program includes computer program code, which can be in source code form, object code form, executable file, or some intermediate form. The computer-readable medium can at least include: any entity or device capable of carrying computer program code to the camera / terminal device, recording medium, computer memory, read-only memory (ROM), random access memory (RAM), electric carrier signal, telecommunication signal, and software distribution medium. For example, a USB flash drive, mobile hard drive, magnetic disk, or optical disk. In some jurisdictions, based on legislation and patent practice, computer-readable media cannot be electric carrier signals or telecommunication signals.

[0141] The above-described embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. These modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the various embodiments of the present application, and should all be included in the scope of protection of the present application.

Claims

1. A video-based web page generation method, characterized in that: include: Acquire a video file, and convert the format of the video file to obtain a standard video file; Constructing a multi-task learning model, inputting the standard video file into the multi-task learning model, and outputting the frame index of the key frame and the UI element in each key frame; Extracting characteristic information from UI elements, classifying each UI element according to the characteristic information, and determining webpage structure information of the webpage to be generated; Inputting the webpage structure information into a code conversion model and outputting webpage code; Deploy the web page code to a server to generate an accessible web page.

2. The method according to claim 1, wherein The multi-task learning model includes a key frame extraction task and a UI element recognition task; The constructing of a multi-task learning model, inputting the standard video file into the multi-task learning model, and outputting the frame index of the key frame and the UI element in each key frame includes: For the key frame extraction task, a sparsity constraint method is used to determine whether the sparsity of the key frame meets the preset requirements, and an L1 regularization term is introduced to enable the multi-task learning model to learn a sparse key frame representation; For the UI element recognition task, a comprehensive constraint method is adopted to identify all UI elements in the keyframe; A dynamic weight adjustment strategy is adopted to dynamically adjust the weights of the key frame extraction task and the UI element recognition task according to their difficulty and task completion degree; During the multi-task learning model training process, an alternating optimization strategy is adopted to train the key frame extraction task and the UI element recognition task in turn.

3. The method according to claim 2, wherein The constructing of a multi-task learning model, inputting the standard video file into the multi-task learning model, and outputting the frame index of the key frame and the UI element in each key frame further includes: Constructing an encoder-decoder network structure, the encoder-decoder network structure includes an encoder part and a decoder part, the encoder part uses a convolutional neural network to extract the spatiotemporal features of the standard video file, and the decoder part uses a long short-term memory network to generate a key frame index sequence and a UI element information sequence; Adding an attention mechanism module to the last convolutional layer of the encoder part to learn the importance weights of different time steps in the standard video file; Introducing a pointer network structure into the decoder part, judging whether each time step is a key frame based on the spatiotemporal features, and if the time step is a key frame, outputting the frame index corresponding to the time step and the UI element corresponding to the key frame; The key frame extraction task and the UI element recognition task are dynamically optimized and trained through a preset multi-task loss function, wherein the multi-task loss function includes binary cross entropy loss, multi-category cross entropy loss and regression loss.

4. The method according to claim 1, wherein The extracting characteristic information from the UI elements, classifying each UI element according to the characteristic information, and determining the webpage structure information of the webpage to be generated includes: Processing the image of the UI element using image processing technology to obtain image features, wherein the image features include edge features, color features, and texture features; Processing the text of the UI element using natural language processing technology to obtain text features, wherein the text features include text keywords; Using spatial feature extraction technology to extract spatial features of the UI element in the corresponding key frame, the spatial features include position features and size features; Based on the image features and the text features, a machine learning model is used to classify each of the UI elements to obtain element categories, where the element categories include buttons, text boxes, labels, icons, menus, and pictures; Based on the spatial features and the element categories, the layout relationship of each of the UI elements on the web page to be generated is analyzed to generate network structure information.

5. The method according to claim 4, wherein The analyzing, based on the spatial features and the element categories, the layout relationship of the UI elements on the to-be-generated web page to generate web page structure information includes: Obtaining a feature vector representation of each UI element according to the element category of the UI element; Construct element nodes based on the spatial characteristics of the UI elements and determine the edge connection relationships between different element nodes; Constructing a web page structure graph based on the feature vector representation and the edge connection nodes; The web page structure diagram is input into a pre-trained graph neural network model to obtain the layout relationship of each UI element on the web page to be generated, generate a web page hierarchy tree, and output the web page structure information.

6. The method according to claim 1, wherein The code conversion model includes a generator network and a discriminator network; Before inputting the webpage structure information into the code conversion model and outputting the webpage code, the process includes: Using a generative adversarial network method, the webpage structure information is evaluated by the discriminator network, random noise is input into the generator network, and a simulated webpage structure code is generated; Based on the evaluation results, adversarial training is performed on the generator network and the discriminator network, so that the generator network generates high-quality web page code.

7. The method according to claim 6, wherein The step of inputting the webpage structure information into a code conversion model and outputting webpage code comprises: Using a deep learning model to encode the webpage structure information to generate a text sequence representation of a tree structure; According to predefined grammatical rules, the tree-like text sequence is parsed to obtain structured information; Using the attention mechanism to extract features from the structured information to obtain a semantic representation of the web page structure; Inputting the semantic information into a code conversion model for decoding to generate a corresponding HTML tag sequence; A rule matching method is used to supplement the style and script information for the HTML tag sequence to generate web page code.

8. A video-based web page generation system, characterized in that: include: The first processing module is used to obtain a video file, convert the format of the video file, and obtain a standard video file; The second processing module is used to build a multi-task learning model, input the standard video file into the multi-task learning model, and output the frame index of the key frame and the UI element in each key frame; The third processing module is used to extract characteristic information from UI elements, classify each UI element according to the characteristic information, and determine the webpage structure information of the webpage to be generated; A fourth processing module is used to input the webpage structure information into a code conversion model and output webpage code; The fifth processing module is used to deploy the web page code to the server to generate an accessible web page.

9. A terminal device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the method according to any one of claims 1 to 7 is implemented.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images