Screen projection method and system of embedded device

Abstract

The invention relates to the technical field of equipment screen projection, and discloses a screen projection method and system for embedded equipment, and the method comprises the steps: scheduling a terminal display task corresponding to a projection display terminal, carrying out the display updating processing of a display screen of the embedded equipment, obtaining updated embedded equipment, and collecting a current screen display scene corresponding to the updated embedded equipment; analyzing display coupling between the visual scene visual elements and the terminal display task, and scheduling display streaming media corresponding to the equipment display task from the current scene of the screen display; analyzing display key frames in the display streaming media, extracting dynamic visual features in the display key frames, and analyzing interface display logic between the display streaming media; and transmitting the coded streaming media to a projection display terminal to obtain a streaming media to be displayed, and performing teaching screen projection processing on the streaming media to be displayed by using the projection display terminal to obtain a teaching screen projection result. The invention mainly aims to solve the problem of poor screen projection practical effect of embedded equipment.

Description

Technical Field

[0001] The present invention relates to a screen projection method and system for an embedded device, belonging to the technical field of device screen projection. Background Art

[0002] In today's era of rapid technological development, embedded devices are making great strides towards intelligence, compactness and wirelessness. While this development trend brings many advantages to embedded devices, it also has a significant impact on their traditional interface configuration. The HDMI interface, as a common interface used for high-definition video transmission and screen projection in the past, has gradually become less necessary under the new design concept. More and more embedded devices no longer have HDMI interfaces in order to pursue smaller size, lower power consumption and simpler appearance design. However, in some specific application scenarios, such as venue teaching scenarios, this change has exposed obvious limitations. Traditional venue teaching often relies on the screen projection function to display the teaching content in the embedded device to a wide range of audiences. Without the HDMI interface, the embedded device is only suitable for a small range of on-site observation and teaching, and it is difficult to project the content onto a large screen through screen projection, thus failing to meet the needs of large-scale demonstrations in large conference rooms, tiered classrooms and other places. This has largely limited the effective application of embedded devices in education, training and other fields, and has led to poor practical effects of screen projection for embedded devices. Summary of the invention

[0003] The present invention provides a screen projection method and system for an embedded device, the main purpose of which is to solve the problem of poor practical effect of screen projection of the embedded device.

[0004] To achieve the above object, the present invention provides a screen projection method for an embedded device, comprising:

[0005] Acquire the embedded device and projection display terminal to be processed, schedule the terminal display task corresponding to the projection display terminal, perform display update processing on the display screen of the embedded device according to the preset embedded program, obtain the updated embedded device, and collect the current screen display scene corresponding to the updated embedded device;

[0006] Extracting visual elements of the scene in the current scene displayed on the screen, analyzing the display coupling between the visual elements of the scene and the terminal display task, and scheduling display streaming media corresponding to the device display task from the current scene displayed on the screen based on the display coupling;

[0007] Analyze the display key frames in the display streaming media, extract the dynamic visual features in the display key frames, and analyze the interface display logic between the display streaming media according to the dynamic visual features;

[0008] The display streaming media is encoded to obtain encoded streaming media, based on the interface display logic, the encoded streaming media is transmitted to the projection display terminal to obtain streaming media to be displayed, and the projection display terminal is used to perform teaching screen projection processing on the streaming media to be displayed to obtain a teaching screen projection result.

[0009] Optionally, performing display update processing on a display screen of the embedded device according to a preset embedded program to obtain an updated embedded device includes:

[0010] Performing code parsing processing on the embedded program to obtain a program instruction sequence;

[0011] Sorting and optimizing the program instruction sequence to obtain an optimized instruction set;

[0012] Performing memory allocation processing on the optimized instruction set to obtain executable instruction data;

[0013] Performing data loading processing on the executable instruction data to obtain loading data;

[0014] Performing interface element construction processing on the loaded data to obtain display data;

[0015] Performing rendering and adaptation processing on the display data to obtain adapted image data for the display screen of the embedded device;

[0016] The adapted image data is subjected to display driving processing to obtain an updated embedded device.

[0017] Optionally, extracting visual elements of the scene in the current scene displayed on the screen includes:

[0018] Performing image noise reduction processing on the current visual scene displayed on the screen to obtain a noise-reduced visual scene image;

[0019] Performing element detection on the denoised visual image to obtain visual image elements;

[0020] Performing element clustering processing on the visual image elements to obtain clustered visual elements;

[0021] Calculating the element importance corresponding to each element in the clustered visual elements;

[0022] Based on the element importance, the visual elements of the scene in the current scene displayed on the screen are extracted from the clustered scene elements.

[0023] Optionally, the analyzing the display coupling between the visual element of the scene and the terminal display task includes:

[0024] Performing content recognition on the terminal display task to obtain display task content;

[0025] Performing semantic analysis on the displayed task content to obtain task semantics;

[0026] Based on the task semantics, determining a task display requirement corresponding to the terminal display task;

[0027] Analyzing the element visual attributes corresponding to the visual elements of the scene;

[0028] Calculating a display coupling coefficient between the element visual attribute and the task display requirement;

[0029] The display coupling between the scene visual element and the terminal display task is analyzed according to the display coupling coefficient.

[0030] Optionally, the calculating a display coupling coefficient between the element visual attribute and the task display requirement includes:

[0031] Vectorizing the element visual attributes and the task display requirements respectively to obtain a visual attribute vector and a display requirement vector;

[0032] Combining the visual attribute vector and the display requirement vector, the display matching degree between the element visual attribute and the task display requirement is calculated by the following formula:

[0033]

[0034] Among them, A represents the display matching degree between the element visual attributes and the task display requirements, and B a represents the vector corresponding to the visual attribute of the ath element in the visual attribute vector, a represents the sequence number of the element visual attribute, m represents the number of the element visual attributes, D b represents the vector corresponding to the b-th task display requirement in the display requirement vector, b represents the sequence number of the task display requirement, and n represents the number of task display requirements;

[0035] A display coupling coefficient between the element visual attribute and the task display requirement is calculated according to the display matching degree.

[0036] Optionally, analyzing the display key frames in the display streaming media includes:

[0037] Performing frame processing on the display streaming media to obtain a display frame set;

[0038] Performing optimal processing on the display frame set to obtain a target display frame;

[0039] Performing grayscale processing on the target display frame to obtain a grayscale display frame;

[0040] The texture complexity corresponding to the grayscale display sub-frame is calculated, and based on the texture complexity, the display key frame in the display streaming media is analyzed from the target display sub-frame.

[0041] Optionally, the calculating the texture complexity corresponding to the grayscale display frame includes:

[0042] Measuring the frame grayscale level corresponding to the grayscale display frame and the number of grayscale levels corresponding thereto;

[0043] Traversing the grayscale pairs in the frame grayscales to construct a grayscale co-occurrence matrix corresponding to the grayscale display frame;

[0044] Based on the gray level co-occurrence matrix, calculating the gray level co-occurrence probability of the gray level pair;

[0045] Combining the frame grayscale, the number of grayscales and the grayscale co-occurrence probability, the texture complexity corresponding to the grayscale display frame is calculated by the following formula:

[0046]

[0047] Among them, C represents the texture complexity corresponding to the grayscale display frame, i and j represent the i grayscale level and j grayscale level in the frame grayscale, Q represents the number of grayscale levels, and E(i, j) represents the grayscale co-occurrence probability of the i grayscale level and the j grayscale level in the frame grayscale.

[0048] Optionally, extracting the dynamic visual features in the displayed key frame includes:

[0049] Performing subject detection on the displayed key frame to obtain the key frame subject;

[0050] Extracting subject feature points corresponding to the key frame subject, and identifying a display timestamp corresponding to the display key frame;

[0051] Tracking the motion trajectory between the subject feature points in combination with the display timestamp and the display key frame;

[0052] The trajectory motion parameters corresponding to the motion trajectory are calculated, and the motion trajectory and the trajectory motion parameters are combined to generate dynamic visual features in the display key frame.

[0053] Optionally, the transmitting the encoded streaming media to the projection display terminal based on the interface display logic to obtain the streaming media to be displayed includes:

[0054] Extracting display meta parameters corresponding to the encoded streaming media;

[0055] Combining the interface display logic and the display meta-parameters, encapsulating the encoded streaming media to obtain encapsulated streaming media;

[0056] Respectively query the wireless protocols corresponding to the update embedded device and the projection display terminal to obtain the device wireless protocol and the terminal wireless protocol;

[0057] Combining the device wireless protocol and the terminal wireless protocol, constructing a wireless transmission trunk between the update embedded device and the projection display terminal;

[0058] The encoded streaming media is transmitted to the projection display terminal by using the wireless transmission trunk channel to obtain the streaming media to be displayed.

[0059] A screen projection system for an embedded device, characterized in that the system comprises:

[0060] A visual acquisition module is used to obtain the embedded device and projection display terminal to be processed, schedule the terminal display task corresponding to the projection display terminal, perform display update processing on the display screen of the embedded device according to a preset embedded program, obtain an updated embedded device, and acquire the current screen display visual scene corresponding to the updated embedded device;

[0061] A display data scheduling module, used for extracting visual elements of the scene in the current scene displayed on the screen, analyzing the display coupling between the visual elements of the scene and the terminal display task, and scheduling the display streaming media corresponding to the device display task from the current scene displayed on the screen based on the display coupling;

[0062] A display logic analysis module, used to analyze the display key frames in the display streaming media, extract the dynamic visual features in the display key frames, and analyze the interface display logic between the display streaming media according to the dynamic visual features;

[0063] The teaching screen projection module is used to encode the display streaming media to obtain encoded streaming media, transmit the encoded streaming media to the projection display terminal based on the interface display logic to obtain the streaming media to be displayed, and use the projection display terminal to perform teaching screen projection processing on the streaming media to be displayed to obtain the teaching screen projection result.

[0064] Compared with the problems described in the background technology, the present invention performs display update processing on the display screen of the embedded device according to a preset embedded program to obtain an updated embedded device, which can improve the accuracy and timeliness of the displayed content and enhance the visualization effect and interactive performance of the operation of the embedded device. The present invention can obtain the visually perceived components of the current visual scene of the screen by extracting the visual elements of the visual scene in the current visual scene of the screen, thereby laying a foundation for the subsequent analysis of the display coupling between the visual elements of the visual scene and the terminal display task. The present invention can obtain the key content information in the display streaming media by analyzing the display key frames in the display streaming media, reduce the data volume of the display streaming media, and improve the analysis accuracy of the interface display logic between the subsequent display streaming media. The present invention can effectively compress the data volume by encoding the display streaming media, reduce the space and bandwidth required for subsequent storage and transmission, and transmit the encoded streaming media to the projection display terminal based on the interface display logic, which can ensure that the content of the encoded streaming media is accurately presented in a reasonable order and layout, such as the teaching demonstration can be carried out in an orderly manner, improve the accuracy of information transmission, and optimize the transmission and display process at the same time, reduce freezes and errors, make the projection picture smooth and stable, and enhance the interactive effect of screen projection. Therefore, the screen projection method and system of the embedded device proposed in the present invention can improve the screen projection interactive effect of the embedded device. BRIEF DESCRIPTION OF THE DRAWINGS

[0065] Figure 1 A schematic diagram of a flow chart of a screen projection method for an embedded device provided in one embodiment of the present invention;

[0066] Figure 2 A functional module diagram of a screen projection system for implementing the embedded device provided in one embodiment of the present invention.

[0067] The purpose, features and advantages of the present invention will be further described with reference to the accompanying drawings in conjunction with the embodiments. DETAILED DESCRIPTION

[0068] It should be understood that the specific embodiments described herein are only used to explain the present invention, and are not used to limit the present invention.

[0069] The embodiment of the present application provides a screen projection method for an embedded device. The execution subject of the screen projection method of the embedded device includes but is not limited to at least one of the electronic devices such as a server, a terminal, etc. that can be configured to execute the method provided by the embodiment of the present application. In other words, the screen projection method of the embedded device can be executed by software or hardware installed on a terminal device or a server device. The server includes but is not limited to: a single server, a server cluster, a cloud server or a cloud server cluster, etc.

[0070] Embodiment 1:

[0071] Reference Figure 1 FIG. 1 is a flow chart of a screen projection method for an embedded device provided in an embodiment of the present invention. In this embodiment, the screen projection method for an embedded device includes:

[0072] S1. Obtain the embedded device and projection display terminal to be processed, schedule the terminal display task corresponding to the projection display terminal, perform display update processing on the display screen of the embedded device according to a preset embedded program, obtain an updated embedded device, and collect the current screen view corresponding to the updated embedded device.

[0073] The present invention performs display update processing on the display screen of the embedded device according to a preset embedded program to obtain an updated embedded device, which can improve the accuracy and timeliness of the displayed content and enhance the visualization effect and interactive performance of the operation of the embedded device. It should be explained that the embedded device is an electronic device with specific functions, the projection display terminal is used to display the relevant content of the embedded device, the terminal display task is the specific display requirement corresponding to the projection display terminal, the preset embedded program is a pre-set program code for controlling the relevant operation and display operation of the embedded device, the updated embedded device is a device obtained after the display content of the display screen of the embedded device is updated, and the current screen view is the current display screen corresponding to the display screen of the updated embedded device; further, the scheduling of the terminal display task corresponding to the projection display terminal can be achieved through a display task scheduler, and the display task scheduler is compiled by a programming language; the collection of the current screen view corresponding to the updated embedded device can be achieved through a display driver interface.

[0074] In detail, the display screen of the embedded device is updated according to a preset embedded program to obtain an updated embedded device, including:

[0075] Performing code parsing processing on the embedded program to obtain a program instruction sequence;

[0076] Sorting and optimizing the program instruction sequence to obtain an optimized instruction set;

[0077] Performing memory allocation processing on the optimized instruction set to obtain executable instruction data;

[0078] Performing data loading processing on the executable instruction data to obtain loading data;

[0079] Performing interface element construction processing on the loaded data to obtain display data;

[0080] Performing rendering and adaptation processing on the display data to obtain adapted image data for the display screen of the embedded device;

[0081] The adapted image data is subjected to display driving processing to obtain an updated embedded device.

[0082] It should be explained that the program instruction sequence is the instruction arrangement sorted out by the embedded program according to the execution logic, the optimized instruction set is a more efficient and reasonable instruction set after the program instruction sequence is sequentially optimized and adjusted, the executable instruction data is the relevant data form of the optimized instruction set that is executable after memory allocation, the loaded data is the data for subsequent processing formed after reading and loading the executable instruction data, the display data is the data that can be used for display after the loaded data constructs the interface elements, and the adapted image data is the display data that is rendered and adapted to match the characteristics of the embedded device display screen.

[0083] Furthermore, the code parsing processing of the embedded program can be implemented through a dedicated code analysis tool; the sorting optimization processing of the program instruction sequence can be implemented through an instruction scheduler; the memory allocation processing of the optimized instruction set can be implemented through the memory management mechanism of the operating system; the data loading processing of the executable instruction data can be implemented through related components such as a storage controller; the interface element construction processing of the loaded data can be implemented through a graphics drawing library; the rendering adaptation processing of the display data can be implemented through a graphics rendering engine, a display adaptation algorithm, etc.; the display driver processing of the adapted image data can be implemented through the interaction between the corresponding display driver, the hardware abstraction layer and the underlying hardware.

[0084] S2. Extracting visual elements from the current visual scene displayed on the screen, analyzing display coupling between the visual elements and the terminal display task, and scheduling display streaming media corresponding to the device display task from the current visual scene displayed on the screen based on the display coupling.

[0085] The present invention can obtain the various components in the current scene of the screen display that can be visually perceived by extracting the scene visual elements in the current scene of the screen display, thereby laying a foundation for the subsequent analysis of the display coupling between the scene visual elements and the terminal display task. It should be explained that the scene visual elements are the visual perception components in the current scene of the screen display.

[0086] Specifically, the extracting of the visual elements of the scene in the current scene displayed on the screen includes:

[0087] Performing image noise reduction processing on the current visual scene displayed on the screen to obtain a noise-reduced visual scene image;

[0088] Performing element detection on the denoised visual image to obtain visual image elements;

[0089] Performing element clustering processing on the visual image elements to obtain clustered visual elements;

[0090] Calculating the element importance corresponding to each element in the clustered visual elements;

[0091] Based on the element importance, the visual elements of the scene in the current scene displayed on the screen are extracted from the clustered scene elements.

[0092] It should be explained that the denoised visual image is an image obtained after the current visual scene on the screen is subjected to image denoising processing, the visual image elements are constituent elements with different visual effects extracted from the denoised visual image through element detection, the clustered visual elements are a set of elements formed after the visual image elements are classified and aggregated according to certain rules (such as semantics), and the element importance indicates the importance of each element in the clustered visual elements in the overall display of the screen.

[0093] Furthermore, the image denoising of the current visual scene displayed on the screen can be implemented by a low-pass filter; the element detection of the denoised visual scene image can be implemented by an edge detection operator, such as a Sobel operator; the element categories corresponding to the visual image elements are analyzed, and the visual image elements are clustered according to the element categories to obtain clustered visual elements; the color difference between each element in the clustered visual elements is calculated, and the color difference can be calculated using a color difference formula, such as a CIELAB color difference formula, and the element color region area corresponding to the clustered visual element is calculated, the color difference and the element color region area are normalized and summed to obtain the element importance corresponding to the clustered visual element; based on the numerical value of the element importance, the visual elements in the current visual scene displayed on the screen are extracted from the clustered visual elements, and the clustered visual element with the largest element importance is selected as the visual element corresponding to each category.

[0094] The present invention analyzes the display coupling between the scene visual elements and the terminal display tasks, and can understand the correlation between the scene visual elements and the terminal display tasks through the display coupling, and then can accurately match the display content and task requirements. It should be explained that the display coupling represents the display correlation between the scene visual elements and the terminal display tasks, and the display streaming media is the display teaching data about the device display task in the current scene of the screen display. Furthermore, when the display coupling is high, the display streaming media corresponding to the device display task can be scheduled from the current scene of the screen display through a scheduling function, and the scheduling function is compiled in Java language.

[0095] In detail, the analyzing the display coupling between the visual elements of the scene and the terminal display task includes:

[0096] Performing content recognition on the terminal display task to obtain display task content;

[0097] Performing semantic analysis on the displayed task content to obtain task semantics;

[0098] Based on the task semantics, determining a task display requirement corresponding to the terminal display task;

[0099] Analyzing the element visual attributes corresponding to the visual elements of the scene;

[0100] Calculating a display coupling coefficient between the element visual attribute and the task display requirement;

[0101] The display coupling between the scene visual element and the terminal display task is analyzed according to the display coupling coefficient.

[0102] It should be explained that the display task content is the information content that the terminal display task is to present to the user, the task semantics is the meaning contained in the display task content, the task display requirement is the requirement for information display corresponding to the terminal display task, the element visual attribute is the characteristics of the scene visual element in terms of shape, color, size, position and other visual aspects, and the display coupling coefficient represents a quantitative value of the degree of fit between the element visual attribute and the task display requirement.

[0103] Furthermore, the content recognition of the terminal display task can be achieved through OCR recognition technology; the semantic parsing of the display task content can be achieved through semantic analysis; based on the task semantics, the task display requirements corresponding to the terminal display task are determined; the element visual attributes corresponding to the visual scene elements can be analyzed by a computer vision algorithm, and the computer vision algorithm includes a convolutional neural network; according to the numerical value of the display coupling coefficient, the display coupling between the visual scene elements and the terminal display task is analyzed, if the display coupling coefficient is between 0.8-1, it indicates that the display coupling is high, if the display coupling coefficient is between 0.5-0.8, it indicates that the display coupling is medium, and if the display coupling coefficient is below 0.5, it indicates that the display coupling is low.

[0104] Further, as an optional embodiment of the present invention, the calculating the display coupling coefficient between the element visual attribute and the task display requirement includes:

[0105] Vectorizing the element visual attributes and the task display requirements respectively to obtain a visual attribute vector and a display requirement vector;

[0106] Combining the visual attribute vector and the display requirement vector, the display matching degree between the element visual attribute and the task display requirement is calculated by the following formula:

[0107]

[0108] Among them, A represents the display matching degree between the element visual attributes and the task display requirements, and B a represents the vector corresponding to the visual attribute of the ath element in the visual attribute vector, a represents the sequence number of the element visual attribute, m represents the number of the element visual attributes, D b represents the vector corresponding to the b-th task display requirement in the display requirement vector, b represents the sequence number of the task display requirement, and n represents the number of task display requirements;

[0109] A display coupling coefficient between the element visual attribute and the task display requirement is calculated according to the display matching degree.

[0110] It should be explained that the visual attribute vector and the display requirement vector are respectively the quantitative representation forms corresponding to the element visual attributes and the task display requirements, and the display matching degree represents the display adaptation degree between the element visual attributes and the task display requirements. Furthermore, the vectorization processing of the element visual attributes and the task display requirements can be achieved through a support vector machine; the display matching degree is mapped to a range of 0-1, so as to obtain the display coupling coefficient between the element visual attributes and the task display requirements.

[0111] S3, analyzing the display key frames in the display streaming media, extracting dynamic visual features in the display key frames, and analyzing the interface display logic between the display streaming media according to the dynamic visual features.

[0112] The present invention can obtain key content information in the display streaming media by analyzing the display key frames in the display streaming media, reduce the data volume of the display streaming media, and improve the analysis accuracy of the subsequent interface display logic between the display streaming media. It should be explained that the display key frames are video frames corresponding to the key content information in the display streaming media.

[0113] In detail, analyzing the display key frames in the display streaming media includes:

[0114] Performing frame processing on the display streaming media to obtain a display frame set;

[0115] Performing optimal processing on the display frame set to obtain a target display frame;

[0116] Performing grayscale processing on the target display frame to obtain a grayscale display frame;

[0117] The texture complexity corresponding to the grayscale display sub-frame is calculated, and based on the texture complexity, the display key frame in the display streaming media is analyzed from the target display sub-frame.

[0118] It should be explained that the display frame set is a collection of all frames obtained by splitting the display streaming media in chronological order, the target display frame is a partial frame obtained after removing duplicate frames in the display frame set, the grayscale display frame is a frame after the target display frame is converted into a grayscale image after removing color information, and the texture complexity represents the complexity of the texture of the target display frame image.

[0119] Furthermore, the frame processing of the display streaming media can be achieved through a fixed time interval frame method; the similarity of each frame in the display frame set is calculated, and the display frame set is optimized according to the similarity to obtain a target display frame; the grayscale processing of the target display frame can be achieved through a simple channel weighted average method; based on the texture complexity, the frame corresponding to the maximum texture complexity value is selected from the target display frame as the display key frame in the display streaming media.

[0120] Further, as an optional embodiment of the present invention, the calculating the texture complexity corresponding to the grayscale display frame includes:

[0121] Measuring the frame grayscale level corresponding to the grayscale display frame and the number of grayscale levels corresponding thereto;

[0122] Traversing the grayscale pairs in the frame grayscales to construct a grayscale co-occurrence matrix corresponding to the grayscale display frame;

[0123] Based on the gray level co-occurrence matrix, calculating the gray level co-occurrence probability of the gray level pair;

[0124] Combining the frame grayscale, the number of grayscales and the grayscale co-occurrence probability, the texture complexity corresponding to the grayscale display frame is calculated by the following formula:

[0125]

[0126] Among them, C represents the texture complexity corresponding to the grayscale display frame, i and j represent the i grayscale level and j grayscale level in the frame grayscale, Q represents the number of grayscale levels, and E(i, j) represents the grayscale co-occurrence probability of the i grayscale level and the j grayscale level in the frame grayscale.

[0127] It should be explained that the frame grayscale represents the grayscale value corresponding to the grayscale display frame, the grayscale pair is a combination of different grayscale values ​​in the frame grayscale, the grayscale co-occurrence matrix is ​​a square matrix constructed by the grayscale values ​​in the grayscale display frame, and the grayscale co-occurrence probability represents the probability of occurrence of the grayscale pair.

[0128] Furthermore, the measurement of the frame grayscale corresponding to the grayscale display frame and the corresponding number of grayscale levels can be achieved through a histogram statistical method; the traversal of the grayscale pairs in the frame grayscale can be achieved through a nested loop traversal method; the construction of the grayscale co-occurrence matrix corresponding to the grayscale display frame can be achieved through a matrix function; the calculation of the grayscale co-occurrence probability of the grayscale pair can be obtained by dividing the number of occurrences by the total number of elements in the matrix.

[0129] By extracting the dynamic visual features in the display key frames, the present invention can obtain relevant information about the motion state of the subject in the display key frames, thereby facilitating the subsequent analysis and processing of the interface display logic between the display streaming media. It should be explained that the dynamic visual features are feature descriptions of the motion state of the object in the display key frames.

[0130] In detail, the extracting of dynamic visual features in the displayed key frame includes:

[0131] Performing subject detection on the displayed key frame to obtain the key frame subject;

[0132] Extracting subject feature points corresponding to the key frame subject, and identifying a display timestamp corresponding to the display key frame;

[0133] Tracking the motion trajectory between the subject feature points in combination with the display timestamp and the display key frame;

[0134] The trajectory motion parameters corresponding to the motion trajectory are calculated, and the motion trajectory and the trajectory motion parameters are combined to generate dynamic visual features in the display key frame.

[0135] It should be explained that the key frame subject is the main presentation object (such as a person, object, etc.) in the display key frame, the subject feature point is a point in the image corresponding to the key frame subject that is representative and can be used to track the location of the local feature of its movement, the display timestamp is the specific time mark corresponding to the display key frame that appears in the display streaming media, the motion trajectory is the spatial movement path formed by the change between the subject feature points over time, and the trajectory motion parameter is the relevant numerical index corresponding to the motion trajectory that can quantitatively describe its motion state (such as speed, acceleration, direction, etc.). Furthermore, the subject detection of the display key frame can be performed by a target detection algorithm based on deep learning (such as YOLO, Faster R-CNN, etc.); the main feature points corresponding to the key frame main body can be extracted by feature point extraction algorithm (such as Shi-Tomasi corner point detection, SIFT, etc.); the display timestamp corresponding to the display key frame can be identified by parsing video encoding information or according to the order and frame rate of key frames in the video stream; in combination with the display timestamp and the display key frame, the motion trajectory between the main feature points can be tracked by optical flow method (such as Lucas-Kanade optical flow algorithm, Farneback optical flow algorithm, etc.); the speed can be calculated by mathematical calculation method (calculating the speed, speed change and time according to the displacement and time interval) The method comprises the following steps: calculating the trajectory motion parameters corresponding to the motion trajectory by calculating the acceleration at intervals and analyzing the trajectory direction to determine the direction; combining the motion trajectory and the trajectory motion parameters to generate dynamic visual features in the displayed key frames. The motion trajectory can reflect the movement path of the teacher's hands when operating the ventilator components. The trajectory motion parameters include the speed and acceleration of the hand movement. By combining these motion trajectories and parameters, such as the trajectory and speed changes of the hands moving quickly and accurately between the ventilator adjustment knobs, dynamic visual features can be generated to characterize the visual dynamic information of key actions in the teaching process, so as to facilitate the subsequent analysis, retrieval or teaching effect evaluation of the teaching video.

[0136] The present invention analyzes the interface display logic between the display streaming media according to the dynamic visual features, and can accurately sort out the presentation context of the display streaming media content, such as the switching order and rhythm of the operation demonstration and explanation screen, which is helpful to improve the quality of teaching resources. It should be explained that the interface display logic is the content presentation order rule corresponding to the display streaming media. Furthermore, if the dynamic visual features show that the teacher's hands frequently move quickly in the heart area of ​​the model and there is a zooming action, it can be analyzed that the interface display logic is to focus on the hand movements and then zoom in on the local details when explaining the operation of the heart area, so as to highlight the key teaching content and enhance students' visual perception and understanding of knowledge points.

[0137] S4. Encode the display streaming media to obtain encoded streaming media, transmit the encoded streaming media to the projection display terminal based on the interface display logic to obtain streaming media to be displayed, and use the projection display terminal to perform teaching screen projection processing on the streaming media to be displayed to obtain a teaching screen projection result.

[0138] The present invention can effectively compress the amount of data by encoding the display streaming media, reduce the space and bandwidth required for subsequent storage and transmission, and transmit the encoded streaming media to the projection display terminal based on the interface display logic, which can ensure that the content of the encoded streaming media is accurately presented in a reasonable order and layout, such as teaching demonstrations can be carried out in an orderly manner, improve the accuracy of information transmission, and optimize the transmission and display processes, reduce freezes and errors, make the projection picture smooth and stable, and enhance the screen projection effect. It should be explained that the encoded streaming media is the streaming media obtained after the display streaming media is compressed, and the streaming media to be displayed is the streaming media obtained after the encoded streaming media is transmitted to the projection display terminal and subjected to a series of processing. Furthermore, the encoding processing of the display streaming media can be implemented by a video encoding standard algorithm, such as H.264 and other video encoding standard algorithms.

[0139] In detail, based on the interface display logic, the encoded streaming media is transmitted to the projection display terminal to obtain the streaming media to be displayed, including:

[0140] Extracting display meta parameters corresponding to the encoded streaming media;

[0141] Combining the interface display logic and the display meta-parameters, encapsulating the encoded streaming media to obtain encapsulated streaming media;

[0142] Respectively query the wireless protocols corresponding to the update embedded device and the projection display terminal to obtain the device wireless protocol and the terminal wireless protocol;

[0143] Combining the device wireless protocol and the terminal wireless protocol, constructing a wireless transmission trunk between the update embedded device and the projection display terminal;

[0144] The encoded streaming media is transmitted to the projection display terminal by using the wireless transmission trunk channel to obtain the streaming media to be displayed.

[0145] It should be explained that the display meta-parameters are the basic information corresponding to the encoded streaming media for describing its own display-related characteristics (such as resolution, frame rate, color space, sampling rate, etc.); the encapsulated streaming media is the streaming media format that is obtained by encapsulating and integrating the encoded streaming media according to a certain interface display logic and in combination with the display meta-parameters to facilitate subsequent transmission and playback; the device wireless protocol and the terminal wireless protocol are the relevant communication protocols (such as Wi-Fi protocol, Bluetooth protocol, etc.) corresponding to the updated embedded device and the projection display terminal for realizing wireless communication and stipulating wireless transmission rules and methods; the wireless transmission trunk is a channel between the updated embedded device and the projection display terminal that is constructed with the help of corresponding wireless protocols to realize wireless data transmission.

[0146] Furthermore, the display meta-parameters corresponding to the encoded streaming media can be extracted through professional streaming media analysis software (such as MediaInfo, etc.); in combination with the interface display logic and the display meta-parameters, the encoded streaming media can be encapsulated and processed through a special streaming media encapsulation tool (such as FFmpeg, etc.) to obtain encapsulated streaming media; the wireless protocols corresponding to the updated embedded device and the projection display terminal can be obtained by querying the corresponding device official website; in combination with the device wireless protocol and the terminal wireless protocol, a wireless transmission trunk between the updated embedded device and the projection display terminal can be constructed through a wireless connection configuration wizard or a corresponding network programming interface (such as a network connection API based on a specific operating system).

[0147] The present invention utilizes the projection display terminal to perform teaching screen projection processing on the streaming media to be displayed, and then obtains a visual teaching picture, thereby achieving the large-scale demonstration purpose of the embedded device. Furthermore, the streaming media to be displayed can be decoded, signal converted, resolution adapted, color corrected, etc. by the projection display terminal, and finally the processed streaming media can be subjected to teaching screen projection processing.

[0148] Compared with the problems described in the background technology, the present invention performs display update processing on the display screen of the embedded device according to a preset embedded program to obtain an updated embedded device, which can improve the accuracy and timeliness of the displayed content and enhance the visualization effect and interactive performance of the operation of the embedded device. The present invention can obtain the visually perceived components of the current visual scene of the screen by extracting the visual elements of the visual scene in the current visual scene of the screen, thereby laying a foundation for the subsequent analysis of the display coupling between the visual elements of the visual scene and the terminal display task. The present invention can obtain the key content information in the display streaming media by analyzing the display key frames in the display streaming media, reduce the data volume of the display streaming media, and improve the analysis accuracy of the interface display logic between the subsequent display streaming media. The present invention can effectively compress the data volume by encoding the display streaming media, reduce the space and bandwidth required for subsequent storage and transmission, and transmit the encoded streaming media to the projection display terminal based on the interface display logic, which can ensure that the content of the encoded streaming media is accurately presented in a reasonable order and layout, such as the teaching demonstration can be carried out in an orderly manner, improve the accuracy of information transmission, and optimize the transmission and display process at the same time, reduce freezes and errors, make the projection picture smooth and stable, and enhance the interactive effect of screen projection. Therefore, the screen projection method of the embedded device proposed in the present invention improves the screen projection interactive effect of the embedded device.

[0149] Embodiment 2:

[0150] like Figure 2 The figure shows a functional module diagram of a screen projection system for an embedded device provided by an embodiment of the present invention.

[0151] The screen projection system 100 of an embedded device described in the present invention can be installed in an electronic device. According to the functions implemented, the screen projection system 100 of an embedded device may include a visual acquisition module 101, a display data scheduling module 102, a display logic analysis module 103 and a teaching screen projection module 104. The module described in the present invention can also be referred to as a unit, which refers to a series of computer program segments that can be executed by an electronic device processor and can complete fixed functions, which are stored in the memory of the electronic device.

[0152] In this embodiment, the functions of each module / unit are as follows:

[0153] The view acquisition module 101 is used to obtain the embedded device and projection display terminal to be processed, schedule the terminal display task corresponding to the projection display terminal, perform display update processing on the display screen of the embedded device according to the preset embedded program, obtain the updated embedded device, and acquire the current screen display view corresponding to the updated embedded device;

[0154] The display data scheduling module 102 is used to extract the visual elements of the scene in the current scene displayed on the screen, analyze the display coupling between the visual elements of the scene and the terminal display task, and schedule the display streaming media corresponding to the device display task from the current scene displayed on the screen based on the display coupling;

[0155] The display logic analysis module 103 is used to analyze the display key frames in the display streaming media, extract the dynamic visual features in the display key frames, and analyze the interface display logic between the display streaming media according to the dynamic visual features;

[0156] The teaching screen projection module 104 is used to encode the display streaming media to obtain encoded streaming media, transmit the encoded streaming media to the projection display terminal based on the interface display logic to obtain the streaming media to be displayed, and use the projection display terminal to perform teaching screen projection on the streaming media to be displayed to obtain a teaching screen projection result.

[0157] In detail, each module described in the screen projection system 100 of an embedded device described in the embodiment of the present application is used in the same manner as described above. Figure 1 The same technical means are used as the screen projection method of an embedded device described in , and can produce the same technical effects, so I will not go into details here.

[0158] It is obvious to those skilled in the art that the present invention is not limited to the details of the above exemplary embodiments, and that the present invention can be implemented in other specific forms without departing from the spirit or essential characteristics of the present invention.

[0159] Finally, it should be noted that the above embodiments are only used to illustrate the technical solution of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solution of the present invention can be modified or replaced by equivalents without departing from the spirit and scope of the technical solution of the present invention.

Claims

1. A screen projection method for an embedded device, characterized in that: The method comprises: Acquire the embedded device and projection display terminal to be processed, schedule the terminal display task corresponding to the projection display terminal, perform display update processing on the display screen of the embedded device according to the preset embedded program, obtain the updated embedded device, and collect the current screen display scene corresponding to the updated embedded device; Extracting visual elements of the scene in the current scene displayed on the screen, analyzing the display coupling between the visual elements of the scene and the terminal display task, and scheduling display streaming media corresponding to the device display task from the current scene displayed on the screen based on the display coupling; Analyze the display key frames in the display streaming media, extract the dynamic visual features in the display key frames, and analyze the interface display logic between the display streaming media according to the dynamic visual features; The display streaming media is encoded to obtain encoded streaming media, based on the interface display logic, the encoded streaming media is transmitted to the projection display terminal to obtain streaming media to be displayed, and the projection display terminal is used to perform teaching screen projection processing on the streaming media to be displayed to obtain a teaching screen projection result.

2. The screen projection method of an embedded device as claimed in claim 1, characterized in that: The method of performing display update processing on the display screen of the embedded device according to the preset embedded program to obtain an updated embedded device includes: Performing code parsing processing on the embedded program to obtain a program instruction sequence; Sorting and optimizing the program instruction sequence to obtain an optimized instruction set; Performing memory allocation processing on the optimized instruction set to obtain executable instruction data; Performing data loading processing on the executable instruction data to obtain loading data; Performing interface element construction processing on the loaded data to obtain display data; Performing rendering and adaptation processing on the display data to obtain adapted image data for the display screen of the embedded device; The adapted image data is subjected to display driving processing to obtain an updated embedded device.

3. The screen projection method of an embedded device as claimed in claim 1, characterized in that: The extracting of the visual elements of the scene in the current scene displayed on the screen includes: Performing image noise reduction processing on the current visual scene displayed on the screen to obtain a noise-reduced visual scene image; Performing element detection on the denoised visual image to obtain visual image elements; Performing element clustering processing on the visual image elements to obtain clustered visual elements; Calculating the element importance corresponding to each element in the clustered visual elements; Based on the element importance, the visual elements of the scene in the current scene displayed on the screen are extracted from the clustered scene elements.

4. The screen projection method of an embedded device as claimed in claim 1, characterized in that: The analyzing the display coupling between the visual elements of the scene and the terminal display task includes: Performing content recognition on the terminal display task to obtain display task content; Performing semantic analysis on the displayed task content to obtain task semantics; Based on the task semantics, determining a task display requirement corresponding to the terminal display task; Analyzing the element visual attributes corresponding to the visual elements of the scene; Calculating a display coupling coefficient between the element visual attribute and the task display requirement; The display coupling between the scene visual element and the terminal display task is analyzed according to the display coupling coefficient.

5. The screen projection method of an embedded device as claimed in claim 4, characterized in that: The calculating the display coupling coefficient between the element visual attribute and the task display requirement includes: Vectorizing the element visual attributes and the task display requirements respectively to obtain a visual attribute vector and a display requirement vector; Combining the visual attribute vector and the display requirement vector, the display matching degree between the element visual attribute and the task display requirement is calculated by the following formula: Among them, A represents the display matching degree between the element visual attributes and the task display requirements, and B a represents the vector corresponding to the visual attribute of the ath element in the visual attribute vector, a represents the sequence number of the element visual attribute, m represents the number of the element visual attributes, D b represents the vector corresponding to the b-th task display requirement in the display requirement vector, b represents the sequence number of the task display requirement, and n represents the number of task display requirements; A display coupling coefficient between the element visual attribute and the task display requirement is calculated according to the display matching degree.

6. The screen projection method of an embedded device as claimed in claim 1, characterized in that: The step of analyzing the display key frames in the display streaming media includes: Performing frame processing on the display streaming media to obtain a display frame set; Performing optimal processing on the display frame set to obtain a target display frame; Performing grayscale processing on the target display frame to obtain a grayscale display frame; The texture complexity corresponding to the grayscale display sub-frame is calculated, and based on the texture complexity, the display key frame in the display streaming media is analyzed from the target display sub-frame.

7. The screen projection method of an embedded device as claimed in claim 6, characterized in that: The calculating the texture complexity corresponding to the grayscale display frame includes: Measuring the frame grayscale level corresponding to the grayscale display frame and the number of grayscale levels corresponding thereto; Traversing the grayscale pairs in the frame grayscales to construct a grayscale co-occurrence matrix corresponding to the grayscale display frame; Based on the gray level co-occurrence matrix, calculating the gray level co-occurrence probability of the gray level pair; Combining the frame grayscale, the number of grayscales and the grayscale co-occurrence probability, the texture complexity corresponding to the grayscale display frame is calculated by the following formula: Among them, C represents the texture complexity corresponding to the grayscale display frame, i and j represent the i grayscale level and j grayscale level in the frame grayscale, Q represents the number of grayscale levels, and E(i, j) represents the grayscale co-occurrence probability of the i grayscale level and the j grayscale level in the frame grayscale.

8. The screen projection method of an embedded device as claimed in claim 1, characterized in that: The extracting of dynamic visual features in the displayed key frame comprises: Performing subject detection on the displayed key frame to obtain the key frame subject; Extracting subject feature points corresponding to the key frame subject, and identifying a display timestamp corresponding to the display key frame; Tracking the motion trajectory between the subject feature points in combination with the display timestamp and the display key frame; The trajectory motion parameters corresponding to the motion trajectory are calculated, and the motion trajectory and the trajectory motion parameters are combined to generate dynamic visual features in the display key frame.

9. The screen projection method of an embedded device as claimed in claim 1, characterized in that: The step of transmitting the encoded streaming media to the projection display terminal based on the interface display logic to obtain the streaming media to be displayed includes: Extracting display meta parameters corresponding to the encoded streaming media; Combining the interface display logic and the display meta-parameters, encapsulating the encoded streaming media to obtain encapsulated streaming media; Respectively query the wireless protocols corresponding to the update embedded device and the projection display terminal to obtain the device wireless protocol and the terminal wireless protocol; Combining the device wireless protocol and the terminal wireless protocol, constructing a wireless transmission trunk between the update embedded device and the projection display terminal; The encoded streaming media is transmitted to the projection display terminal by using the wireless transmission trunk channel to obtain the streaming media to be displayed.

10. A screen projection system for an embedded device, characterized in that: The system comprises: A visual acquisition module is used to obtain the embedded device and projection display terminal to be processed, schedule the terminal display task corresponding to the projection display terminal, perform display update processing on the display screen of the embedded device according to a preset embedded program, obtain an updated embedded device, and acquire the current screen display visual scene corresponding to the updated embedded device; A display data scheduling module, used for extracting visual elements of the scene in the current scene displayed on the screen, analyzing the display coupling between the visual elements of the scene and the terminal display task, and scheduling the display streaming media corresponding to the device display task from the current scene displayed on the screen based on the display coupling; A display logic analysis module, used to analyze the display key frames in the display streaming media, extract the dynamic visual features in the display key frames, and analyze the interface display logic between the display streaming media according to the dynamic visual features; The teaching screen projection module is used to encode the display streaming media to obtain encoded streaming media, transmit the encoded streaming media to the projection display terminal based on the interface display logic to obtain the streaming media to be displayed, and use the projection display terminal to perform teaching screen projection processing on the streaming media to be displayed to obtain the teaching screen projection result.

Images