Touch all-in-one machine interaction control method and device, computer equipment and medium

Abstract

The invention relates to a touch all-in-one machine interaction control method and device, computer equipment and a medium, and the method comprises the steps: receiving image data through a preset video stream protocol, and obtaining a corresponding image data stream; acquiring a real-time network bandwidth condition, and then performing adaptive compression processing on the image data stream based on the real-time network bandwidth condition, and dynamically adjusting image transmission parameters; comprehensively analyzing the image data stream by using a deep learning algorithm, extracting operation characteristics of a user, generating an output control instruction based on the operation characteristics, and further controlling and adjusting image display and / or an equipment state; content information is extracted from the image data stream, and content analysis is carried out by adopting a corresponding content analysis strategy according to the content information, so that real-time display and / or storage are / is carried out according to the analyzed content; and obtaining a user interaction behavior, generating a corresponding adjustment feedback instruction based on the user interaction behavior, and performing personalized adjustment on the user interface. The method has the effect of improving the image transmission quality.

Description

Technical Field

[0001] The present application relates to the technical field of image processing, and in particular to a touch-based integrated machine interactive control method, device, computer equipment and medium. Background Art

[0002] At present, with the popularity of smart devices, touch all-in-one computers, as one of the user interaction interfaces, are widely used in various devices, such as smart TVs, interactive display screens, etc. These devices process and display image data, and control the functions of the devices through touch and voice commands input by users.

[0003] Existing touch all-in-one computers usually process image data through static or preset compression strategies, and respond interactively based on simple user input. They are unable to dynamically adjust image quality according to real-time network conditions or provide personalized feedback based on different user operations. The analysis strategies for image content are often relatively simple and lack intelligence and flexibility, resulting in difficulty in optimizing image transmission quality in complex or bandwidth unstable network environments.

[0004] The above-mentioned existing technical solutions have the following defects: the existing touch all-in-one machines lack real-time adaptive adjustment to network bandwidth fluctuations, and have poor feedback adjustment capabilities for user operations, resulting in uneven quality of video transmission, so there is room for improvement. Summary of the invention

[0005] In order to improve the quality of image transmission, the present application provides a touch all-in-one machine interactive control method, device, computer equipment and medium.

[0006] The above-mentioned invention objective of the present application is achieved through the following technical solutions: A touch all-in-one machine interactive control method, the touch all-in-one machine interactive control method comprising: Receive image data through a preset video stream protocol to obtain a corresponding image data stream; Acquire real-time network bandwidth status, and then perform adaptive compression processing on the image data stream based on the real-time network bandwidth status, and dynamically adjust image transmission parameters; Using a deep learning algorithm to comprehensively analyze the image data stream, extract the user's operation characteristics, and generate output control instructions based on the operation characteristics, thereby controlling and adjusting the image display and / or device status to optimize the user experience; Extracting content information from the image data stream, and performing content analysis using a corresponding content analysis strategy according to the content information, so as to perform real-time display and / or storage according to the analyzed content; The user interaction behavior is acquired, and corresponding adjustment feedback instructions are generated based on the user interaction behavior, so as to perform personalized adjustment on the user interface through an adaptive feedback mechanism.

[0007] By adopting the above technical solution, image data is received through a preset video stream protocol, which can ensure real-time reception and transmission of image data, avoid delays and data loss, and thus improve the real-time and stability of the image processing system; by obtaining the real-time network bandwidth status and adaptively compressing the image data stream based on the real-time network bandwidth status, the image transmission parameters can be dynamically adjusted to adapt to changes in network bandwidth, ensure the smoothness of image transmission, and reduce freezes and delays; by using a deep learning algorithm to comprehensively analyze the image data stream and extract user operation features, the user's intention can be accurately identified and quickly responded to, thereby optimizing the user experience; by extracting content information from the image data stream and parsing it, the key information in the image can be obtained in real time and displayed or stored, thereby improving the information processing capability of the device and the flexibility of application; by obtaining user interaction behavior and generating adjustment feedback instructions, the interface settings can be adjusted according to the user's interaction method, thereby improving the device's personalized adjustment capability and the convenience of user interaction.

[0008] In one example, the present application may be further configured as follows: the adaptive compression processing of the image data stream based on the real-time network bandwidth status and the dynamic adjustment of the image transmission parameters specifically include: Performing hierarchical forecasting on the real-time fluctuation of the network bandwidth status to obtain short-term fluctuation trends; Based on the short-term fluctuation trend, a corresponding image compression strategy combination is dynamically switched during the image data transmission process, wherein the image compression strategy includes a resolution degradation strategy and a frame rate adjustment strategy.

[0009] By adopting the above technical solution and making hierarchical predictions on the real-time fluctuations of network bandwidth conditions, it is possible to capture the changing trend of network bandwidth in advance and make appropriate bandwidth adjustments, thereby reducing the impact of unstable bandwidth on image transmission quality; by dynamically switching the image compression strategy combination based on short-term fluctuation trends, it is possible to automatically adjust the image resolution and frame rate according to the actual situation of bandwidth fluctuations, optimize image quality and transmission efficiency, and ensure a good user experience when bandwidth is limited.

[0010] In one example, the present application may be further configured as follows: performing hierarchical prediction on the real-time fluctuation of the network bandwidth status to obtain a short-term fluctuation trend specifically includes: Continuously sampling the real-time network bandwidth status, and performing layered processing on the sampled data according to the time window; A multi-level filtering algorithm is used, and different weights are used to perform weighted smoothing on each layer of data based on the amplitude and change rate of bandwidth fluctuations; By analyzing the smoothed data, the short-term fluctuation trend is extracted.

[0011] By adopting the above technical solution, by continuously sampling the real-time network bandwidth status and layering the sampled data according to the time window, it is possible to perform a more accurate analysis of network bandwidth fluctuations and identify fluctuation trends at different time scales, thereby providing more accurate data support for the adjustment of image transmission strategies; by adopting a multi-level filtering algorithm and weighted smoothing each layer of data based on the amplitude and change rate of bandwidth fluctuations, it is possible to reduce noise interference and make the prediction of bandwidth fluctuation trends more stable and reliable, thereby improving the adaptability and stability of image data streams; by analyzing the smoothed data to extract short-term fluctuation trends, it is possible to more accurately capture the instantaneous changes in bandwidth, provide real-time response for the adjustment of image compression strategies, and effectively avoid the impact of bandwidth fluctuations.

[0012] In one example, the present application may be further configured as follows: the dynamically switching corresponding image compression strategy combination during image data transmission based on the short-term fluctuation trend specifically includes: Performing amplitude judgment on the short-term fluctuation trend to determine the image compression strategy according to the corresponding fluctuation amplitude; In the case where the fluctuation amplitude exceeds the preset stable fluctuation range, based on the bandwidth consumption of the image data stream, the resolution of the corresponding areas is reduced from large to small, and the transmission of redundant frames is reduced through inter-frame difference analysis; When the fluctuation amplitude is within a preset stable fluctuation range, the corresponding frame rate and resolution are increased in sequence from high to low based on the complexity of the image data stream.

[0013] By adopting the above technical solution, by judging the amplitude of short-term fluctuation trends, it is possible to judge the intensity of bandwidth fluctuations, and adjust the image compression strategy according to the fluctuation amplitude, thereby reducing the image data flow when the bandwidth fluctuation is large and ensuring stable transmission; by reducing the resolution of the corresponding area from large to small based on the bandwidth consumption of the image data stream when the fluctuation amplitude exceeds the preset stable fluctuation range, and reducing the transmission of redundant frames through frame-to-frame difference analysis, it is possible to reduce bandwidth pressure and improve the efficiency of image transmission, and optimize the use of bandwidth; by increasing the corresponding frame rate and resolution from high to low based on the complexity of the image data stream when the fluctuation amplitude is within the preset stable fluctuation range, it is possible to improve the image quality when the bandwidth is stable and optimize the user's visual experience.

[0014] In one example, the present application may be further configured as follows: the use of a deep learning algorithm to comprehensively analyze the image data stream and extract user operation features specifically includes: Processing the key frames of the image data stream through a multi-scale feature extraction network to generate a feature map; Extracting action key points from the feature graph, and then constructing a temporal feature vector of the user operation; Based on matching the timing feature vector with a preset operation feature model, a control instruction is generated.

[0015] By adopting the above technical solution, the key frames of the image data stream are processed through a multi-scale feature extraction network, which can extract multi-level information of the image, capture detailed features and motion information, and thus provide more accurate input data for subsequent user operation recognition; by extracting action key points in the feature map, the key actions in the user operation can be accurately located, and a time series feature vector can be constructed. Based on this, the user's intention is analyzed to ensure the accuracy of the control instructions; by matching the time series feature vector with the preset operation feature model, the user's operation requirements can be efficiently identified and the corresponding control instructions can be generated, thereby improving the system response speed and the accuracy of the interactive experience.

[0016] In one example, the present application may be further configured as follows: performing content parsing using a corresponding content parsing strategy according to the content information specifically includes: Identifying the type of the content information, wherein the type of the content information includes text information and encoding information; For the text information, extract the content of the text information using a context-dependent multilingual character recognition model; The coded information is located by combining the color distribution characteristics and position characteristics of the image area, and the barcode / QR code recognition technology is used to decode the content of the coded information.

[0017] By adopting the above technical solution, by identifying the type of content information, the text information and coded information in the image can be accurately identified, providing basic data support for subsequent content analysis; by using a context-related multilingual character recognition model to extract text information, the recognition accuracy can be improved, especially the processing capabilities of multiple languages ​​or complex characters, and the versatility of the device can be enhanced; by combining the color distribution characteristics and position characteristics of the image area for positioning and using barcode / QR code recognition technology to decode the coded information, the coded data in the image can be efficiently processed, and support for subsequent data storage or display can be provided.

[0018] In one example, the present application may be further configured as follows: generating a corresponding adjustment feedback instruction based on the user interaction behavior to personalize the user interface through an adaptive feedback mechanism, specifically including: Classifying the user interaction behavior into behavior patterns, wherein the behavior patterns include touch behavior and voice commands; For the touch behavior, a behavior sequence is generated by recording the touch position, force and duration, and a preset user interface adjustment strategy is matched according to the behavior sequence; For the voice command, the feedback content is dynamically adjusted in combination with the real-time voice recognition result and the user's current interface status.

[0019] By adopting the above technical solution, by classifying the behavior patterns of user interaction behaviors, different user interaction methods, such as touch behaviors and voice commands, can be distinguished, thereby providing a clear input type for subsequent interface adjustments; by recording the touch position, force and duration of the touch behavior to generate a behavior sequence, the interface layout can be dynamically adjusted according to the user's touch habits to optimize the touch experience; by combining real-time voice recognition results and the user's current interface status to dynamically adjust the feedback content, it is possible to achieve interface interaction and personalized adjustment based on voice commands, thereby improving user experience and interaction flexibility.

[0020] The second object of the invention is achieved by the following technical solutions: A touch all-in-one machine interactive control device, the touch all-in-one machine interactive control device comprising: An image receiving module is used to receive image data through a preset video stream protocol to obtain a corresponding image data stream; A network bandwidth monitoring module, used to obtain real-time network bandwidth status, and then perform adaptive compression processing on the image data stream based on the real-time network bandwidth status, and dynamically adjust image transmission parameters; An operation feature extraction module is used to perform a comprehensive analysis on the image data stream using a deep learning algorithm, extract the user's operation features, and generate output control instructions based on the operation features, thereby controlling and adjusting the image display and / or device status to optimize the user experience; A content information extraction module, used to extract content information from the image data stream, and perform content analysis using a corresponding content analysis strategy according to the content information, so as to perform real-time display and / or storage according to the analyzed content; The user interaction behavior monitoring module is used to obtain user interaction behaviors and generate corresponding adjustment feedback instructions based on the user interaction behaviors, so as to personalize the user interface through an adaptive feedback mechanism.

[0021] By adopting the above technical solution, image data is received through a preset video stream protocol, which can ensure real-time reception and transmission of image data, avoid delays and data loss, and thus improve the real-time and stability of the image processing system; by obtaining the real-time network bandwidth status and adaptively compressing the image data stream based on the real-time network bandwidth status, the image transmission parameters can be dynamically adjusted to adapt to changes in network bandwidth, ensure the smoothness of image transmission, and reduce freezes and delays; by using a deep learning algorithm to comprehensively analyze the image data stream and extract user operation features, the user's intention can be accurately identified and quickly responded to, thereby optimizing the user experience; by extracting content information from the image data stream and parsing it, the key information in the image can be obtained in real time and displayed or stored, thereby improving the information processing capability of the device and the flexibility of application; by obtaining user interaction behavior and generating adjustment feedback instructions, the interface settings can be adjusted according to the user's interaction method, thereby improving the device's personalized adjustment capability and the convenience of user interaction.

[0022] The third objective of the present application is achieved through the following technical solutions: A computer device comprises a memory, a processor and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the above-mentioned touch integrated machine interactive control method when executing the computer program.

[0023] The fourth objective of the present application is achieved through the following technical solutions: A computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the above-mentioned touch-integrated machine interactive control method are implemented.

[0024] In summary, this application includes the following beneficial technical effects: 1. Receiving image data through a preset video streaming protocol can ensure real-time reception and transmission of image data, avoid delays and data loss, and thus improve the real-time and stability of the image processing system; by obtaining real-time network bandwidth conditions and adaptively compressing the image data stream based on the real-time network bandwidth conditions, the image transmission parameters can be dynamically adjusted to adapt to changes in network bandwidth, ensure the smoothness of image transmission, and reduce freezes and delays; by using deep learning algorithms to comprehensively analyze image data streams and extract user operation features, the user's intentions can be accurately identified and quickly responded to, thereby optimizing the user experience; by extracting content information from the image data stream and parsing it, the key information in the image can be obtained in real time and displayed or stored, thereby improving the information processing capability of the device and the flexibility of application; by obtaining user interaction behaviors and generating adjustment feedback instructions, the interface settings can be adjusted according to the user's interaction method, thereby improving the device's personalized adjustment capabilities and the convenience of user interaction; 2. By continuously sampling the real-time network bandwidth status and processing the sampled data in layers according to the time window, it is possible to analyze the network bandwidth fluctuation more accurately and identify the fluctuation trends of different time scales, thereby providing more accurate data support for the adjustment of image transmission strategies; by adopting a multi-level filtering algorithm and weighted smoothing each layer of data based on the amplitude and change rate of bandwidth fluctuation, it is possible to reduce noise interference and make the prediction of bandwidth fluctuation trends more stable and reliable, thereby improving the adaptability and stability of image data streams; by analyzing the smoothed data to extract short-term fluctuation trends, it is possible to more accurately capture the instantaneous changes in bandwidth, provide real-time response for the adjustment of image compression strategies, and effectively avoid the impact of bandwidth fluctuations; 3. By judging the amplitude of short-term fluctuation trends, the intensity of bandwidth fluctuations can be judged, and the image compression strategy can be adjusted according to the fluctuation amplitude, thereby reducing image data traffic when the bandwidth fluctuation is large and ensuring stable transmission; by reducing the resolution of the corresponding area from large to small based on the bandwidth consumption of the image data stream when the fluctuation amplitude exceeds the preset stable fluctuation range, and reducing the transmission of redundant frames through inter-frame difference analysis, it is possible to reduce bandwidth pressure and improve image transmission efficiency and optimize bandwidth usage; by increasing the corresponding frame rate and resolution from high to low based on the complexity of the image data stream when the fluctuation amplitude is within the preset stable fluctuation range, it is possible to improve image quality when the bandwidth is stable and optimize the user's visual experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] Figure 1 is a flow chart of a touch integrated machine interactive control method in an embodiment of the present application; Figure 2 It is a flowchart for implementing step S20 in the touch integrated machine interactive control method in one embodiment of the present application; Figure 3 It is a flowchart for implementing step S21 in the touch all-in-one machine interactive control method in one embodiment of the present application; Figure 4 It is a flowchart for implementing step S22 in the touch integrated machine interactive control method in one embodiment of the present application; Figure 5 It is a flowchart for implementing step S30 in the touch integrated machine interactive control method in one embodiment of the present application; Figure 6 It is a flowchart for implementing step S40 in the touch integrated machine interactive control method in one embodiment of the present application; Figure 7 is a flowchart for implementing step S50 in the touch all-in-one machine interactive control method in one embodiment of the present application; Figure 8This is a principle block diagram of a touch integrated machine interactive control device in an embodiment of the present application; Fig. 9 It is a schematic diagram of a device in an embodiment of the present application. DETAILED DESCRIPTION

[0026] The present application is further described in detail below in conjunction with the accompanying drawings.

[0027] In one embodiment, if Figure 1 As shown, the present application discloses a touch integrated machine interactive control method, which specifically includes the following steps: S10: receiving image data through a preset video stream protocol to obtain a corresponding image data stream.

[0028] Specifically, image data is transmitted from the source device to the target device through a preset video streaming protocol. The image data stream usually contains frame-by-frame data of the image. The video streaming protocol used may be common protocols such as RTSP and HLS. These protocols perform real-time transmission of image data in a standardized format. The image data stream is transmitted through the network to ensure real-time and stability. The protocol used in the transmission process will handle issues such as image compression and frame synchronization to ensure that each frame of image data can be received on time on the target device for subsequent processing.

[0029] S20: Acquire the real-time network bandwidth status, and then perform adaptive compression processing on the image data stream based on the real-time network bandwidth status, and dynamically adjust the image transmission parameters.

[0030] Specifically, the real-time network bandwidth status is collected through the network bandwidth monitoring tool, which regularly detects key performance indicators such as the network transmission rate, delay, packet loss rate, etc. According to the changes in network bandwidth, an adaptive compression algorithm is used to adjust the compression ratio and image quality parameters of the image data stream. When the network bandwidth is sufficient, a higher image quality and a higher frame rate are retained. When the bandwidth is narrow, the data flow is reduced by reducing the resolution and frame rate, thereby ensuring stable image transmission and reducing the freeze and delay caused by bandwidth limitations.

[0031] S30: Use a deep learning algorithm to comprehensively analyze the image data stream, extract the user's operation characteristics, and generate output control instructions based on the operation characteristics, thereby controlling and adjusting the image display and / or device status to optimize the user experience.

[0032] Specifically, each frame in the image data stream is analyzed through a trained deep learning model, such as a convolutional neural network, to extract the user's operation features, such as gestures, facial expressions, etc. The network model can recognize the user's intention according to different input scenarios and generate control instructions. These instructions can be used to adjust the display effect of the image, such as zooming in, zooming out, rotating or switching screen modes, or to achieve the user's intended operations through device control instructions, such as starting an application or adjusting the volume, thereby optimizing the user's interactive experience.

[0033] S40: extracting content information from the image data stream, and performing content analysis using a corresponding content analysis strategy according to the content information, so as to perform real-time display and / or storage according to the analyzed content.

[0034] Specifically, the content information in the image data stream includes multiple data forms such as text, images or QR codes. The text information in the image is extracted through OCR technology, and the barcode or QR code recognition technology is used to decode the encoded information therein. According to the different types of information, the corresponding parsing strategy is used for processing. For example, when text information is extracted, the context-related multilingual character recognition model is used to improve the recognition accuracy; and for QR codes, the regional positioning and color features of the image are combined for efficient decoding. The parsed information can be displayed in real time on the device, or stored in the database for subsequent operations as needed.

[0035] S50: Acquire user interaction behavior, and generate corresponding adjustment feedback instructions based on the user interaction behavior, so as to perform personalized adjustment on the user interface through an adaptive feedback mechanism.

[0036] Specifically, user interaction behavior is obtained through touch screens, voice recognition and other input devices. When a user performs a touch operation, the system will record data such as touch position, touch force and touch duration; and voice commands are converted into text in real time through voice recognition technology, combined with the user's current operating interface status, to analyze the user's interaction needs, and generate adjustment feedback instructions based on these data. These feedback instructions will dynamically adjust the layout, function settings or display mode of the user interface according to the preset user interface adjustment strategy, so as to better match the user's operating habits and needs and improve interaction efficiency.

[0037] In one embodiment, if Figure 2 As shown, in step S20, adaptive compression processing is performed on the image data stream based on the real-time network bandwidth status, and image transmission parameters are dynamically adjusted, specifically including: S21: Perform hierarchical prediction on the real-time fluctuation of network bandwidth status to obtain short-term fluctuation trends.

[0038] Specifically, the bandwidth data collected in real time is processed in layers through time windows to analyze the short-term fluctuation trend of bandwidth. The layered processing method is to divide the collected bandwidth data into multiple levels according to different time scales, and calculate the bandwidth fluctuation situation at each level to identify the fluctuation characteristics at different levels. Through this layered processing, the trend of bandwidth fluctuation can be captured more accurately, avoiding the problem that the fluctuation information of a single time scale cannot fully reflect the bandwidth change, thereby providing more accurate prediction data for subsequent compression strategies.

[0039] S22: Dynamically switch a corresponding image compression strategy combination during image data transmission based on a short-term fluctuation trend, wherein the image compression strategy includes a resolution degradation strategy and a frame rate adjustment strategy.

[0040] Specifically, based on the forecast data of short-term fluctuation trends, the system will determine the amplitude of bandwidth fluctuations and dynamically switch different combinations of image compression strategies according to the amplitude of fluctuations. When the fluctuation amplitude is large, the image resolution is reduced and unnecessary frame transmission is reduced. By analyzing the differences between frames, the transmission of redundant information is reduced; when the bandwidth fluctuation is within a relatively stable range, the image resolution and frame rate are increased, the image quality is improved, and the image transmission rate is adjusted according to the bandwidth consumption to ensure the image clarity while improving the smoothness of the video stream.

[0041] In one embodiment, if Figure 3 As shown, in step S21, the real-time fluctuation of the network bandwidth status is predicted in layers to obtain the short-term fluctuation trend, which specifically includes: S211: Continuously sample the real-time network bandwidth status and perform layered processing on the sampled data according to the time window.

[0042] Specifically, the real-time network bandwidth status is continuously sampled through the network monitoring tool. The system will periodically record the network bandwidth data and process the data in layers according to the set time window. The data in each time window will be layered and analyzed according to different time granularities such as seconds and minutes, so as to capture the periodic characteristics and short-term fluctuation trends of bandwidth fluctuations. In this way, the changes in bandwidth fluctuations can be effectively predicted.

[0043] S212: A multi-level filtering algorithm is used, and different weights are used to perform weighted smoothing on each layer of data based on the amplitude and change rate of bandwidth fluctuation.

[0044] Specifically, a multi-layer filtering algorithm is used to smooth the bandwidth fluctuation data of each layer. During the processing, the weights of different layers are adjusted according to the amplitude and change rate of bandwidth fluctuations to better reflect the changing trend of real-time bandwidth. Layers with larger bandwidth fluctuation amplitudes will receive higher weights, making them more sensitive to short-term fluctuations, while layers with smaller change rates will receive lower weights to avoid affecting the accuracy of the overall fluctuation trend.

[0045] S213: Extract short-term fluctuation trends by analyzing the smoothed data.

[0046] Specifically, the filtered data will be analyzed for fluctuation trends to extract short-term fluctuation trends. The system will identify the short-term trend of bandwidth changes by curve fitting the data, and further analyze the short-term change direction of bandwidth to provide a basis for the subsequent selection of image compression strategies, ensuring that the image transmission process can adapt to bandwidth changes in real time, thereby optimizing image quality and data transmission efficiency.

[0047] In one embodiment, if Figure 4 As shown, in step S22, the corresponding image compression strategy combination is dynamically switched during the image data transmission process based on the short-term fluctuation trend, specifically including: S221: Determine the amplitude of the short-term fluctuation trend to determine the image compression strategy according to the corresponding fluctuation amplitude.

[0048] Specifically, first, the intensity of the current network bandwidth fluctuation is determined by calculating the amplitude of bandwidth fluctuation. If the fluctuation amplitude is large, it means that the network bandwidth changes drastically. At this time, the image resolution and frame rate need to be reduced to adapt to the lower bandwidth; if the fluctuation amplitude is small, it means that the network is relatively stable, and the image resolution and frame rate can be appropriately increased to enhance the image quality. Therefore, different image compression strategies are selected according to the different fluctuation amplitudes to ensure the stability and smoothness of image transmission.

[0049] S222: When the fluctuation amplitude exceeds the preset stable fluctuation range, based on the bandwidth consumption of the image data stream, the resolution of the corresponding area is reduced from large to small, and the transmission of redundant frames is reduced through inter-frame difference analysis.

[0050] Specifically, when the bandwidth fluctuation exceeds the preset stable fluctuation range, the bandwidth consumption of the image data stream will increase significantly. At this time, it is necessary to reduce the bandwidth occupancy by reducing the resolution, especially the low-frequency areas in the image, such as the background, static objects, etc., whose resolution can be reduced to reduce the data volume. At the same time, the inter-frame difference analysis technology is used to retain only the key frames with large changes, reduce the transmission of redundant frames, and ensure the smoothness and stability of image transmission under limited bandwidth.

[0051] S223: When the fluctuation amplitude is within a preset stable fluctuation range, based on the complexity of the image data stream, the corresponding frame rate and resolution are increased in sequence from high to low.

[0052] Specifically, when the bandwidth fluctuation range is within the preset stable fluctuation range, it means that the network bandwidth is sufficient to support higher quality image transmission. At this time, the frame rate and resolution can be dynamically adjusted according to the complexity of the image. For image areas with higher complexity, such as areas with fast motion or rich details, the frame rate and resolution are increased to improve the smoothness and detail presentation of the image; for areas with lower complexity, the resolution is appropriately increased to optimize image quality while ensuring data transmission efficiency.

[0053] In one embodiment, if Figure 5 As shown, in step S30, the image data stream is comprehensively analyzed using a deep learning algorithm and user operation features are extracted, specifically including: S31: Process the key frames of the image data stream through a multi-scale feature extraction network to generate a feature map.

[0054] Specifically, each frame in the image data stream is processed by a multi-scale feature extraction network, which can extract multi-level feature information in the image through convolution operations at different scales. The key frames are obtained by sampling the key frames of the image stream. These key frames represent important information in the image sequence. The feature map extracts the most recognizable features in the image, such as edges and textures, to provide basic data for subsequent user operation feature recognition.

[0055] S32: Extract action key points from the feature graph, and then construct a temporal feature vector of the user operation.

[0056] Specifically, in the generated feature map, the system extracts key points in user operations, such as gestures, facial expressions, or changes in the position of objects, through a key point detection algorithm. By tracking the changes in these key points, a time series feature vector is constructed to represent the user's operation trajectory within a certain period of time. The time series feature vector contains the time series change information of the user's operation, and provides a numerical expression of the user's intention for generating control instructions.

[0057] S33: Generate a control instruction based on matching the time series feature vector with a preset operation feature model.

[0058] Specifically, the timing feature vector is matched with a pre-trained operation feature model, which contains the characteristic patterns of different user operations. By matching with the preset operation model, the user's intention can be accurately identified and corresponding control instructions can be generated. These control instructions can be used to adjust image display, start applications, change device status, etc., thereby achieving the goal of optimizing user experience.

[0059] In one embodiment, if Figure 6 As shown, in step S40, content analysis is performed using a corresponding content analysis strategy according to the content information, specifically including: S41: Identify the type of content information, where the type of content information includes text information and coding information.

[0060] Specifically, the image data stream may contain different types of content information, such as text, QR code, barcode, etc. The system identifies the types of these contents by analyzing the structural features of the image content. Text information usually contains words, symbols, etc., while coded information includes graphic coded content such as barcodes and QR codes. The system automatically distinguishes these different types of information based on image features to provide a basis for subsequent analysis.

[0061] S42: For text information, a context-dependent multilingual character recognition model is used to extract the content of the text information.

[0062] Specifically, text information is processed through optical character recognition technology. The system adopts a context-related multilingual character recognition model to identify and extract text information in the image. The model can recognize text information in different languages ​​and fonts, improve recognition accuracy through the association of contextual information, and format the extracted text for subsequent display or storage.

[0063] S43: For the coded information, the color distribution characteristics and position characteristics of the image area are combined for positioning, and the barcode / QR code recognition technology is used to decode the content of the coded information.

[0064] Specifically, barcodes and QR codes are located through the color distribution and position information of the image area. The system extracts the area of ​​​​the encoded information through image analysis technology, and accurately locates the encoded area according to the position characteristics. Then, the barcode or QR code decoding technology is used to decode the barcode or QR code information in the image, obtain the data stored therein, and convert it into usable text or digital information for further processing.

[0065] In one embodiment, if Figure 7 As shown, in step S50, corresponding adjustment feedback instructions are generated based on the user interaction behavior, so as to personalize the user interface through the adaptive feedback mechanism, specifically including: S51: classifying the user interaction behavior into behavior patterns, wherein the behavior patterns include touch behavior and voice commands.

[0066] Specifically, user interaction behaviors are collected through input methods such as touch screens and voice input devices, and through behavior pattern classification algorithms, user interaction behaviors are divided into two categories: touch behaviors and voice commands. Touch behaviors include touch position, strength, duration, etc.; voice commands include user voice input content, which is converted and analyzed through voice recognition technology.

[0067] S52: For the touch behavior, a behavior sequence is generated by recording the touch position, force and duration, and a preset user interface adjustment strategy is matched according to the behavior sequence.

[0068] Specifically, the touch screen records the user's touch position, strength, duration and other data to form a touch behavior sequence. By analyzing these data, the content and method of the interface display are dynamically adjusted according to the preset user interface adjustment strategy, such as adjusting the size of menu items for light touch operations and displaying more options or details for long press operations, thereby improving the user's interactive experience.

[0069] S53: For the voice command, dynamically adjust the feedback content in combination with the real-time voice recognition result and the user's current interface status.

[0070] Specifically, combined with real-time speech recognition technology, the user's voice commands are converted into text, the content is analyzed and dynamic feedback is provided based on the current interface status. If the user requests other functions, the system will adjust and display the corresponding content based on the interface situation, thereby achieving personalized and intelligent interactive feedback.

[0071] It should be understood that the size of the serial numbers of the steps in the above embodiments does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0072] In one embodiment, a touch integrated machine interactive control device is provided, and the touch integrated machine interactive control device corresponds one-to-one to the touch integrated machine interactive control method in the above embodiment. Figure 8 As shown, the touch integrated machine interactive control device includes an image receiving module, a network bandwidth monitoring module, an operation feature extraction module, a content information extraction module, and a user interaction behavior monitoring module. The detailed description of each functional module is as follows: An image receiving module is used to receive image data through a preset video stream protocol to obtain a corresponding image data stream; The network bandwidth monitoring module is used to obtain the real-time network bandwidth status, and then perform adaptive compression processing on the image data stream based on the real-time network bandwidth status, and dynamically adjust the image transmission parameters; An operation feature extraction module is used to perform a comprehensive analysis of the image data stream using a deep learning algorithm, extract the user's operation features, and generate output control instructions based on the operation features, thereby controlling and adjusting the image display and / or device status to optimize the user experience; A content information extraction module is used to extract content information from the image data stream, and perform content analysis using a corresponding content analysis strategy according to the content information, so as to perform real-time display and / or storage according to the analyzed content; The user interaction behavior monitoring module is used to obtain user interaction behaviors and generate corresponding adjustment feedback instructions based on the user interaction behaviors, so as to personalize the user interface through an adaptive feedback mechanism.

[0073] Optionally, the network bandwidth monitoring module specifically includes: The bandwidth fluctuation prediction submodule is used to perform hierarchical prediction on the real-time fluctuation of network bandwidth status to obtain short-term fluctuation trends; The compression strategy adjustment submodule is used to dynamically switch the corresponding image compression strategy combination during the image data transmission process based on the short-term fluctuation trend, wherein the image compression strategy includes a resolution degradation strategy and a frame rate adjustment strategy.

[0074] Optionally, the bandwidth fluctuation prediction submodule specifically includes: A real-time data collection unit is used to continuously sample the real-time network bandwidth status and process the sampled data in layers according to the time window; A multi-layer filtering unit is used to adopt a multi-level filtering algorithm and use different weights to perform weighted smoothing on each layer of data based on the amplitude and change rate of bandwidth fluctuation; The trend extraction unit is used to extract short-term fluctuation trends by analyzing the smoothed data.

[0075] Optionally, the compression strategy adjustment submodule specifically includes: An amplitude judgment unit, used to judge the amplitude of the short-term fluctuation trend, so as to determine the image compression strategy according to the corresponding fluctuation amplitude; An image area compression adjustment unit is used to reduce the resolution of the corresponding area from large to small based on the bandwidth consumption of the image data stream when the fluctuation amplitude exceeds the preset stable fluctuation range, and reduce the transmission of redundant frames through inter-frame difference analysis; The image quality optimization unit is used to increase the corresponding frame rate and resolution from high to low based on the complexity of the image data stream when the fluctuation amplitude is within a preset stable fluctuation range.

[0076] Optionally, the operation feature extraction module specifically includes: A multi-scale feature extraction submodule is used to process the key frames of the image data stream through a multi-scale feature extraction network to generate a feature map; The action key point extraction submodule is used to extract the action key points in the feature map and then construct the temporal feature vector of the user operation; The operation model matching submodule is used to match the timing feature vector with the preset operation feature model to generate a control instruction.

[0077] Optionally, the content information extraction module specifically includes: A content type identification submodule, used to identify the type of content information, wherein the type of content information includes text information and encoding information; A text information extraction submodule is used to extract the content of text information using a context-dependent multilingual character recognition model; The coded information decoding submodule is used to locate the coded information in combination with the color distribution characteristics and position characteristics of the image area, and decode the content of the coded information using barcode / QR code recognition technology.

[0078] Optionally, the user interaction behavior monitoring module specifically includes: A behavior pattern classification submodule is used to classify the behavior patterns of user interaction behaviors, wherein the behavior patterns include touch behaviors and voice commands; A touch behavior processing submodule is used to generate a behavior sequence for touch behavior by recording the touch position, force and duration, and match a preset user interface adjustment strategy according to the behavior sequence; The voice command processing submodule is used to dynamically adjust the feedback content of the voice command based on the real-time voice recognition results and the user's current interface status.

[0079] For the specific definition of the touch all-in-one interactive control device, please refer to the definition of the touch all-in-one interactive control method above, which will not be repeated here. Each module in the above-mentioned touch all-in-one interactive control device can be implemented in whole or in part by software, hardware and a combination thereof. The above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above modules.

[0080] In one embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as follows: Fig. 9As shown. The computer device includes a processor, a memory, a network interface and a database connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a touch all-in-one interactive control method is implemented.

[0081] In one embodiment, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the following steps when executing the computer program: Receive image data through a preset video stream protocol to obtain a corresponding image data stream; Obtain the real-time network bandwidth status, and then perform adaptive compression processing on the image data stream based on the real-time network bandwidth status, and dynamically adjust the image transmission parameters; Use deep learning algorithms to comprehensively analyze image data streams, extract user operation features, and generate output control instructions based on the operation features, thereby controlling and adjusting image display and / or device status to optimize user experience; Extracting content information from the image data stream, and performing content analysis using a corresponding content analysis strategy according to the content information, so as to perform real-time display and / or storage according to the analyzed content; The user interaction behavior is obtained, and corresponding adjustment feedback instructions are generated based on the user interaction behavior, so as to personalize the user interface through an adaptive feedback mechanism.

[0082] In one embodiment, a computer readable storage medium is provided, on which a computer program is stored, and when the computer program is executed by a processor, the following steps are implemented: Receive image data through a preset video stream protocol to obtain a corresponding image data stream; Obtain the real-time network bandwidth status, and then perform adaptive compression processing on the image data stream based on the real-time network bandwidth status, and dynamically adjust the image transmission parameters; Use deep learning algorithms to comprehensively analyze image data streams, extract user operation features, and generate output control instructions based on the operation features, thereby controlling and adjusting image display and / or device status to optimize user experience; Extracting content information from the image data stream, and performing content analysis using a corresponding content analysis strategy according to the content information, so as to perform real-time display and / or storage according to the analyzed content; The user interaction behavior is obtained, and corresponding adjustment feedback instructions are generated based on the user interaction behavior, so as to personalize the user interface through an adaptive feedback mechanism.

[0083] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application may include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0084] Those skilled in the art can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example. In actual applications, the above-mentioned functions can be distributed and completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above.

[0085] The embodiments described above are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, a person skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. Such modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application, and should all be included in the protection scope of the present application.

Claims

1. A touch integrated machine interactive control method, characterized in that: The touch integrated machine interactive control method comprises: Receive image data through a preset video stream protocol to obtain a corresponding image data stream; Acquire real-time network bandwidth status, and then perform adaptive compression processing on the image data stream based on the real-time network bandwidth status, and dynamically adjust image transmission parameters; Using a deep learning algorithm to comprehensively analyze the image data stream, extract the user's operation characteristics, and generate output control instructions based on the operation characteristics, thereby controlling and adjusting the image display and / or device status to optimize the user experience; Extracting content information from the image data stream, and performing content analysis using a corresponding content analysis strategy according to the content information, so as to perform real-time display and / or storage according to the analyzed content; The user interaction behavior is acquired, and corresponding adjustment feedback instructions are generated based on the user interaction behavior, so as to perform personalized adjustment on the user interface through an adaptive feedback mechanism.

2. The touch integrated machine interactive control method according to claim 1, characterized in that: The adaptively compressing the image data stream based on the real-time network bandwidth status and dynamically adjusting the image transmission parameters specifically include: Performing hierarchical forecasting on the real-time fluctuation of the network bandwidth status to obtain short-term fluctuation trends; Based on the short-term fluctuation trend, a corresponding image compression strategy combination is dynamically switched during the image data transmission process, wherein the image compression strategy includes a resolution degradation strategy and a frame rate adjustment strategy.

3. The touch integrated machine interactive control method according to claim 2, characterized in that: The real-time fluctuation of the network bandwidth status is predicted in layers to obtain a short-term fluctuation trend, specifically including: Continuously sampling the real-time network bandwidth status, and performing layered processing on the sampled data according to the time window; A multi-level filtering algorithm is used, and different weights are used to perform weighted smoothing on each layer of data based on the amplitude and change rate of bandwidth fluctuations; By analyzing the smoothed data, the short-term fluctuation trend is extracted.

4. The touch integrated machine interactive control method according to claim 2, characterized in that: The dynamically switching the corresponding image compression strategy combination during the image data transmission process based on the short-term fluctuation trend specifically includes: Performing amplitude judgment on the short-term fluctuation trend to determine the image compression strategy according to the corresponding fluctuation amplitude; In the case where the fluctuation amplitude exceeds the preset stable fluctuation range, based on the bandwidth consumption of the image data stream, the resolution of the corresponding areas is reduced from large to small, and the transmission of redundant frames is reduced through inter-frame difference analysis; When the fluctuation amplitude is within a preset stable fluctuation range, the corresponding frame rate and resolution are increased in sequence from high to low based on the complexity of the image data stream.

5. The touch integrated machine interactive control method according to claim 1, characterized in that: The use of a deep learning algorithm to comprehensively analyze the image data stream and extract user operation features specifically includes: Processing the key frames of the image data stream through a multi-scale feature extraction network to generate a feature map; Extracting action key points from the feature graph, and then constructing a temporal feature vector of the user operation; Based on matching the timing feature vector with a preset operation feature model, a control instruction is generated.

6. The touch integrated machine interactive control method according to claim 1, characterized in that: The adopting a corresponding content analysis strategy to perform content analysis according to the content information specifically includes: Identifying the type of the content information, wherein the type of the content information includes text information and encoding information; For the text information, extract the content of the text information using a context-dependent multilingual character recognition model; The coded information is located by combining the color distribution characteristics and position characteristics of the image area, and the barcode / QR code recognition technology is used to decode the content of the coded information.

7. The touch integrated machine interactive control method according to claim 1, characterized in that: The generating corresponding adjustment feedback instructions based on the user interaction behavior to personalize the user interface through an adaptive feedback mechanism specifically includes: Classifying the user interaction behavior into behavior patterns, wherein the behavior patterns include touch behavior and voice commands; For the touch behavior, a behavior sequence is generated by recording the touch position, force and duration, and a preset user interface adjustment strategy is matched according to the behavior sequence; For the voice command, the feedback content is dynamically adjusted in combination with the real-time voice recognition result and the user's current interface status.

8. A touch integrated machine interactive control device, characterized in that: The touch integrated machine interactive control device comprises: An image receiving module is used to receive image data through a preset video stream protocol to obtain a corresponding image data stream; A network bandwidth monitoring module, used to obtain real-time network bandwidth status, and then perform adaptive compression processing on the image data stream based on the real-time network bandwidth status, and dynamically adjust image transmission parameters; An operation feature extraction module is used to perform a comprehensive analysis on the image data stream using a deep learning algorithm, extract the user's operation features, and generate output control instructions based on the operation features, thereby controlling and adjusting the image display and / or device status to optimize the user experience; A content information extraction module, used to extract content information from the image data stream, and perform content analysis using a corresponding content analysis strategy according to the content information, so as to perform real-time display and / or storage according to the analyzed content; The user interaction behavior monitoring module is used to obtain user interaction behaviors and generate corresponding adjustment feedback instructions based on the user interaction behaviors, so as to personalize the user interface through an adaptive feedback mechanism.

9. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the steps of the touch-integrated machine interactive control method according to any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by the processor, the steps of the touch-integrated machine interactive control method according to any one of claims 1 to 7 are implemented.

Images