Hardware device driving strategy generation method and device, storage medium and program product

Abstract

The invention relates to the technical field of artificial intelligence, and particularly provides a hardware equipment driving strategy generation method and device, a storage medium and a program product, and the hardware equipment driving strategy generation method based on multimedia content comprises the steps: analyzing the multimedia content, and obtaining the feature information of the multimedia content; generating driving strategy information associated with the multimedia content based on the feature information; wherein the driving policy information is used for indicating one or more hardware devices associated with the multimedia content, and driving predetermined processing executed by the one or more hardware devices. According to the hardware equipment driving strategy generation method disclosed by the invention, the automatic analysis of the multimedia content in the on-demand scene and the live broadcast scene and the automatic generation of the related hardware equipment driving strategy are realized, and the real-time linkage based on the multimedia content is provided.

Description

Technical Field

[0001] The present disclosure relates to the field of artificial intelligence technology, and in particular to a method, device, storage medium and program product for generating a hardware device driver strategy. Background Art

[0002] With the development of IT and artificial intelligence technologies, multimedia content is further integrated with other hardware devices besides the display devices themselves, providing users with a richer experience. Devices such as smart cockpits and multimedia entertainment cabins can not only provide video and audio reproduction of multimedia content, but also cooperate with the reproduced content to provide visual, auditory, and tactile outputs through the linkage of other hardware devices.

[0003] Currently, the association between multimedia content and other hardware devices mainly relies on manual marking in the multimedia content, which causes additional labor costs. In addition, for multimedia content in live broadcast scenarios, real-time marking cannot be performed, so that real-time linkage based on multimedia content cannot be provided. Summary of the invention

[0004] The present disclosure is proposed in view of the above problems. The present disclosure provides a method, an apparatus, an electronic device, a smart cockpit, a storage medium and a computer program product for generating a hardware device driver strategy.

[0005] According to one aspect of the present disclosure, a method for generating a hardware device driver strategy based on multimedia content is provided, comprising: analyzing the multimedia content and acquiring feature information of the multimedia content; based on the feature information, generating driver strategy information associated with the multimedia content; wherein the driver strategy information is used to indicate one or more hardware devices associated with the multimedia content, and to drive the one or more hardware devices to perform predetermined processing.

[0006] In addition, according to an aspect of the present disclosure, a method for generating a hardware device driver strategy based on multimedia content analyzes multimedia content and obtains feature information of the multimedia content, including: identifying one or more target objects in the multimedia content, and obtaining content feature information of the one or more target objects; and / or, identifying one or more specific picture frames in the multimedia content, and obtaining picture feature information of the one or more specific picture frames, wherein the picture feature information includes picture feature information determined based on the picture change amplitude of multiple intra-frame coded picture frames in the one or more specific picture frames; and / or, identifying an audio signal in the multimedia content, and obtaining audio feature information of the audio signal, wherein the audio feature information includes audio feature information determined based on the change amplitude of the level value corresponding to the audio signal in the multimedia content; using one or more of the content feature information, picture feature information and audio feature information as feature information.

[0007] In addition, according to one aspect of the present disclosure, a method for generating a hardware device driving strategy based on multimedia content also includes: generating a script file based on driving strategy information, the script file being used to instruct driving one or more hardware devices to perform predetermined processing at a predetermined time point; and generating a driving signal for driving one or more hardware devices to perform predetermined processing based on the playback progress of the multimedia content and the script file.

[0008] In addition, according to one aspect of the present disclosure, the method for generating a hardware device driving strategy based on multimedia content further includes: generating a driving signal for driving one or more hardware devices to perform a predetermined process based on the real-time content of the multimedia content and the driving strategy information.

[0009] In addition, according to one aspect of the present disclosure, the method for generating a hardware device driver strategy based on multimedia content also includes: when the multimedia content is based on live streaming data, before analyzing the multimedia content, determining the time difference between the multimedia content and the live streaming data for the same picture frame; and based on the time difference, adding event information for indicating a predetermined event in the live streaming data.

[0010] In addition, according to an aspect of the present disclosure, a method for generating a hardware device driver strategy based on multimedia content determines the time difference between the multimedia content and the live streaming data, including: generating synchronization data consistent with the live streaming data; identifying first event time information of the first event in the synchronization data, and determining first time difference information based on the first event time information and second event time information of the first event marked in the live streaming data; and / or determining second time difference information based on first picture time information corresponding to the first picture of the synchronization data and second picture time information corresponding to the first picture of the live streaming data; and / or determining third time difference information based on the marked time information of the first event marked in the live streaming data and the display time information of the first event; determining the time difference based on one or more of the first time difference information, the second time difference information and the third time difference information.

[0011] According to another aspect of the present disclosure, a device for generating a hardware device driving strategy based on multimedia content is provided, comprising: a feature information acquisition unit, configured to analyze multimedia content and acquire feature information of the multimedia content; a driving strategy information generation unit, configured to generate driving strategy information associated with the multimedia content based on the feature information; wherein the driving strategy information is used to indicate one or more hardware devices associated with the multimedia content, and to drive the one or more hardware devices to perform predetermined processing.

[0012] According to another aspect of the present disclosure, an electronic device is provided, including: a memory for storing computer-readable instructions; and a processor for executing the computer-readable instructions so that the electronic device executes the above method for generating a hardware device driver strategy based on multimedia content.

[0013] According to another aspect of the present disclosure, a content playback device is provided, comprising: a display device for playing multimedia content; one or more hardware devices for performing predetermined processing corresponding to the multimedia content; and a hardware device driving device for analyzing the multimedia content and obtaining feature information of the multimedia content; based on the feature information, generating driving strategy information associated with the multimedia content; wherein the driving strategy information is used to indicate one or more hardware devices associated with the multimedia content, and to drive the one or more hardware devices to perform predetermined processing.

[0014] According to another aspect of the present disclosure, a non-transitory computer-readable storage medium is provided for storing computer-readable instructions. When the computer-readable instructions are executed by a processor, the processor executes the method for generating a hardware device driver strategy based on multimedia content as described above.

[0015] According to another aspect of the present disclosure, a computer program product is provided, including a computer program, characterized in that when the computer program is executed by a processor, the method for generating a hardware device driver strategy based on multimedia content as described above is implemented.

[0016] As will be described in detail below, according to the hardware device driver strategy generation method, apparatus, electronic device, smart cockpit, storage medium and computer program product of the embodiments of the present disclosure, feature information of the multimedia content is obtained by analyzing the multimedia content, and then driver strategy information associated with the multimedia content is generated based on the feature information, thereby realizing automatic analysis of multimedia content and automatic generation of related hardware device driver strategies in on-demand and live broadcast scenarios, and providing a real-time linkage solution based on multimedia content.

[0017] It is to be understood that both the foregoing general description and the following detailed description are exemplary, and are intended to provide further explanation of the technology as claimed. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] The above and other purposes, features and advantages of the present disclosure will become more apparent by describing the embodiments of the present disclosure in more detail in conjunction with the accompanying drawings. The accompanying drawings are used to provide a further understanding of the embodiments of the present disclosure and constitute a part of the specification. Together with the embodiments of the present disclosure, they are used to explain the present disclosure and do not constitute a limitation of the present disclosure. In the accompanying drawings, the same reference numerals generally represent the same components or steps.

[0019] Figure 1A and Figure 1B 2 is a schematic diagram illustrating a content playback device to which the method for generating a hardware device driver strategy based on multimedia content according to an embodiment of the present disclosure is applied.

[0020] Figure 2 4 is a flowchart illustrating a method for generating a hardware device driver strategy based on multimedia content according to an embodiment of the present disclosure.

[0021] Figure 3 It is a flowchart illustrating a method for generating a hardware device driver strategy based on multimedia content according to an embodiment of the present disclosure.

[0022] Figure 4 It is a diagram illustrating the implementation process of a method for generating a hardware device driver strategy based on multimedia content according to an embodiment of the present disclosure.

[0023] Figure 5 It is a flowchart further illustrating the feature information acquisition process in the method for generating a hardware device driver strategy based on multimedia content according to an embodiment of the present disclosure.

[0024] Figure 6 It is a flowchart further illustrating a method for generating a hardware device driver strategy based on multimedia content according to an embodiment of the present disclosure in a video-on-demand scenario.

[0025] Figure 7 It is a flowchart further illustrating a method for generating a hardware device driver strategy based on multimedia content according to an embodiment of the present disclosure in a live broadcast scenario.

[0026] Figure 8 It is a flowchart further illustrating a method for generating a hardware device driver strategy based on multimedia content according to an embodiment of the present disclosure in a live broadcast scenario.

[0027] Fig. 9 4 is a functional block diagram illustrating an apparatus for generating a hardware device driving strategy based on multimedia content according to an embodiment of the present disclosure.

[0028] Fig.10 is a hardware block diagram illustrating an electronic device according to an embodiment of the present disclosure.

[0029] Fig.11 is a schematic diagram illustrating a computer program product according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0030] In order to make the purpose, technical solution and advantages of the present disclosure more obvious, the exemplary embodiments according to the present disclosure will be described in detail with reference to the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present disclosure, rather than all the embodiments of the present disclosure, and it should be understood that the present disclosure is not limited to the exemplary embodiments described here.

[0031] First, refer to Figure 1A and Figure 1B A content playback device to which the method for generating a hardware device driver strategy based on multimedia content according to an embodiment of the present disclosure is applied is described. Figure 1A and Figure 1B Content playback devices 10A and 10B are shown respectively.

[0032] Specifically, Figure 1A The content playback device 10A shown can be implemented as a smart cockpit of a vehicle or as a component thereof. The content playback device 10A includes at least a display device 101A for playing multimedia content. The content playback device 10A also includes one or more hardware devices (not shown) for performing predetermined processing corresponding to the multimedia content. In the application scenario of the smart cockpit, one or more hardware devices may be, for example, lighting equipment, audio equipment, seat equipment, air conditioning equipment, and fragrance equipment of the vehicle. In addition, the content playback device 10A also includes a hardware device driver (not shown) for analyzing multimedia content and obtaining feature information of the multimedia content; based on the feature information, generating driving strategy information associated with the multimedia content. In the application scenario of the smart cockpit, the hardware device driver may be, for example, a dedicated or general-purpose control chip or processor configured in the vehicle. The driving strategy information is used to indicate one or more hardware devices associated with the multimedia content, and to drive the predetermined processing performed by one or more hardware devices. In the application scenario of the smart cockpit, the driving strategy information is used, for example, to instruct lighting equipment (such as interior ambient lights, exterior low and high beam lights, etc.) to switch lighting colors, modes, and brightness; instruct audio equipment to amplify and switch different sound channels; instruct seat equipment to perform vibration, massage, ventilation, and heating, etc.; instruct air-conditioning equipment to perform temperature settings, wind speed settings, and air purification, etc.; and instruct fragrance equipment to perform fragrance concentration and type settings, etc.

[0033] Similarly, if Figure 1BThe content playback device 10B shown can be implemented as an intelligent multimedia entertainment cabin or as a component thereof. The content playback device 10B includes at least a display device 101B for playing multimedia content. The content playback device 10B also includes one or more hardware devices (not shown) for performing predetermined processing corresponding to the multimedia content. In the application scenario of the intelligent multimedia entertainment cabin, one or more hardware devices may be, for example, lighting equipment, audio equipment, seat equipment, air conditioning equipment, and fragrance equipment. In addition, the content playback device 10B also includes a hardware device driver (not shown) for analyzing multimedia content and obtaining feature information of the multimedia content; based on the feature information, generating drive strategy information associated with the multimedia content. In the application scenario of the intelligent multimedia entertainment cabin, the hardware device driver may be, for example, a dedicated or general control chip or processor configured in the intelligent multimedia entertainment cabin. The drive strategy information is used to indicate one or more hardware devices associated with the multimedia content, and to drive one or more hardware devices to perform predetermined processing. In the application scenario of the intelligent multimedia entertainment cabin, the driving strategy information is used, for example, to instruct the lighting equipment to switch the lighting color, mode and brightness; to instruct the audio equipment to amplify and switch different channels; to instruct the seat equipment to perform vibration, massage, ventilation and heating, etc.; to instruct the air-conditioning equipment to perform temperature setting, wind speed setting and air purification, etc.; and to instruct the fragrance equipment to perform fragrance concentration and type setting, etc.

[0034] Below, we will refer to Figures 2 to 8 The method for generating a hardware device driver strategy based on multimedia content according to an embodiment of the present disclosure is further described in detail.

[0035] like Figure 2 As shown, the method for generating a hardware device driver strategy based on multimedia content according to an embodiment of the present disclosure includes the following steps.

[0036] In step S201, the multimedia content is analyzed to obtain feature information of the multimedia content. In one embodiment of the present disclosure, the feature information of the multimedia content includes content feature information obtained by analyzing the picture content. In one embodiment of the present disclosure, the feature information of the multimedia content also includes picture feature information and audio feature information obtained independently of the specific content of the picture. One or more of the content feature information, picture feature information and audio feature information are used as feature information. Figure 5 The feature information acquisition process is described in detail.

[0037] In step S202, based on the feature information, drive strategy information associated with the multimedia content is generated. The drive strategy information is used to indicate one or more hardware devices associated with the multimedia content, and to drive the one or more hardware devices to perform a predetermined process.

[0038] In one embodiment of the present disclosure, based on one or more of the content feature information, the picture feature information, and the audio feature information, driving strategy information associated with the multimedia content is generated. The driving strategy information at least indicates weight information associated with one or more target objects in the multimedia content, one or more hardware devices corresponding to the one or more target objects, and predetermined processing corresponding to the one or more hardware devices. Figure 6 and Figure 7 The specific process of the hardware device driver strategy generation method in the on-demand scenario and the live broadcast scenario is described in detail.

[0039] Figure 3 It is a flowchart illustrating a method for generating a hardware device driver strategy based on multimedia content according to an embodiment of the present disclosure.

[0040] like Figure 3 As shown, the first stage performs content understanding, which corresponds to the above step S201. Figure 3 As shown, content understanding includes analysis of images and audio, achieving shot segmentation, character recognition, object recognition, emotion classification, scene recognition, optical flow algorithm, primary color extraction, sound classification, etc., so as to give automatic labels to one or more identified target objects, such as actions, scenes, objects, atmosphere, theme colors, emotions, etc.

[0041] The second stage executes the generation of the strategy table, which corresponds to the above step S202. Based on the analysis results of the first stage, the driving strategies in multiple aspects such as vision, hearing, touch, and smell are generated.

[0042] The third stage executes hardware linkage, that is, drives one or more associated hardware devices to execute predetermined processing based on the driving strategies generated after the first two stages. Figure 3 Specifically, in the application scenario of the smart cockpit, the ambient light equipment switches the lighting color, mode and brightness; the power amplifier equipment performs amplification and switching of different channels; the seat equipment performs vibration, massage, ventilation and heating, etc.; the air-conditioning equipment performs temperature setting, wind speed setting and air purification, etc.; and the fragrance equipment performs fragrance concentration and type setting, etc.

[0043] Figure 4 It is a diagram illustrating the implementation process of a method for generating a hardware device driver strategy based on multimedia content according to an embodiment of the present disclosure.

[0044] like Figure 4As shown, multimedia content such as videos, games, music, etc., which have content tags, picture information, and audio information, enter the middleware. It should be understood that the middleware is a bridge to establish the linkage between content and hardware, and can call the hardware driver interface according to the content analysis results. In addition, the middleware can also adapt to different systems to achieve the same content and hardware linkage on different system devices (for example, different models). Specifically, the middleware is connected to the software development kit SDK provided by each system equipment manufacturer (for example, car company) that can call hardware devices to implement the operation of the hardware application program interface API. Different equipment manufacturers have different calling methods, calling functions, and value methods for the same hardware. Therefore, for the same type of hardware from different equipment manufacturers, a unified calling method is needed to adapt to different hardware. In addition, the values ​​of the same type of equipment need to be normalized. For example, the air conditioning wind speed calling method of vehicle A is: HVAC_FUNC_FAN_SPEED_HARD_KEY, with a value range of: 0-9, and the air conditioning wind speed calling method of vehicle B is: AC_FUNC_FAN_SPEED_HARD_KEY, with a value range of: 0-100. It is necessary to map the calling methods of vehicle A and vehicle B, and normalize their values ​​to 0-100. In addition, due to the inconsistency of systems and system versions of different equipment manufacturers, it is also necessary for the middleware to adapt the system and hardware so that it can run normally on each hardware system.

[0045] Figure 5 It is a flowchart further illustrating the feature information acquisition process in the method for generating a hardware device driver strategy based on multimedia content according to an embodiment of the present disclosure.

[0046] In step S501, one or more target objects in multimedia content are identified, and content feature information of the one or more target objects is obtained.

[0047] In one embodiment of the present disclosure, the content feature information depends on the content itself and usually needs to be identified in advance. That is, in the scenario of on-demand existing multimedia content, the content feature information can be acquired in advance, while in the real-time live broadcast scenario, the acquisition of content feature information is usually not performed to ensure the real-time nature of the live broadcast. In the subsequently generated driving strategy information, target objects with different content feature information correspond to different driving strategies.

[0048] In step S502, one or more specific picture frames in the multimedia content are identified, and picture feature information of the one or more specific picture frames is obtained.

[0049] In one embodiment of the present disclosure, one or more specific picture frames are one or more intra-frame coded picture frames (I frames). The multimedia content can be read frame by frame or frame by frame, and the coding format of the multimedia content is converted into a frame image. By judging the frame type or checking the key frame information in the frame data structure, the I frame is found, and the I frame is the largest of all frames. The decoded multimedia content is traversed to find all I frames. Furthermore, the picture feature information is determined based on the picture change amplitude of multiple intra-frame coded picture frames (I frames). For example, the first and second theme colors of the I frame are calculated by clustering. The optical flow method is used to calculate the change amplitude between each frame. Different theme colors and different change amplitudes can correspond to different driving strategies in the driving strategy information generated subsequently.

[0050] In step S503, an audio signal in the multimedia content is identified, and audio feature information of the audio signal is acquired.

[0051] In one embodiment of the present disclosure, the level value corresponding to the audio signal in the multimedia content is identified, and the audio feature information is determined based on the change amplitude of the level value. More specifically, the level value of the audio signal is obtained by measuring the average power or amplitude of the audio signal. Classification is performed according to the change amplitude of the level value, including:

[0052] 1) Segmentation of audio signal: Divide the audio signal into several segments, each of which should be short enough to reflect the trend of level change. Window function or other segmentation methods can be used for segmentation.

[0053] 2). Calculate the root mean square (RMS) level or peak level of each segment: For each segment of the signal, calculate their RMS level or peak level and store them as a list.

[0054] 3) Determine the threshold: By analyzing the list of level values, determine which range of level changes is defined as "gentle", which range should be defined as "small change", which range should be defined as "large change", etc.

[0055] 4) Classification by threshold: Using the threshold determined above, classify each signal segment as “flat”, “small change” or “large change”, etc.

[0056] Different types of variation ranges correspond to different driving strategies in the subsequently generated driving strategy information.

[0057] In addition, automatic classification is performed according to the amplitude and frequency of level changes, including:

[0058] 1). Calculate the average level and variation range of each signal segment: For each signal segment, calculate their average level and range, such as standard deviation or range.

[0059] 2) Cluster each signal segment: Take each signal segment as a data point and use a clustering algorithm to cluster the data. The number of cluster centers can be set according to actual needs, and different cluster numbers can represent different level change ranges.

[0060] 3). Determine the frequency in each category: After assigning each data point to a cluster center, the frequency of the category represented by each cluster center can be calculated.

[0061] 4). Classify the levels based on clustering and frequency analysis: By combining the clustering results and frequency analysis, the levels of the audio signal can be classified into different categories.

[0062] Different occurrence frequency types correspond to different driving strategies in the subsequently generated driving strategy information. For example, when the least common level appears, the seat can be driven, and when the most common level appears, the ambient light can be driven.

[0063] In step S504, one or more of the content feature information, the picture feature information and the audio feature information is used as the feature information.

[0064] In one embodiment of the present disclosure, depending on whether the multimedia content is on-demand or live, and whether the multimedia content is video or audio, one or more of content feature information, picture feature information, and audio feature information are obtained as corresponding feature information.

[0065] Figure 6 It is a flowchart further illustrating a method for generating a hardware device driver strategy based on multimedia content according to an embodiment of the present disclosure in a video-on-demand scenario.

[0066] In step S601, the multimedia content is analyzed to obtain feature information of the multimedia content. In the on-demand scenario, the feature information of the multimedia content includes content feature information obtained by analyzing the picture content, picture feature information obtained independently of the specific content of the picture, and audio feature information. The method for obtaining the content feature information, picture feature information, and audio feature information is as described in detail in the following. Figure 5 The above description will not be repeated here.

[0067] In step S602, based on the feature information, drive strategy information associated with the multimedia content is generated. The drive strategy information is used to indicate one or more hardware devices associated with the multimedia content, and to drive the one or more hardware devices to perform a predetermined process.

[0068] Table 1 schematically shows a driving strategy information table associated with multimedia content, which is generated based on feature information of the multimedia content.

[0069]

[0070] Table 1

[0071] The data types in Table 1 include audio and video. Data classification and secondary classification are determined by analyzing content feature information. For example, it is divided into people and objects, and people are further divided into nurses, soldiers, firefighters, etc., and objects further include flowers.

[0072] There are two weighting type values ​​in Table 1: 1 and 2. 1 indicates weighted processing, and 2 indicates independent weight processing.

[0073] In the case of weighted processing, when the corresponding device takes effect, its device value is a range type (eg, 1-20, 0%-100%), and is processed in an overlapping manner according to the value range (cannot exceed the maximum value).

[0074] For example, for nurses, the weighted type value is 1 and the value range is 5%, then the final equipment value range is n%+5%.

[0075] In the case of independent weight processing, all objects with a weight type value of 2 will be executed according to the highest weight if the hardware devices they call are the same. When a content has multiple categories and drives hardware devices at the same time, only the hardware device with the highest weight will be effective. When the weight type value is 1, the weight is not effective.

[0076] The mutually exclusive relationship in Table 1 indicates that the same values ​​are mutually exclusive, and only one of them can take effect. The order of effectiveness can be random or in the order from top to bottom in the policy table.

[0077] The capability key in Table 1 represents the object that calls the hardware device. The capability keys of different hardware devices may be different and need to be mapped to be compatible with different device platforms.

[0078] The capability value in Table 1 indicates the range of the input parameters after the hardware device is called. Different vehicle computers with the same hardware but different values ​​need to be normalized.

[0079] The start time and end time in Table 1 represent the time range for the hardware device driver.

[0080] In step S603, a script file is generated based on the driving strategy information. The script file is used to instruct to drive the one or more hardware devices to perform a predetermined process at a predetermined time point.

[0081] In step S604, a driving signal for driving one or more hardware devices to perform a predetermined process is generated based on the play progress of the multimedia content and the script file.

[0082] Specifically, in the on-demand scenario, when playing I frames and audio, real-time analysis is performed, and the script file is parsed based on the current playback progress to determine the driving strategy for the corresponding time point, thereby generating a driving signal for driving one or more hardware devices to perform predetermined processing.

[0083] Figure 7 It is a flowchart further illustrating a method for generating a hardware device driver strategy based on multimedia content according to an embodiment of the present disclosure in a live broadcast scenario. Figure 8 It is a flowchart further illustrating a method for generating a hardware device driver strategy based on multimedia content according to an embodiment of the present disclosure in a live broadcast scenario.

[0084] In step S701, the time difference between the multimedia content and the live streaming data is determined.

[0085] Reference Figure 8 Describe step S701. In a live broadcast scenario, scheduled events may be manually marked at the live broadcast site before the live broadcast stream is connected to the content distribution network. The live broadcast stream marked with scheduled events is played on the user player side after transcoding and distribution. There may be a time difference between the time point of the scheduled event marked in the live broadcast stream and the time point when the scheduled event appears in the display screen. In addition, during the transcoding process, the process of pressing the event information into the multimedia content through supplementary enhancement information may also cause a time difference.

[0086] Specifically, Figure 8 As shown, generate synchronization data consistent with the live streaming data. Identify the first event time information of the first event in the synchronization data, and determine the first time difference information based on the first event time information and the second event time information of the first event marked in the live streaming data. For example, in the case of a live broadcast of a game, use OCR, motion recognition, object recognition and other capabilities to identify the score or event in the game screen. Obtain the score in the current screen through OCR timed scanning, and record the time when the score changes. Let this time be T utc Similarly, action recognition and object recognition mainly identify events in the competition and determine the time when the event occurs T utc . T utc That is, the first event time information of the first event. From the event data, obtain detailed information of the event, including the event name, score, event object, and the time when the event occurred. Let this time be t utc , t utc That is, the time information of the second event marked with the first event in the streaming data. utc and t utc For example, when the screen event is obtained, utcAfter that, query the corresponding event in the current event data, and then match the same event. Similarly, when using OCR to identify the score of the screen, you can match by score, such as: the current screen changes from 2:3 to 2:4, then query the event data for the score corresponding to 2:4 utc Time. utc With t utc The time difference is the first time difference information t1=T utc -t utc .

[0087] In addition, the second time difference information is determined based on the first picture time information corresponding to the first picture of the synchronization data and the second picture time information corresponding to the first picture of the live stream data. The synchronization stream is a copy of the source stream (i.e., the live stream), and there may be a time error between the two. In the same computer room, if the network architecture remains unchanged, this time error is fixed. For example, when the pictures of the live stream and the synchronization stream are shot at the same time, the difference between the same pictures is the second time difference information t2. In the live stream, the number is pushed in through SEI (in digital increment order), and when it is decoded again, the time when the same number appears is recorded separately. The time difference when the same number appears is the second time difference information t2.

[0088] In addition, the third time difference information is determined based on the marked time information of the first event and the display time information of the first event in the live stream data. During the transcoding process, the time required for the event information to be pushed into the live stream, this time error is also fixed. The numbers are pushed in through SEI (in numerical increment order, at intervals), and the UTC timestamp is included when it is compiled. 3) When the number changes in the video screen, the current UTC timestamp is recorded (for example, it can be recorded using a shooting device). The difference between the timestamp pushed in by SEI (i.e., the marked time information of the first event) and the timestamp of the number change in the screen (i.e., the display time information of the first event) is calculated, which is the third time difference information t3.

[0089] The time difference is determined based on one or more of the first time difference information t1, the second time difference information t2 and the third time difference information t3. For example, the time difference t=t1+t2+t3.

[0090] In step S702, event information indicating a predetermined event is added to the live streaming data based on the time difference. As described above, the time difference between the multimedia content and the live streaming data is determined in step S701, so that the time difference is taken into account when the event information is pushed in through the SEI information, and a certain delay is performed, so that the occurrence time of the predetermined event during the multimedia content playback is aligned with the time indicated by the event information as much as possible. In the live broadcast scenario, since the content features of the multimedia content are generally not analyzed, but the predetermined time point for driving the hardware is determined by referring to the event information pushed into the live stream, the time for driving the hardware will be more accurate after the compensation for the time difference between the multimedia content and the live streaming data is performed.

[0091] In step S703, the multimedia content is analyzed to obtain characteristic information of the multimedia content. In the live broadcast scenario, the characteristic information of the multimedia content is mainly obtained by referring to Figure 5 The picture feature information and audio feature information obtained in steps S502 and S503 described above. Step S703 corresponds to Figure 8 Player parsing after content delivery network distribution as shown in .

[0092] In step S704, based on the feature information, drive strategy information associated with the multimedia content is generated. In the live broadcast scenario, the generation of the drive strategy information is similar to that in the on-demand scenario, except that the content feature information obtained by real-time analysis of the content is not considered. Step S704 corresponds to Figure 8 The drive strategy shown in is generated.

[0093] In step S705, based on the real-time content of the multimedia content and the driving strategy information, a driving signal is generated to drive one or more hardware devices to perform predetermined processing. For example, during a live broadcast of an event, after receiving the I frame and audio from the player, real-time analysis is performed. At the same time, the event events in the live stream are parsed through the player decoding specification. After the event event is parsed, the event event is parsed through the event event mapping table, and the hardware is called through the strategy table. Similarly, after real-time analysis of the I frame and audio, the strategy table is called to call the hardware. The driving signal generated in step S705 is used Figure 8 The driver hardware shown executes the hardware response.

[0094] Fig. 9 2 is a functional block diagram of a hardware device driver strategy generation apparatus based on multimedia content according to an embodiment of the present disclosure. Fig. 9As shown, the multimedia content-based hardware device driving strategy generating apparatus 90 according to the embodiment of the present disclosure includes a feature information acquiring unit 901 and a driving strategy information generating unit 902. Those skilled in the art will readily understand that these unit modules can be implemented in various ways by hardware alone, software alone, or a combination thereof, and the present disclosure is not limited to any of them.

[0095] Specifically, the feature information acquisition unit 901 is configured to analyze multimedia content and acquire feature information of the multimedia content. The feature information acquisition unit 901 analyzes multimedia content, identifies one or more target objects in the multimedia content, and acquires content feature information of the one or more target objects; and / or identifies one or more specific picture frames in the multimedia content, and acquires picture feature information of one or more specific picture frames; and / or identifies an audio signal in the multimedia content, and acquires audio feature information of the audio signal; one or more of the content feature information, picture feature information, and audio feature information are used as feature information. More specifically, the feature information acquisition unit 901 identifies one or more intra-frame coded picture frames in the multimedia content, and determines picture feature information based on the picture change amplitude of multiple intra-frame coded picture frames. The feature information acquisition unit 901 identifies the level value corresponding to the audio signal in the multimedia content, and determines the audio feature information based on the change amplitude of the level value.

[0096] The driving strategy information generating unit 902 is configured to generate driving strategy information associated with the multimedia content based on the feature information. The driving strategy information is used to indicate one or more hardware devices associated with the multimedia content, and to drive the predetermined processing performed by the one or more hardware devices. The driving strategy information generating unit 902 generates the driving strategy information associated with the multimedia content based on one or more of the content feature information, the picture feature information, and the audio feature information. The driving strategy information at least indicates weight information associated with one or more target objects, one or more hardware devices corresponding to the one or more target objects, and the predetermined processing corresponding to the one or more hardware devices.

[0097] Fig.10 1 is a hardware block diagram of an electronic device according to an embodiment of the present disclosure. The electronic device according to an embodiment of the present disclosure includes at least a processor; and a memory for storing computer-readable instructions. When the computer-readable instructions are loaded and executed by the processor, the processor executes the method for generating a hardware device driver strategy based on multimedia content as described above.

[0098] Fig.10The electronic device 1000 shown specifically includes: a central processing unit (CPU) 1001, a graphics processing unit (GPU) 1002 and a memory 1003. These units are interconnected via a bus 1004. The central processing unit (CPU) 1001 and / or the graphics processing unit (GPU) 1002 can be used as the above-mentioned processor, and the memory 1003 can be used as the above-mentioned memory for storing computer-readable instructions. In addition, the electronic device 1000 may also include a communication unit 1005, a storage unit 1006, an output unit 1007, an input unit 1008 and an external device 1009, which are also connected to the bus 1004.

[0099] Fig.11 is a schematic diagram illustrating a computer program product according to an embodiment of the present disclosure. Fig.11 As shown, a computer program product 1100 according to an embodiment of the present disclosure has a computer program 1101 stored thereon. When the computer program 1101 is executed by a processor, the method for generating a hardware device driver strategy based on multimedia content according to an embodiment of the present disclosure described with reference to the above figures is executed. The computer-readable storage medium includes, but is not limited to, for example, a volatile memory and / or a non-volatile memory. The volatile memory may, for example, include a random access memory (RAM) and / or a cache memory (cache), etc. The non-volatile memory may, for example, include a read-only memory (ROM), a hard disk, a flash memory, an optical disk, a magnetic disk, etc.

[0100] The above describes the hardware device driver strategy generation method, device, electronic device, content playback device, storage medium and computer program product according to the embodiments of the present disclosure with reference to the accompanying drawings. The hardware device driver strategy generation method according to the embodiments of the present disclosure obtains feature information of the multimedia content by analyzing the multimedia content, thereby generating driver strategy information associated with the multimedia content based on the feature information, thereby realizing automatic analysis of multimedia content and automatic generation of related hardware device driver strategies in on-demand scenarios and live broadcast scenarios, and providing real-time linkage based on multimedia content. Through comprehensive constraints based on automatic content analysis and policy tables, better control effects are achieved by relying solely on automatic analysis. In addition, high-precision alignment of live events and images is performed to reduce errors when content and hardware are linked.

[0101] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this disclosure.

[0102] The basic principles of the present disclosure are described above in conjunction with specific embodiments. However, it should be noted that the advantages, strengths, effects, etc. mentioned in the present disclosure are only examples and not limitations, and it cannot be considered that these advantages, strengths, effects, etc. are required by each embodiment of the present disclosure. In addition, the specific details disclosed above are only for the purpose of illustration and ease of understanding, and are not limitations. The above details do not limit the present disclosure to the necessity of adopting the above specific details to be implemented.

[0103] The block diagrams of the devices, apparatuses, equipment, and systems involved in this disclosure are only illustrative examples and are not intended to require or imply that they must be connected, arranged, and configured in the manner shown in the block diagrams. As will be appreciated by those skilled in the art, these devices, apparatuses, equipment, and systems can be connected, arranged, and configured in any manner. Words such as "including," "comprising," "having," and the like are open words, referring to "including but not limited to," and can be used interchangeably therewith. The words "or" and "and" used herein refer to the words "and / or," and can be used interchangeably therewith, unless the context clearly indicates otherwise. The word "such as" used herein refers to the phrase "such as but not limited to," and can be used interchangeably therewith.

[0104] Additionally, as used herein, "or" used in a list of items beginning with "at least one" indicates a separate list, so that, for example, a list of "at least one of A, B, or C" means A or B or C, or AB or AC or BC, or ABC (i.e., A and B and C). Furthermore, the word "exemplary" does not mean that the example described is preferred or better than other examples.

[0105] It should also be noted that in the system and method of the present disclosure, each component or each step can be decomposed and / or recombined. Such decomposition and / or recombination should be regarded as equivalent solutions of the present disclosure.

[0106] Various changes, substitutions, and modifications of the techniques described herein may be made without departing from the teachings defined by the appended claims. Furthermore, the scope of the claims of the present disclosure is not limited to the specific aspects of the processes, machines, manufactures, compositions of events, means, methods, and actions described above. Currently existing or later to be developed processes, machines, manufactures, compositions of events, means, methods, or actions that perform substantially the same functions or achieve substantially the same results as the corresponding aspects described herein may be utilized. Thus, the appended claims include such processes, machines, manufactures, compositions of events, means, methods, or actions within their scope.

[0107] The above description of the disclosed aspects is provided to enable any person skilled in the art to make or use the present disclosure. Various modifications to these aspects will be readily apparent to those skilled in the art, and the general principles defined herein may be applied to other aspects without departing from the scope of the present disclosure. Therefore, the present disclosure is not intended to be limited to the aspects shown herein, but rather to the widest scope consistent with the principles and novel features disclosed herein.

[0108] The above description has been given for the purpose of illustration and description. In addition, this description is not intended to limit the embodiments of the present disclosure to the forms disclosed herein. Although multiple example aspects and embodiments have been discussed above, those skilled in the art will recognize certain variations, modifications, changes, additions and sub-combinations thereof.

Claims

1. A method for generating a hardware device driver strategy based on multimedia content, characterized in that: The method comprises: Analyzing the multimedia content to obtain feature information of the multimedia content; Based on the feature information, generating driving strategy information associated with the multimedia content; The driving strategy information is used to indicate one or more hardware devices associated with the multimedia content, and to drive the one or more hardware devices to perform a predetermined process.

2. The method for generating a hardware device driver strategy based on multimedia content according to claim 1, characterized in that: The step of analyzing the multimedia content to obtain feature information of the multimedia content includes: identifying one or more target objects in the multimedia content, and acquiring content feature information of the one or more target objects; and / or, Identifying one or more specific picture frames in the multimedia content, and acquiring picture feature information of the one or more specific picture frames, wherein the picture feature information includes picture feature information determined based on picture change amplitudes of multiple intra-frame coded picture frames in the one or more specific picture frames; and / or, Identifying an audio signal in the multimedia content, and acquiring audio feature information of the audio signal, wherein the audio feature information includes audio feature information determined based on a change amplitude of a level value corresponding to the audio signal in the multimedia content; One or more of the content feature information, the picture feature information and the audio feature information is used as the feature information.

3. The method for generating a hardware device driver strategy based on multimedia content according to claim 2, characterized in that: The step of generating driving strategy information associated with the multimedia content based on the feature information includes: Generate driving strategy information associated with the multimedia content based on one or more of the content feature information, the picture feature information, and the audio feature information; The driving strategy information at least indicates weight information associated with the one or more target objects, the one or more hardware devices corresponding to the one or more target objects, and the predetermined processing corresponding to the one or more hardware devices.

4. The method for generating a hardware device driver strategy based on multimedia content according to any one of claims 1 to 3, characterized in that: The method further comprises: Based on the driving strategy information, a script file is generated, wherein the script file is used to instruct to drive the one or more hardware devices to perform a predetermined process at a predetermined time point; Based on the playback progress of the multimedia content and the script file, a driving signal for driving the one or more hardware devices to perform a predetermined process is generated.

5. The method for generating a hardware device driver strategy based on multimedia content according to any one of claims 1 to 3, characterized in that: The method further comprises: Based on the real-time content of the multimedia content and the driving strategy information, a driving signal for driving the one or more hardware devices to perform a predetermined process is generated.

6. The method for generating a hardware device driver strategy based on multimedia content according to any one of claims 1 to 3, characterized in that: The method further comprises: In the case where the multimedia content is generated based on live streaming data, before analyzing the multimedia content, the time difference between the multimedia content and the live streaming data is determined for the same picture frame; and based on the time difference, event information for indicating a predetermined event is added to the live streaming data.

7. The method for generating a hardware device driver strategy based on multimedia content according to claim 6, characterized in that: The determining the time difference between the multimedia content and the live streaming data includes: Generate synchronization data consistent with the live streaming data; Identify first event time information of a first event in the synchronization data, and determine first time difference information based on the first event time information and second event time information of the first event marked in the live streaming data; and / or, Determine second time difference information based on first picture time information corresponding to the first picture of the synchronization data and second picture time information corresponding to the first picture of the live streaming data; and / or, Determine third time difference information based on the marking time information of the first event in the live streaming data and the display time information of the first event; The time difference is determined based on one or more of the first time difference information, the second time difference information, and the third time difference information.

8. An electronic device, characterized in that: include: a memory for storing computer readable instructions; as well as A processor is used to run the computer-readable instructions so that the electronic device executes the method for generating a hardware device driver strategy based on multimedia content as described in any one of claims 1 to 7.

9. A non-transitory computer-readable storage medium for storing computer-readable instructions, characterized in that: When the computer-readable instructions are executed by a processor, the processor is caused to execute the method for generating a hardware device driver strategy based on multimedia content according to any one of claims 1 to 7.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the method for generating a hardware device driver strategy based on multimedia content according to any one of claims 1 to 7 is implemented.

Images