Hardware device driving strategy generation method, device, storage medium and program product

Abstract

The present disclosure relates to the technical field of artificial intelligence, and particularly provides a hardware device driving strategy generation method, device, storage medium and program product. The hardware device driving strategy generation method based on multimedia content comprises: analyzing multimedia content to obtain characteristic information of the multimedia content; and generating driving strategy information associated with the multimedia content based on the characteristic information; wherein the driving strategy information is used to indicate one or more hardware devices associated with the multimedia content and a predetermined processing performed by the one or more hardware devices. According to the hardware device driving strategy generation method, automatic analysis of the multimedia content and automatic generation of the related hardware device driving strategy in an on-demand scenario and a live scenario are realized, and real-time linkage based on the multimedia content is provided.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to the technical field of artificial intelligence, and in particular, to a hardware device driving strategy generation method, device, storage medium and program product. BACKGROUND

[0002] With the development of IT and artificial intelligence technology, multimedia content is further integrated with other hardware devices besides the display device itself to provide users with a more rich experience. Devices such as smart cockpits and multimedia entertainment cockpits can not only provide video and audio reproduction of multimedia content, but also can cooperate with the reproduced content to provide visual, auditory and tactile related outputs through other hardware devices.

[0003] Currently, the association of multimedia content with other hardware devices mainly relies on manual dot annotation in the multimedia content, resulting in additional labor costs. In addition, for multimedia content in a live scenario, real-time annotation cannot be performed, so that real-time linkage based on multimedia content cannot be provided. SUMMARY

[0004] The present disclosure is proposed in view of the above problems. The present disclosure provides a hardware device driving strategy generation method, device, electronic device, smart cockpit, storage medium and computer program product.

[0005] According to one aspect of the present disclosure, a hardware device driving strategy generation method based on multimedia content is provided, comprising: analyzing the multimedia content to obtain feature information of the multimedia content; and generating driving strategy information associated with the multimedia content based on the feature information; wherein the driving strategy information is used to indicate one or more hardware devices associated with the multimedia content and a predetermined processing performed by the one or more hardware devices.

[0006] In addition, according to the hardware device driving strategy generation method based on multimedia content of one aspect of the present disclosure, the multimedia content is analyzed to obtain the feature information of the multimedia content, comprising: identifying one or more target objects in the multimedia content, and obtaining content feature information of the one or more target objects; and / or, identifying one or more specific picture frames in the multimedia content, and obtaining picture feature information of the one or more specific picture frames, wherein the picture feature information comprises picture feature information determined based on a picture change amplitude of a plurality of intra-coded picture frames in the one or more specific picture frames; and / or, identifying an audio signal in the multimedia content, and obtaining audio feature information of the audio signal, wherein the audio feature information comprises audio feature information determined based on a change amplitude of a level value corresponding to the audio signal in the multimedia content; and one or more of the content feature information, the picture feature information and the audio feature information is used as the feature information.

[0007] Further, the method for generating a hardware device driving strategy based on multimedia content according to an aspect of the present disclosure further comprises: generating a script file based on the driving strategy information, the script file being used to instruct the one or more hardware devices to perform the predetermined processing at a predetermined time point; and generating a driving signal for driving the one or more hardware devices to perform the predetermined processing based on the playing progress of the multimedia content and the script file.

[0008] Further, the method for generating a hardware device driving strategy based on multimedia content according to an aspect of the present disclosure further comprises: generating a driving signal for driving the one or more hardware devices to perform the predetermined processing based on the real-time content of the multimedia content and the driving strategy information.

[0009] Further, the method for generating a hardware device driving strategy based on multimedia content according to an aspect of the present disclosure further comprises: in the case that the multimedia content is based on live streaming data, determining a time difference between the multimedia content and the live streaming data for the same picture frame before analyzing the multimedia content; and adding event information for indicating the predetermined event in the live streaming data based on the time difference.

[0010] Further, the method for generating a hardware device driving strategy based on multimedia content according to an aspect of the present disclosure, the determining of the time difference between the multimedia content and the live streaming data comprises: generating synchronization data consistent with the live streaming data; identifying first event time information of a first event in the synchronization data, determining first time difference information based on the first event time information and second event time information of the first event marked in the live streaming data; and / or determining second time difference information based on first picture time information corresponding to a first picture in the synchronization data and second picture time information corresponding to the first picture in the live streaming data; and / or determining third time difference information based on marking time information of the first event marked in the live streaming data and display time information of the first event; and determining the time difference based on one or more of the first time difference information, the second time difference information and the third time difference information.

[0011] According to another aspect of the present disclosure, a device for generating a hardware device driving strategy based on multimedia content is provided, comprising: a feature information acquisition unit configured to analyze the multimedia content and acquire feature information of the multimedia content; and a driving strategy information generation unit configured to generate driving strategy information associated with the multimedia content based on the feature information, wherein the driving strategy information is used to indicate one or more hardware devices associated with the multimedia content and predetermined processing performed by the one or more hardware devices.

[0012] According to still another aspect of the present disclosure, an electronic device is provided, comprising: a memory configured to store computer readable instructions; and a processor configured to execute the computer readable instructions, so that the electronic device performs the multimedia content-based hardware device driving strategy generation method as above.

[0013] According to yet another aspect of the present disclosure, a content playing apparatus is provided, comprising: a display device configured to play multimedia content; one or more hardware devices configured to perform predetermined processing corresponding to the multimedia content; and a hardware device driving device configured to analyze the multimedia content, obtain feature information of the multimedia content; based on the feature information, generate driving strategy information associated with the multimedia content; wherein the driving strategy information is used to indicate one or more hardware devices associated with the multimedia content, and the predetermined processing performed by the one or more hardware devices.

[0014] According to yet another aspect of the present disclosure, a non-transitory computer readable storage medium is provided, configured to store computer readable instructions, when the computer readable instructions are executed by a processor, the processor performs the multimedia content-based hardware device driving strategy generation method as above.

[0015] According to yet another aspect of the present disclosure, a computer program product is provided, comprising a computer program, characterized in that the computer program is executed by a processor to implement the multimedia content-based hardware device driving strategy generation method as above.

[0016] As will be described in detail below, the hardware device driving strategy generation method, apparatus, electronic device, intelligent cockpit, storage medium and computer program product according to embodiments of the present disclosure, by analyzing the multimedia content, obtaining the feature information of the multimedia content, and based on the feature information, generating the driving strategy information associated with the multimedia content, realizes automatic analysis of the multimedia content and automatic generation of the related hardware device driving strategy in the on-demand scenario and the live scenario, and provides a real-time linkage scheme based on the multimedia content.

[0017] It is to be understood that both the foregoing general description and the following detailed description are exemplary, and are intended to provide further explanation of the subject technology. BRIEF DESCRIPTION OF DRAWINGS

[0018] The foregoing and other objects, features, and advantages of the present disclosure will become more apparent from the following detailed description, which proceeds with reference to the accompanying drawings. The drawings are intended to provide a further understanding, but are not intended for limitation of the present disclosure. The drawings illustrate embodiments of the present disclosure and, together with their description, serve to explain the present disclosure. In the drawings:

[0019] Figure 1A and Figure 1B is a schematic diagram illustrating a content playing device applying the multimedia content-based hardware device driving strategy generation method according to the embodiments of the present disclosure.

[0020] Figure 2 is a flow chart illustrating the multimedia content-based hardware device driving strategy generation method according to the embodiments of the present disclosure.

[0021] Figure 3 is a flow chart illustrating the multimedia content-based hardware device driving strategy generation method according to the embodiments of the present disclosure.

[0022] Figure 4 is a flow chart illustrating the implementation process of the multimedia content-based hardware device driving strategy generation method according to the embodiments of the present disclosure.

[0023] Figure 5 is a flow chart further illustrating the feature information acquisition process in the multimedia content-based hardware device driving strategy generation method according to the embodiments of the present disclosure.

[0024] Figure 6 is a flow chart further illustrating the multimedia content-based hardware device driving strategy generation method according to the embodiments of the present disclosure in the on-demand scenario.

[0025] Figure 7 is a flow chart further illustrating the multimedia content-based hardware device driving strategy generation method according to the embodiments of the present disclosure in the live scenario.

[0026] Figure 8 is a flow chart further illustrating the multimedia content-based hardware device driving strategy generation method according to the embodiments of the present disclosure in the live scenario.

[0027] Figure 9 is a functional block diagram illustrating the multimedia content-based hardware device driving strategy generation apparatus according to the embodiments of the present disclosure.

[0028] Figure 10 is a hardware block diagram illustrating an electronic device according to the embodiments of the present disclosure.

[0029] Figure 11 is a schematic diagram illustrating a computer program product according to the embodiments of the present disclosure. DETAILED DESCRIPTION

[0030] In order to make the objectives, technical solutions and advantages of the present disclosure more apparent, the following will describe the example embodiments according to the present disclosure in detail with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present disclosure, rather than all the embodiments of the present disclosure, and it should be understood that the present disclosure is not limited to the example embodiments described herein.

[0031] First, refer to Figure 1A and Figure 1B The content playback device applying the multimedia content-based hardware device driving strategy generation method according to the embodiments of the present disclosure is described. Figure 1A and Figure 1B The content playback devices 10A and 10B are respectively shown.

[0032] Specifically, as Figure 1A indicated, the content playback device 10A may, for example, be implemented as or as a component of an intelligent cabin of a vehicle. The content playback device 10A at least includes a display device 101A for playing multimedia content. The content playback device 10A further includes one or more hardware devices (not shown) for performing predetermined processing corresponding to the multimedia content. In the application scenario of the intelligent cabin, the one or more hardware devices may, for example, be lighting devices, audio devices, seat devices, air conditioning devices, and fragrance devices of the vehicle. In addition, the content playback device 10A further includes a hardware device driving device (not shown) for analyzing the multimedia content, obtaining feature information of the multimedia content; based on the feature information, generating driving strategy information associated with the multimedia content. In the application scenario of the intelligent cabin, the hardware device driving device may, for example, be a special or general control chip, processor configured in the vehicle. The driving strategy information is used to indicate one or more hardware devices associated with the multimedia content, and the predetermined processing performed by driving the one or more hardware devices. In the application scenario of the intelligent cabin, the driving strategy information is used to, for example, indicate the lighting devices (such as interior ambient light, exterior near, high beam, etc.) to switch the lighting color, mode and brightness; indicate the audio devices to perform amplification and switching of different sound channels; indicate the seat devices to perform vibration, massage, ventilation and heating, etc.; indicate the air conditioning devices to perform temperature setting, air speed setting and air purification, etc.; and indicate the fragrance devices to perform fragrance concentration and type setting, etc.

[0033] Similarly, as Figure 1BThe content playback device 10B shown can be implemented as or as a component of a smart multimedia entertainment cabin. The content playback device 10B at least includes a display device 101B for playing multimedia content. The content playback device 10B further includes one or more hardware devices (not shown) for performing predetermined processing corresponding to the multimedia content. In the application scenario of the smart multimedia entertainment cabin, the one or more hardware devices can be, for example, lighting devices, sound devices, seat devices, air conditioning devices, and fragrance devices. In addition, the content playback device 10B further includes a hardware device driving device (not shown) for analyzing the multimedia content, obtaining feature information of the multimedia content, and generating driving policy information associated with the multimedia content based on the feature information. In the application scenario of the smart multimedia entertainment cabin, the hardware device driving device can be, for example, a special or general control chip or processor configured in the smart multimedia entertainment cabin. The driving policy information is used to indicate one or more hardware devices associated with the multimedia content and predetermined processing performed by driving the one or more hardware devices. In the application scenario of the smart multimedia entertainment cabin, the driving policy information is used to, for example, instruct the lighting device to switch the lighting color, mode, and brightness; instruct the sound device to perform amplification and switching of different sound channels; instruct the seat device to perform vibration, massage, ventilation, and heating; instruct the air conditioning device to perform temperature setting, air speed setting, and air purification; and instruct the fragrance device to perform fragrance concentration and type setting.

[0034] In the following, the multimedia content-based hardware device driving policy generation method according to an embodiment of the present disclosure will be described in detail with reference to Figures 2 to 8 The multimedia content-based hardware device driving policy generation method according to an embodiment of the present disclosure will be described in detail.

[0035] As shown in Figure 2 The multimedia content-based hardware device driving policy generation method according to an embodiment of the present disclosure includes the following steps.

[0036] In step S201, the multimedia content is analyzed to obtain feature information of the multimedia content. In an embodiment of the present disclosure, the feature information of the multimedia content includes content feature information obtained by analyzing the picture content. In an embodiment of the present disclosure, the feature information of the multimedia content further includes picture feature information and audio feature information obtained without depending on the specific content of the picture. One or more of the content feature information, the picture feature information, and the audio feature information is used as the feature information. In the following, the feature information obtaining process will be described in detail with reference to Figure 5 The feature information obtaining process will be described in detail.

[0037] In step S202, driving policy information associated with the multimedia content is generated based on the feature information. The driving policy information is used to indicate one or more hardware devices associated with the multimedia content and predetermined processing performed by driving the one or more hardware devices.

[0038] In one embodiment of the present disclosure, the driving strategy information associated with the multimedia content is generated based on one or more of the content feature information, the picture feature information, and the audio feature information. The driving strategy information indicates at least the weight information associated with one or more target objects in the multimedia content, one or more hardware devices corresponding to the one or more target objects, and a predetermined processing corresponding to the one or more hardware devices. In the following, the specific process of the hardware device driving strategy generation method in the on-demand scenario and the live scenario will be described in detail. Figure 6 and Figure 7 The specific process of the hardware device driving strategy generation method in the on-demand scenario and the live scenario will be described in detail.

[0039] Figure 3 is a flowchart illustrating the hardware device driving strategy generation method based on multimedia content according to an embodiment of the present disclosure.

[0040] As shown in Figure 3 , the first stage performs content understanding, i.e., corresponding to step S201 described above. As shown in Figure 3 , the content understanding includes analysis of the picture and the audio, realizing shot segmentation, person recognition, object recognition, emotion classification, scene recognition, optical flow algorithm, main color extraction, sound classification, etc., so as to give automatic labels to the recognized one or more target objects, such as action, scene, object, atmosphere, theme color, emotion, etc.

[0041] The second stage performs strategy table generation, i.e., corresponding to step S202 described above. Based on the analysis results of the first stage, the driving strategies in multiple aspects such as vision, hearing, touch, smell, etc. are generated.

[0042] The third stage performs hardware linkage, i.e., driving the associated one or more hardware devices to perform the predetermined processing based on the driving strategies generated after the previous two stages. Figure 3 Specifically, in the application scenario of the intelligent cockpit, the atmosphere lamp device switches the lighting color, mode and brightness; the power amplifier device performs amplification and switching of different sound channels; the seat device performs vibration, massage, ventilation and heating, etc.; the air conditioner device performs temperature setting, wind speed setting and air purification, etc.; and the fragrance device performs fragrance concentration and type setting, etc.

[0043] Figure 4 is a flowchart illustrating the hardware device driving strategy generation method based on multimedia content according to an embodiment of the present disclosure.

[0044] As shown in Figure 4As shown, multimedia content such as video, game, music, etc. has content tags, picture information and audio information into middleware. It needs to be understood that the middleware is a bridge to establish linkage between the content and the hardware, and can call the hardware driver interface according to the content analysis result. In addition, the middleware can also adapt to different systems, and realize the linkage of the same content with the hardware on different system devices (for example, different vehicle models). Specifically, the middleware is connected with the software development kit (SDK) provided by each system device manufacturer (for example, vehicle manufacturer) which can call hardware devices, to realize the operation of the application program interface (API) of the hardware. The calling method, calling function and value method of different device manufacturers for the same hardware are different. Therefore, for the same hardware of different device manufacturers, the calling method needs to be unified to adapt to each different hardware. In addition, the value of the same device needs to be normalized. For example, the calling method of the air conditioner fan speed of vehicle A is HVAC_FUNC_FAN_SPEED_HARD_KEY, and the value range is 0-9. The calling method of the air conditioner fan speed of vehicle B is AC_FUNC_FAN_SPEED_HARD_KEY, and the value range is 0-100. Therefore, the calling method of vehicle A and vehicle B needs to be mapped, and the value is normalized to 0-100. In addition, since the systems and system versions of different device manufacturers are inconsistent, the middleware also needs to adapt to the system and hardware, so as to normally run on each hardware system.

[0045] Figure 5 is a further diagram illustrating the feature information acquisition process in the multimedia content-based hardware device driving strategy generation method according to an embodiment of the present disclosure.

[0046] In step S501, one or more target objects in the multimedia content are identified, and content feature information of the one or more target objects is acquired.

[0047] In an embodiment of the present disclosure, the content feature information depends on the content itself, and usually needs to be pre-executed. That is, in the scenario of playing existing multimedia content, the content feature information can be acquired in advance, while in the real-time live scenario, the acquisition of the content feature information is usually not executed, so as to ensure the real-time of the live. In the driving strategy information generated subsequently, the target objects with different content feature information correspond to different driving strategies.

[0048] In step S502, one or more specific picture frames in the multimedia content are identified, and picture feature information of the one or more specific picture frames is acquired.

[0049] In one embodiment of the disclosure, the one or more specific picture frames are one or more intra-coded picture frames (I-frames). The multimedia content can be read frame by frame or jump-read, and the encoding format of the multimedia content is converted into frame images. By judging the frame type or checking the key frame information in the frame data structure, the I-frames are found, which are the largest among all the frames. The decoded multimedia content is traversed to find all the I-frames. Further, the picture feature information is determined based on the picture change amplitude of the one or more intra-coded picture frames (I-frames). For example, the first and second dominant colors of the I-frames are calculated by clustering method. The change amplitude between each frame of picture is calculated by using optical flow method. Different dominant colors and different change amplitudes can correspond to different driving strategies in the subsequently generated driving strategy information.

[0050] In step S503, the audio signal in the multimedia content is identified, and the audio feature information of the audio signal is obtained.

[0051] In one embodiment of the disclosure, the level value corresponding to the audio signal in the multimedia content is identified, and the audio feature information is determined based on the change amplitude of the level value. More specifically, the level value of the audio signal is obtained by measuring the average power or amplitude of the audio signal. According to the change amplitude of the level value, classification is performed, including:

[0052] 1). Segment the audio signal: divide the audio signal into several segments, each of which should be short enough to reflect the trend of level change. Window function or other segmentation methods can be used for segmentation.

[0053] 2). Calculate the root mean square (RMS) level or peak level of each segment: for each segment of signal, calculate their RMS level or peak level, and store them as a list.

[0054] 3). Determine the threshold: by analyzing the list of level values, determine which range of level change is defined as "flat", which range should be defined as "small change", which range should be defined as "large change", etc.

[0055] 4). Classification according to the threshold: using the threshold determined above, classify each segment of signal as "flat", "small change" or "large change", etc.

[0056] Different change amplitude types correspond to different driving strategies in the subsequently generated driving strategy information.

[0057] In addition, automatic classification is performed according to the amplitude and frequency of level change, including:

[0058] 1). Calculate the average level and change range of each segment of signal: for each segment of signal, calculate their average level and range, such as standard deviation or range, etc.

[0059] 2). Clustering each segment of signal: Each segment of signal is taken as a data point, and clustering algorithm is used to cluster the data. The number of cluster centers can be set according to actual needs, and different cluster numbers can represent different level change amplitudes.

[0060] 3). Determining the frequency in each category: After each data point is assigned to a cluster center, the frequency of the category represented by each cluster center can be calculated.

[0061] 4). Classifying the level according to the clustering and frequency analysis: By synthesizing the clustering results and frequency analysis, the level of the audio signal can be classified into different categories.

[0062] Different frequency types correspond to different driving strategies in the subsequently generated driving strategy information. For example, when the least common level appears, the seat can be driven, and when the most common level appears, the atmosphere lamp can be driven.

[0063] In step S504, one or more of the content feature information, the picture feature information and the audio feature information are taken as the feature information.

[0064] In one embodiment of the present disclosure, depending on whether the multimedia content is on-demand or live, whether the multimedia content is video or audio, one or more of the content feature information, the picture feature information and the audio feature information are obtained as the corresponding feature information.

[0065] Figure 6 is a flowchart further illustrating the method for generating the driving strategy of the hardware device based on the multimedia content according to the embodiment of the present disclosure in the on-demand scenario.

[0066] In step S601, the multimedia content is analyzed to obtain the feature information of the multimedia content. In the on-demand scenario, the feature information of the multimedia content includes the content feature information obtained by analyzing the picture content, the picture feature information obtained without relying on the specific content of the picture, and the audio feature information. The method for obtaining the content feature information, the picture feature information and the audio feature information is as described above with reference to Figure 5 , which will not be described again here.

[0067] In step S602, based on the feature information, the driving strategy information associated with the multimedia content is generated. The driving strategy information is used to indicate one or more hardware devices associated with the multimedia content and the predetermined processing performed by driving the one or more hardware devices.

[0068] Table 1 schematically shows a table of the driving strategy information associated with the multimedia content generated based on the feature information of the multimedia content.

[0069]

[0070] Table 1

[0071] The data types in Table 1 include audio types and video types. The data classification and secondary classification are determined through analysis of content feature information. For example, the data is classified into people and objects, and the people are further classified into nurses, soldiers, firefighters, etc., and the objects further include flowers.

[0072] The weighting type values in Table 1 are 1 and 2. 1 represents weighting processing, and 2 represents independent weight processing.

[0073] In the case of weighting processing, when the corresponding device is effective, the device value is in the range type (e.g., 1-20, 0%-100%), and the value range is superimposed (cannot exceed the maximum value).

[0074] For example, if the weighting type value of a nurse is 1 and the value range is 5%, the final device value range is n%+5%.

[0075] In the case of independent weight processing, all objects with a weighting type value of 2 are executed according to the highest weight if the same hardware device is called. When a content has multiple classifications and simultaneously drives a hardware device, only the hardware device with the highest weight is effective. When the weighting type value is 1, the weight is not effective.

[0076] The mutual exclusion relationship in Table 1 indicates that the same value is mutually exclusive and can only be effective for one. The order of effectiveness can be random or in the order of top to bottom on the strategy table.

[0077] The capability Key in Table 1 indicates the object that calls the hardware device. The capability Key of different hardware devices can be different and needs to be mapped for compatibility with different device platforms.

[0078] The capability Value in Table 1 indicates the range of the input parameter of the hardware device after being called. Different car machines can have the same hardware but different values, which need to be normalized.

[0079] The start time and end time in Table 1 represent the time range for driving the hardware device.

[0080] In step S603, a script file is generated based on the driving strategy information. The script file is used to instruct driving the one or more hardware devices to perform the predetermined processing at a predetermined time point.

[0081] In step S604, a driving signal for driving the one or more hardware devices to perform the predetermined processing is generated based on the play progress of the multimedia content and the script file.

[0082] Specifically, in the on-demand scenario, when playing the I-frame and the audio, real-time analysis is performed, and the script file is parsed based on the current playing progress to determine the driving strategy of the corresponding time point, thereby generating a driving signal for driving one or more hardware devices to perform the predetermined processing.

[0083] Figure 7 is a flowchart further illustrating the method for generating a hardware device driving strategy based on multimedia content according to an embodiment of the present disclosure in a live streaming scenario. Figure 8 is a flowchart further illustrating the method for generating a hardware device driving strategy based on multimedia content according to an embodiment of the present disclosure in a live streaming scenario.

[0084] In step S701, the time difference between the multimedia content and the live streaming data is determined.

[0085] Referring to Figure 8 Step S701 is described. In the live streaming scenario, the annotation of the predetermined event in the live streaming may be performed manually at the live streaming site before the live streaming accesses the content distribution network. The annotation is played on the user player end via transcoding and distribution of the predetermined event in the live streaming. The time point of the annotated predetermined event in the live streaming and the time point of the occurrence of the predetermined event in the display screen may have a time difference. In addition, during the transcoding process, the process of pressing the event information into the multimedia content through the supplementary enhancement information may also cause a time difference.

[0086] Specifically, as Figure 8 indicated, the synchronization data consistent with the live streaming data is generated. The first event time information of the first event in the synchronization data is identified, and the first time difference information is determined based on the first event time information and the second event time information of the first event annotated in the live streaming data. For example, in the case of a live streaming competition event, the score or event in the event screen is identified using OCR, motion recognition, object recognition, etc. The score in the current screen is obtained through the timing scanning of the OCR, and the time of the score change is recorded, which is set as T utc . Similarly, the motion recognition and object recognition mainly identify the event in the event, and determine the time T utc of the occurrence of the event. T utc is the first event time information of the first event. From the event data of the event, the detailed information of the event is obtained, which includes the event name, the score, the event object, and the time t utc of the occurrence of the event, which is set as t utc is the second event time information of the first event annotated in the live streaming data. Thereafter, T utc and t utc of the same event are found. For example, when T utcAfterwards, the corresponding event in the current event data is queried, and then the same event is matched. Similarly, when using OCR to identify the score in the picture, the score can be matched, such as: the current picture changes from 2:3 to 2:4, and then the t utc time of the first picture corresponding to the first picture in the event data is determined. utc The time difference between t utc and t utc is the first time difference information t1=T utc .

[0087] In addition, based on the first picture time information corresponding to the first picture of the synchronization data and the second picture time information corresponding to the first picture of the live stream data, the second time difference information is determined. The synchronization stream is a copy of the source stream (i.e. live stream), and there may also be a time error between the two. In the same machine room, the network architecture does not change, and this time error is fixed. For example, the pictures of the live stream and the synchronization stream are taken at the same time, and the difference between the same pictures is the second time difference information t2. In the live stream, the SEI number is pressed (pressed in numerical incremental order), and when decoded again, the time of the same number is recorded. The time difference between the same number is the second time difference information t2.

[0088] In addition, based on the first event annotation time information in the live stream data and the display time information of the first event, the third time difference information is determined. During transcoding, the time required to press the event information into the live stream, and this time error is also fixed. The number is pressed by SEI (pressed in numerical incremental order, interval pressed), and the UTC timestamp is also brought in at the same time. 3) When the number in the video picture changes, the current UTC timestamp is recorded (for example, the shooting device can be used to record). Calculate the difference between the SEI pressed timestamp (i.e. the first event annotation time information) and the number change timestamp in the picture (i.e. the display time information of the first event), which is the third time difference information t3.

[0089] Based on one or more of the first time difference information t1, the second time difference information t2, and the third time difference information t3, the time difference is determined. For example, the time difference t=t1+t2+t3.

[0090] In step S702, event information indicating a predetermined event is added to the live streaming data based on the time difference. As described above, the time difference between the multimedia content and the live streaming data is determined in step S701. This time difference is taken into account when the event information is pushed in through the SEI information, and a certain delay is performed so that the occurrence time of the predetermined event during the multimedia content playback is aligned as much as possible with the time indicated by the event information. In a live broadcast scenario, since the content characteristics of the multimedia content are generally not analyzed, the predetermined time point for driving the hardware is determined by reference to the event information pushed into the live stream. Therefore, after compensating for the time difference between the multimedia content and the live streaming data, the time for driving the hardware will be more accurate.

[0091] In step S703, the multimedia content is analyzed to obtain the characteristic information of the multimedia content. In the live broadcast scenario, the characteristic information of the multimedia content is mainly obtained by referring to Figure 5 The picture feature information and audio feature information obtained in steps S502 and S503 described above. Step S703 corresponds to Figure 8 Player parsing after content delivery network distribution as shown in .

[0092] In step S704, based on the feature information, driving strategy information associated with the multimedia content is generated. In the live broadcast scenario, the generation of driving strategy information is similar to that in the on-demand scenario, except that the content feature information obtained by real-time analysis of the content is not considered. Step S704 corresponds to Figure 8 The drive strategy shown in is generated.

[0093] In step S705, based on the real-time content of the multimedia content and the driving strategy information, a driving signal is generated to drive one or more hardware devices to perform predetermined processing. For example, during a live broadcast of an event, after receiving the I frame and audio from the player, real-time analysis is performed. At the same time, the event events in the live stream are parsed through the player decoding specifications. After the event event is parsed, the event event is parsed through the event event mapping table, and the hardware is called through the strategy table. Similarly, after real-time analysis of the I frame and audio, the strategy table is called to call the hardware. The driving signal generated in step S705 is used Figure 8 The driver hardware shown executes the hardware response.

[0094] Figure 9 FIG is a functional block diagram illustrating a hardware device driver strategy generation apparatus based on multimedia content according to an embodiment of the present disclosure. Figure 9As shown, the multimedia content-based hardware device driving strategy generation apparatus 90 according to the embodiment of the present disclosure comprises a feature information acquisition unit 901 and a driving strategy information generation unit 902. Those skilled in the art can easily understand that these unit modules can be realized in various manners by hardware alone, by software alone, or by a combination thereof, and the present disclosure is not limited to any one of them.

[0095] Specifically, the feature information acquisition unit 901 is configured to analyze the multimedia content and acquire feature information of the multimedia content. The feature information acquisition unit 901 analyzes the multimedia content, identifies one or more target objects in the multimedia content, and acquires content feature information of the one or more target objects; and / or identifies one or more specific picture frames in the multimedia content, and acquires picture feature information of the one or more specific picture frames; and / or identifies an audio signal in the multimedia content, and acquires audio feature information of the audio signal; and takes one or more of the content feature information, the picture feature information, and the audio feature information as the feature information. More specifically, the feature information acquisition unit 901 identifies one or more intra-coded picture frames in the multimedia content, and determines the picture feature information based on a picture variation amplitude of the plurality of intra-coded picture frames. The feature information acquisition unit 901 identifies a level value corresponding to the audio signal in the multimedia content, and determines the audio feature information based on a variation amplitude of the level value.

[0096] The driving strategy information generation unit 902 is configured to generate driving strategy information associated with the multimedia content based on the feature information. The driving strategy information is used to indicate one or more hardware devices associated with the multimedia content and a predetermined processing performed by the one or more hardware devices. The driving strategy information generation unit 902 generates the driving strategy information associated with the multimedia content based on one or more of the content feature information, the picture feature information, and the audio feature information, and the driving strategy information at least indicates weight information associated with the one or more target objects, one or more hardware devices corresponding to the one or more target objects, and a predetermined processing corresponding to the one or more hardware devices.

[0097] Figure 10 is a hardware block diagram illustrating an electronic device according to an embodiment of the present disclosure. The electronic device according to the embodiment of the present disclosure at least comprises a processor; and a memory for storing computer readable instructions. When the computer readable instructions are loaded and run by the processor, the processor performs the multimedia content-based hardware device driving strategy generation method as described above.

[0098] Figure 10The electronic device 1000 shown specifically includes: a central processing unit (CPU) 1001, a graphics processing unit (GPU) 1002, and a memory 1003. These units are interconnected via a bus 1004. The central processing unit (CPU) 1001 and / or the graphics processing unit (GPU) 1002 can be used as the above-mentioned processor, and the memory 1003 can be used as the above-mentioned memory for storing computer-readable instructions. In addition, the electronic device 1000 may also include a communication unit 1005, a storage unit 1006, an output unit 1007, an input unit 1008, and an external device 1009, which are also connected to the bus 1004.

[0099] Figure 11 Schematic diagram illustrating a computer program product according to an embodiment of the present disclosure. Figure 11 As shown, a computer program product 1100 according to an embodiment of the present disclosure has a computer program 1101 stored thereon. When the computer program 1101 is executed by a processor, the method for generating a hardware device driver strategy based on multimedia content according to an embodiment of the present disclosure described with reference to the above figures is executed. The computer-readable storage medium includes, but is not limited to, volatile memory and / or non-volatile memory. The volatile memory may, for example, include a random access memory (RAM) and / or a cache memory. The non-volatile memory may, for example, include a read-only memory (ROM), a hard disk, a flash memory, an optical disk, a magnetic disk, etc.

[0100] The above describes the hardware device driver strategy generation method, device, electronic device, content playback device, storage medium and computer program product according to the embodiment of the present disclosure with reference to the accompanying drawings. The hardware device driver strategy generation method according to the embodiment of the present disclosure obtains feature information of the multimedia content by analyzing the multimedia content, and then generates driver strategy information associated with the multimedia content based on the feature information, thereby realizing automatic analysis of multimedia content and automatic generation of related hardware device driver strategies in on-demand scenarios and live broadcast scenarios, and providing real-time linkage based on multimedia content. Through the comprehensive constraints based on content automatic analysis and policy tables, a better control effect is achieved by relying solely on automatic analysis. In addition, high-precision alignment of live events and images is performed to reduce errors when content and hardware are linked.

[0101] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this disclosure.

[0102] The above describes the basic principles of the present disclosure in conjunction with specific embodiments, but it should be noted that the advantages, benefits, effects and the like mentioned in the present disclosure are only examples and are not limiting, and these advantages, benefits, effects and the like cannot be considered as necessary for each embodiment of the present disclosure. In addition, the specific details of the above disclosure are only for the purpose of example and for the purpose of understanding, and are not limiting, and the above details do not limit the present disclosure to be necessarily implemented with the above specific details.

[0103] The block diagrams of the devices, apparatuses, equipment, systems involved in the present disclosure are only illustrative examples and are not intended to require or imply the connection, arrangement, configuration shown in the block diagram. As those skilled in the art will recognize, these devices, apparatuses, equipment, systems can be connected, arranged, configured in any manner. Words such as "include", "contain", "have" and the like are open-ended words, meaning "including but not limited to", and can be used interchangeably. The words "or" and "and" used herein mean the word "and / or", and can be used interchangeably unless the context clearly indicates otherwise. The word "such as" used herein means the phrase "such as but not limited to", and can be used interchangeably.

[0104] In addition, as used herein, "or" used in the list of items in the phrase "at least one of the items" indicates a disjunctive list such that, for example, a list of "at least one of A, B, or C" means A or B or C or AB or AC or BC or ABC (i.e., A and B and C). In addition, the word "exemplary" does not mean that the described example is preferred or better than other examples.

[0105] It should also be noted that in the systems and methods of the present disclosure, the components or steps can be decomposed and / or recombined. These decompositions and / or recombinations should be considered as equivalents of the present disclosure.

[0106] Various changes, substitutions and alterations can be made to the technology described herein without departing from the teachings of the technology defined by the appended claims. In addition, the scope of the claims of the present disclosure is not limited to the specific aspects of the process, machine, manufacture, composition of matter, means, methods and acts described above. Processes, machines, manufacture, compositions of matter, means, methods or acts currently existing or later developed that perform substantially the same function or achieve substantially the same result as the corresponding aspects described herein can be utilized. Accordingly, the appended claims include within their scope such processes, machines, manufacture, compositions of matter, means, methods or acts.

[0107] The above description of the disclosed aspects is provided to enable any person skilled in the art to make or use the disclosure. Various modifications to these aspects will be readily apparent to those skilled in the art, and the generic principles defined herein can be applied to other aspects without departing from the scope of the disclosure. Thus, the present disclosure is not intended to be limited to the aspects shown herein but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

[0108] The above description has been presented to enable any person skilled in the art to make or use the disclosure. Furthermore, the purpose of the above description is not intended to limit the embodiments of the present disclosure to the form disclosed herein. Although various example aspects and embodiments have been discussed above, those of ordinary skill in the art will appreciate a variety of modifications, alternatives, permutations, additions, and sub-combinations of the described aspects and embodiments.

Claims

1. A method for generating a hardware device driving strategy based on multimedia content, the method comprising: The method comprises: analyzing the multimedia content to obtain feature information of the multimedia content; generating driving strategy information associated with the multimedia content based on the feature information; wherein the driving strategy information is used to indicate one or more hardware devices associated with the multimedia content and a predetermined processing performed by the one or more hardware devices, wherein, in the case that the multimedia content is generated based on live streaming data, before analyzing the multimedia content, for the same picture frame, determining a time difference between the multimedia content and the live streaming data; and based on the time difference, adding event information for indicating a predetermined event in the live streaming data, wherein the determination of the time difference between the multimedia content and the live streaming data comprises: generating synchronization data consistent with the live streaming data; identifying first event time information of a first event in the synchronization data, determining first time difference information based on the first event time information and second event time information of the first event marked in the live streaming data; and / or, determining second time difference information based on first picture time information corresponding to a first picture of the synchronization data and second picture time information corresponding to the first picture of the live streaming data; and / or, determining third time difference information based on marking time information of the first event marked in the live streaming data and display time information of the first event; determining the time difference based on one or more of the first time difference information, the second time difference information and the third time difference information.

2. The method of claim 1, wherein the multimedia content-based hardware device driving policy generation method is characterized by, The analysis of the multimedia content to obtain the feature information of the multimedia content comprises: identifying one or more target objects in the multimedia content and obtaining content feature information of the one or more target objects; and / or, identifying one or more specific picture frames in the multimedia content and obtaining picture feature information of the one or more specific picture frames, wherein the picture feature information comprises picture feature information determined based on a picture change amplitude of multiple intra-coded picture frames in the one or more specific picture frames; and / or, identifying an audio signal in the multimedia content and obtaining audio feature information of the audio signal, wherein the audio feature information comprises audio feature information determined based on a change amplitude of a level value corresponding to the audio signal in the multimedia content; using one or more of the content feature information, the picture feature information and the audio feature information as the feature information.

3. The method of claim 2, wherein the multimedia content-based hardware device driving policy generation method is characterized by, The generation of the driving strategy information associated with the multimedia content based on the feature information comprises: generating the driving strategy information associated with the multimedia content based on one or more of the content feature information, the picture feature information and the audio feature information; wherein the driving strategy information at least indicates weight information associated with the one or more target objects, the one or more hardware devices corresponding to the one or more target objects and the predetermined processing of the one or more hardware devices.

4. The multimedia content-based hardware device driving policy generation method of any one of claims 1 to 3, wherein, The method further comprises: generating a script file based on the driving strategy information, the script file being used to instruct the one or more hardware devices to perform predetermined processing at a predetermined time point; generating a driving signal for driving the one or more hardware devices to perform predetermined processing based on the script file and a play progress of the multimedia content.

5. The multimedia content-based hardware device driving policy generation method of any one of claims 1 to 3, wherein, The method further comprises: generating a driving signal for driving the one or more hardware devices to perform predetermined processing based on the real-time content of the multimedia content and the driving strategy information.

6. An electronic device, comprising: comprising: a memory for storing computer readable instructions; and a processor for running the computer readable instructions, so that the electronic device performs the hardware device driving strategy generation method based on multimedia content as claimed in any one of claims 1 to 5.

7. A non-transitory computer-readable storage medium storing computer-readable instructions, the computer-readable instructions comprising: The computer readable instructions, when executed by the processor, cause the processor to perform the hardware device driving strategy generation method based on multimedia content as claimed in any one of claims 1 to 5.

8. A computer program product comprising a computer program, characterized in that, The computer program, when executed by the processor, implements the hardware device driving strategy generation method based on multimedia content as claimed in any one of claims 1 to 5.

Images