Control method and device, electronic equipment and storage medium

Abstract

The invention provides a control method and device, electronic equipment and a storage medium. The method comprises the steps that first information is obtained; the first information comprises an environment expression of a space where the display equipment is located; second information is determined based on the first information, so that the display device outputs at least one frame of target image pointed by the second information; the second information comprises image contents related to the environment expression in multiple frames of images continuously presented in a target time period.

Description

Technical Field

[0001] The present disclosure relates to the field of information processing technology, and in particular to a control method, device, electronic device and storage medium. Background Art

[0002] During multimedia information output, such as playing a video, if the playback progress needs to be adjusted, the user is often required to adjust it manually or verbally. However, for the user, not only does it require stopping the current activity, but it is also difficult to achieve precise control. Summary of the invention

[0003] The present disclosure provides a control method, device, electronic device and storage medium to at least solve the above technical problems existing in the prior art.

[0004] According to a first aspect of the present disclosure, a control method is provided, the method comprising:

[0005] Obtaining first information; the first information includes an environmental expression of a space where a display device is located;

[0006] The second information is determined based on the first information, so that the display device outputs at least one frame of target image pointed to by the second information; the second information includes image content related to the environmental expression in multiple frames of images continuously presented within a target time period.

[0007] In one possible implementation manner, obtaining the first information includes:

[0008] Collect relevant information in the space where the display device is located to obtain first sub-information of the first target object and / or second sub-information of the second target object; the first information includes the first sub-information and / or the second sub-information;

[0009] The first sub-information represents an operational feature of the first target object, and the second sub-information represents a spatial feature of the second target object.

[0010] In one possible implementation manner, obtaining first sub-information of a first target object includes:

[0011] Process the collected information;

[0012] identifying the first target object;

[0013] identifying at least one action of the first target object and obtaining an operation feature characterizing the action;

[0014] The method further comprises:

[0015] At least one object associated with the action is identified to obtain other target objects; the first information also includes features of the other target objects.

[0016] In one embodiment, the method further comprises:

[0017] Dividing the multiple frames of images presented continuously within the target time period into multiple segments, and obtaining key information of each segment;

[0018] Matching the characteristic information of the target object with the key information of each segment;

[0019] Determine the segment corresponding to the key information that has been successfully matched as the target segment;

[0020] According to the segment currently output by the display device, determining whether the target segment is synchronized with the segment currently output, and obtaining a synchronization result;

[0021] According to the synchronization result, at least one frame of target image pointed to by the second information is determined from multiple frames of images continuously presented within the target time period, so that the display device outputs the at least one frame of target image pointed to by the second information.

[0022] In one possible implementation manner, when there are multiple segments corresponding to the key information that is successfully matched, determining the target segment includes:

[0023] Acquire specified first information within a specified time period, where the specified time period is a time period before the time period corresponding to the currently acquired first information;

[0024] Determining corresponding designated second information based on the designated first information, wherein the designated second information includes image content related to designated feature information of the target object;

[0025] The target segment is determined by re-matching the current feature information of the target object and the specified feature information with the key information of each segment.

[0026] In one possible implementation manner, determining, according to the synchronization result, at least one frame of target image pointed to by the second information from a plurality of frames of images continuously presented within the target time period, includes:

[0027] If the synchronization result indicates that the time of the target segment is earlier than the time of the currently output segment, determine that the at least one frame of the target image pointed to by the second information is at least one frame of the continuous multiple frames of images from the target segment to the currently output segment, so that the display device outputs the target image according to the first strategy;

[0028] If the synchronization result indicates that the time of the target segment is synchronized with the time of the currently output segment, it is determined that the at least one frame of the target image pointed to by the second information is at least one frame of the image in the currently output segment, and the display device continues to output the currently output segment;

[0029] If the synchronization result represents that the time of the target segment is later than the time of the currently output segment, determine that the at least one frame of target image pointed to by the second information is at least one frame of the continuous multiple frames of images from the currently output segment to the target segment, so that the display device outputs the target image according to the second strategy.

[0030] In one possible implementation manner, when the display device outputs at least one frame of target image pointed to by the second information, the method further includes: acquiring eye movement information of the user;

[0031] The step of causing the display device to output the target image according to the first strategy includes:

[0032] If the eye movement information indicates that the user is not gazing at the display device, causing the display device to pause outputting the currently output segment, or outputting first prompt information for prompting the user of lag;

[0033] If the eye movement information indicates that the user is gazing at the display device, causing the display device to output at least one frame of target image pointed to by the second information;

[0034] The causing the display device to output the image according to the second strategy includes:

[0035] If the eye movement information indicates that the user is not gazing at the display device, causing the display device to continue outputting the currently output segment, or to accelerate output of at least one frame of target image pointed to by the second information;

[0036] If the eye movement information indicates that the user is gazing at the display device, the display device is fast-forwarded to a specified frame image in the target segment.

[0037] According to a second aspect of the present disclosure, a control device is provided, the device comprising:

[0038] An acquisition module, configured to obtain first information; the first information includes an environmental expression of a space where a display device is located;

[0039] A determination module is used to determine second information based on the first information, so that the display device outputs at least one frame of target image pointed to by the second information; the second information includes image content related to the environmental expression in multiple frames of images continuously presented within a target time period.

[0040] According to a third aspect of the present disclosure, there is provided an electronic device, including:

[0041] display device, at least one processor, and a memory communicatively connected to the at least one processor; wherein,

[0042] The memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the following steps: obtaining first information; the first information includes an environmental expression of a space where the display device is located; determining second information based on the first information; the second information includes image content related to the environmental expression in a plurality of frames of images continuously presented within a target time period;

[0043] The display device is used to output at least one frame of target image pointed to by the second information.

[0044] According to a fourth aspect of the present disclosure, there is provided a non-transitory computer-readable storage medium storing computer instructions, wherein the computer instructions are used to cause the computer to execute the following steps:

[0045] Obtaining first information; the first information includes an environmental expression of a space where a display device is located;

[0046] The second information is determined based on the first information, so that the display device outputs at least one frame of target image pointed to by the second information; the second information includes image content related to the environmental expression in multiple frames of images continuously presented within a target time period.

[0047] The present invention discloses a control method, device, electronic device and storage medium, the method comprising: obtaining first information; the first information comprising an environmental expression of a space where a display device is located; determining second information based on the first information, so that the display device outputs at least one frame of target image pointed to by the second information; the second information comprising image content related to the environmental expression in multiple frames of images presented continuously within a target time period.

[0048] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] The above and other objects, features and advantages of the exemplary embodiments of the present disclosure will become readily understood by reading the detailed description below with reference to the accompanying drawings. In the accompanying drawings, several embodiments of the present disclosure are shown in an exemplary and non-limiting manner, in which:

[0050] In the drawings, the same or corresponding reference numerals represent the same or corresponding parts.

[0051] Figure 1 A schematic diagram of the implementation flow of the control method of the embodiment of the present disclosure is shown;

[0052] Figure 2 A schematic diagram of the implementation process of obtaining the first sub-information of the first target object according to an embodiment of the present disclosure is shown;

[0053] Figure 3 A schematic diagram of the implementation process of obtaining synchronization results according to an embodiment of the present disclosure is shown;

[0054] Figure 4 A schematic diagram of the implementation process of determining a target segment according to an embodiment of the present disclosure is shown;

[0055] Figure 5 A schematic diagram of an implementation flow of determining at least one frame of target image pointed to by the second information according to an embodiment of the present disclosure is shown;

[0056] Figure 6 A schematic diagram of the implementation process of the display device outputting at least one frame of target image pointed to by the second information according to the embodiment of the present disclosure is shown;

[0057] Figure 7 A schematic diagram of the structure of a control device according to an embodiment of the present disclosure is shown;

[0058] Figure 8 A schematic diagram of the structure of an electronic device according to an embodiment of the present disclosure is shown. DETAILED DESCRIPTION

[0059] In order to make the purpose, features, and advantages of the present disclosure more obvious and easy to understand, the technical solutions in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the drawings in the embodiments of the present disclosure. Obviously, the described embodiments are only part of the embodiments of the present disclosure, not all of the embodiments. Based on the embodiments in the present disclosure, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present disclosure.

[0060] The present disclosure provides a control method, such as Figure 1 As shown, the method includes:

[0061] Step 101: Obtain first information; the first information includes an environmental expression of a space where a display device is located.

[0062] In this example, the first information includes an environmental expression of the space where the display device is located. The environmental expression is a description or representation of the environment of the space where the display device is located, and may include a variety of information to reflect the environmental characteristics of the space, such as the size, shape, and layout of the space where the display device is located, as well as information such as people, objects, and sounds in the space.

[0063] The first information can be obtained by a collection device, which can be a camera, a sensor or other device that can capture images, sounds or other relevant data of the surrounding environment of the display device.

[0064] The display device may include a local screen and at least one extended screen. The display device and the acquisition device may be integrated into one electronic device, such as a mobile phone, a tablet, etc.; the acquisition device may also be set on different electronic devices from the display device.

[0065] Step 102: Determine second information based on the first information so that the display device outputs at least one frame of target image pointed to by the second information; the second information includes image content related to environmental expression in multiple frames of images continuously presented within a target time period.

[0066] In an example, the multiple frames of images presented continuously within the target time period may refer to a video that needs to be displayed to the user via a display device.

[0067] In this example, based on the collected first information, the image content related to the environmental expression in the multiple frames of images presented continuously within the target time period is determined, that is, the second information. After the second information is determined, the image content related to the environment is determined in the multiple frames of images presented continuously within the target time period to determine at least one frame of the target image, so that the display device outputs the at least one frame of the target image pointed to by the second information.

[0068] The above disclosed control method is described below through specific embodiments.

[0069] Example 1: A user learns to cook in the kitchen by playing a teaching video (i.e., multiple frames of images continuously presented within a target time period) on a display device (such as a smart TV or smart kitchen tablet). Existing solutions usually require the user to stop the operation in hand and adjust the output of the display device or control the playback progress by voice, but it is difficult to accurately control the output progress and may even lead to operational errors or forgetting key steps.

[0070] The solution disclosed in the present invention is to capture image information of the kitchen through the camera on the display device, including the location, status and other information of the user, ingredients, cooking utensils, etc. At the same time, the sound information during the cooking process (such as the sound of cutting vegetables, the sound of frying pans, the user's voice, etc.) is captured through the sound sensor. The image information and sound information here are the environmental expression of the kitchen space and constitute the first information.

[0071] The first information is analyzed to determine the second information. For example, if the second information represents the action of the user cutting vegetables, then the vegetable cutting segment in the teaching video can be located based on the second information. According to the strategy, at least one frame of the image included in the vegetable cutting segment can be output as the target image, and at least one frame of the image included in the previous segment of the vegetable cutting segment in the teaching video can also be output as the target image. Different target images can be determined according to different strategies, which will be specifically explained in subsequent examples.

[0072] Embodiment 2: When visiting a house, a user wants to play a video explaining the house (i.e., multiple frames of images presented continuously within a target time period) through a display device (such as smart glasses or a handheld device). Similarly, the existing solution also requires the user to manually adjust or voice control the output progress of the display device.

[0073] The solution disclosed in the present invention uses the camera and sensor of the display device to capture image information such as the internal layout, furniture placement, shape and size of the house, as well as various sound information in the house including human voices. The image information and sound information here are the environmental expressions of the house space and constitute the first information.

[0074] The first information is analyzed to determine the second information. For example, if the second information indicates that the current house is a bedroom, then the bedroom segment in the explanation video can be located based on the second information. According to the strategy, at least one frame of the image included in the bedroom segment can be output as the target image, and at least one frame of the image included in the previous segment of the bedroom segment in the teaching video can also be output as the target image. Different target images can be determined according to different strategies, which will be specifically described in subsequent examples.

[0075] The present disclosure provides a control method, which obtains first information; the first information includes the environmental expression of the space where the display device is located; the second information is determined based on the first information, so that the display device outputs at least one frame of target image pointed to by the second information; the second information includes image content related to the environmental expression in multiple frames of images continuously presented within a target time period. The solution of the present disclosure collects the environmental expression of the space where the display device is located, quickly and accurately matches the target image related to it and outputs it, so that the user does not need to frequently perform manual or language control, reduces the time of manual search and positioning, and thus improves the user experience.

[0076] In one example, the implementation process of obtaining the first information includes: collecting relevant information in the space where the display device is located, obtaining first sub-information of the first target object and / or second sub-information of the second target object; the first information includes first sub-information and / or second sub-information; the first sub-information represents the operational characteristics of the first target object, and the second sub-information represents the spatial characteristics of the second target object.

[0077] In this example, the first target object is a person or object that performs related operations in the space where the display device is located, such as a person who watches a teaching video to learn how to cook or a robot that imitates the video to perform operations. The first sub-information represents the operation characteristics of the first target object, which can be the movement trajectory of the object, the action of the person, and the object being operated. For example, if the first target object is a person, then the first sub-information can be the person's walking path, gestures, and objects.

[0078] The second target object is an object or area in the space where the display device is located, such as cultural relics displayed in a museum or a model room available for viewing in a sales center. The second sub-information represents the spatial characteristics of the second target object, which may be the location, size and shape of the object, or the spatial layout of the area. For example, if the second target object is a room, the second sub-information may be the size, layout, and placement of furniture in the room.

[0079] The space where the display device is located includes at least one target object, and the target object can be the first target object and / or the second target object. Therefore, the first information is composed of the first sub-information and / or the second sub-information. In general, the first information may only contain one of the first sub-information or the second sub-information. For example, in the scene of watching a teaching video to learn cooking, the space where the display device is located only includes the first target object, so the first information only includes the first sub-information; in the scene of visiting a museum, the space where the display device is located only includes the second target object, so the first information only includes the second sub-information. However, in special cases, such as imitating a video to complete a specified action in a specified environment to check in, the space where the display device is located can include both the first target object and the second target object. Therefore, the first information can simultaneously include the first sub-information and the second sub-information, thereby providing a more comprehensive environmental expression.

[0080] In one example, the process of obtaining the first sub-information of the first target object is as follows: Figure 2 As shown, including:

[0081] Step 201: Process the collected information.

[0082] In this example, the first information collected by the acquisition device is preliminarily processed. Depending on the type of the first information, the preliminarily processed method may include operations such as denoising, enhancement, and format conversion. For example, for image information, image enhancement is required to improve clarity; for sound information, denoising is required to improve the signal-to-noise ratio so that subsequent steps can more effectively perform target recognition and feature extraction.

[0083] Step 202: Identify a first target object.

[0084] In this example, according to different types of the first information, the first target object can be identified by using computer vision, machine learning, or sensor detection. For example, when the first information is image information, an image recognition algorithm can be used to identify the first target object in the first information; laser ranging or infrared light detection can also be used to identify the first target object in the first information.

[0085] Step 203: Identify at least one action of the first target object and obtain an operation feature representing the action.

[0086] In this example, after the first target object is identified, at least one action is further identified and operation features characterizing these actions are extracted. For example, based on the captured image or infrared imaging image, an action recognition algorithm is used to identify specific actions and operation features of a person.

[0087] The operational features that characterize an action include the type of action. For example, in a scenario of learning to cook, the recognized action types may include cutting, washing, boiling, frying, etc.; in a scenario of following fitness training, the recognized action types may include running, jumping, lifting, etc.

[0088] In one example, the method further includes:

[0089] Step 204: Identify at least one object associated with the action to obtain other target objects; the first information also includes features of the other target objects.

[0090] In this example, after identifying the operational features that characterize the actions of the first target object, further identify the objects associated with each action, use these objects as other target objects, and further identify the features of the other target objects. Specifically, an object recognition algorithm can be used to locate and identify at least one object associated with the action of the first target object.

[0091] For example, in a cooking learning scenario, objects associated with the action of "cutting" may be kitchen knives, cutting boards, tomatoes, etc., and objects associated with the action of "cooking" may be pots, water, corn, etc. In a fitness training scenario, objects associated with the action of "jumping" may be skipping ropes, and objects associated with the action of "lifting" may be dumbbells.

[0092] In one example, the process of obtaining the second sub-information of the second target object includes: processing the collected information; identifying the second target object; identifying the spatial features of the second target object to obtain the second sub-information.

[0093] The implementation process of the steps of processing the collected first information and identifying the second target object is the same as steps 201-202 of the previous example, and will not be repeated here.

[0094] It should be noted that the second target object is different from the first target object. The second target object is usually a static object or area in a non-operational scenario. Therefore, in the implementation process of obtaining the second sub-information, the spatial features of the second target object are identified. Specifically, the position, size and shape of the object or the spatial layout of the area can be identified by image recognition algorithms, laser ranging, etc.

[0095] For example, in the scenario of visiting a model house, an image recognition algorithm is used to identify the current space and items such as beds and bedside tables included in the current space, and determine them as the second target object; further, the spatial characteristics of the second target object are identified, including the position, size and shape of items such as beds and bedside tables, and the spatial layout of the current space, so that the current space can be further determined to be a bedroom.

[0096] In one example, if Figure 3 As shown, the method also includes:

[0097] Step 301: Divide a plurality of frames of images presented continuously within a target time period into a plurality of segments, and obtain key information of each segment.

[0098] In this example, in order to more efficiently process and analyze a large amount of image data in multiple frames of images presented continuously within a target time period, it is necessary to divide it into smaller, more manageable segments and extract key information from each segment.

[0099] Specifically, the segments can be divided according to preset time intervals, changes in image content, or specific event trigger points. The key information can be the text content of the image frame in each segment, such as summary, subtitle, title, etc., or the iconic features of the image frame in each segment, such as sequence number, icon, etc.

[0100] Step 302: Match the feature information of the target object with the key information of each segment.

[0101] In this example, the feature information is the first sub-information of each first target object or the second sub-information of each second target object, and the feature information is not equivalent to the first information. In order to determine which segments contain key information related to the target object, it is necessary to match the feature information of the target object with the key information of each segment. Specifically, a matching algorithm can be used to achieve matching of the feature information and the key information, such as text matching, feature matching, or a matching algorithm based on deep learning.

[0102] It should be noted that when the key information is text content in the image frame, the text content may be first segmented, and then the segmented text content may be matched with the feature information of the target object.

[0103] Step 303: Determine the segment corresponding to the key information that has been successfully matched as the target segment.

[0104] In this example, based on step 302, the segment corresponding to the key information that has been successfully matched is determined as the target segment, so as to filter out the segment that truly matches the characteristic information of the target object from all the segments.

[0105] Step 304: According to the segment currently output by the display device, determine whether the target segment is synchronized with the segment currently output, and obtain a synchronization result.

[0106] In this example, while the acquisition device is acquiring and processing the first information, the display device is continuously outputting multiple frames of images presented within the target time period. By comparing the timestamp or frame number of the currently output segment with the timestamp or frame number of the target segment to determine whether the two are synchronized, a judgment result of whether the target segment is synchronized with the currently output segment is obtained.

[0107] Step 305: According to the synchronization result, determine at least one frame of target image pointed to by the second information from multiple frames of images continuously presented within the target time period, so that the display device outputs the at least one frame of target image pointed to by the second information.

[0108] In this example, for different synchronization results, the at least one frame of target image pointed to by the second information is determined from the multiple frames of images continuously presented in the target time period, and the specific process of determining the target image will be described in detail in subsequent embodiments. After determining the target image, the display device outputs the at least one frame of target image pointed to by the second information.

[0109] In one example, when the target object is the first target object, the implementation process of matching the feature information of the target object with the key information of each segment includes:

[0110] The operation features representing the action of the first target object in the first sub-information and the activity features obtained by combining the features of other target objects associated with the action are matched with the key information of each segment together with the operation features and the features of other target objects.

[0111] For example, in a smart fitness scenario, a user is using a fitness guidance display device that combines a camera, a sensor, and a smart display. The display device can automatically match and play corresponding fitness tutorial videos based on the user's fitness movements and the fitness equipment used.

[0112] The user is doing fitness training at home, and the fitness exercise is dumbbell squat, where squat is the operation feature A1, and the object B1 associated with the operation feature A1 is a dumbbell. The operation feature A1 and the object B1 are combined to form the activity feature C1 (dumbbell squat), which means that the user is squatting with dumbbells in hand.

[0113] If the key information in the fitness tutorial video is matched only by the operation feature A1 and the object B1, the matching may be inaccurate due to the lack of activity features. For example, a clip in the tutorial video where the coach is squatting with bare hands but has dumbbells placed next to him may be mistakenly matched.

[0114] Therefore, in order to accurately match the clips in the fitness tutorial video, it is necessary to match the feature information A1, B1 and the activity feature C1 with the key information of the clips.

[0115] If multiple segments have been divided from multiple frames of images presented continuously within the target time period, and the key information of each segment has been obtained, then these key information are matched with the characteristic information of the target object. As a result, it is found that the key information of multiple segments successfully matches the characteristic information of the target object. In order to further determine the target segment that best meets the target requirements from these matched segments, the following process needs to be performed:

[0116] In one example, when there are multiple segments corresponding to the key information that is successfully matched, the implementation process of determining the target segment is as follows: Figure 4 As shown, including:

[0117] Step 401: obtaining designated first information within a designated time period, where the designated time period is a time period previous to the time period corresponding to the currently obtained first information.

[0118] In this example, in order to more accurately determine the target segment, it is necessary to refer to the first information in the previous time period, that is, to specify the first information. The specified time period refers to the previous time period of the time period corresponding to the currently acquired first information.

[0119] Step 402: Determine corresponding designated second information based on designated first information, wherein the designated second information includes image content related to designated feature information of the target object.

[0120] In this example, the designated second information related to the designated first information is determined by analyzing the designated first information. The designated second information includes image contents related to the designated feature information of the target object, and the image contents provide relevant features of the target object within a designated time period.

[0121] Step 403: Re-match the current feature information and designated feature information of the target object with the key information of each segment to determine the target segment.

[0122] In this example, the current feature information and the specified feature information of the target object are combined and re-matched with the key information of each successfully matched segment to further narrow the scope and determine the target segment that best meets the target requirements.

[0123] Specifically, the current feature information is first used to perform a preliminary match with the key information of the segment, and then the specified feature information is introduced for a deeper match. For example, it is determined whether the segment obtained by matching the specified feature information is adjacent to the segment obtained by matching the current feature information. If the two segments are adjacent segments, the segment matched by the current feature information is determined to be the target segment.

[0124] In one example, according to the synchronization result, the implementation process of determining at least one frame of target image pointed to by the second information from multiple frames of images continuously presented within the target time period is as follows: Figure 5 As shown, including:

[0125] Step 501: If the synchronization result indicates that the time of the target segment is earlier than the time of the currently output segment, determine that at least one frame of the target image pointed to by the second information is at least one frame of the continuous multiple frames from the target segment to the currently output segment, so that the display device outputs the target image according to the first strategy.

[0126] In this example, when the synchronization result shows that the time of the target segment is earlier than the segment currently being output by the display device, for example, the time of the target segment is 2min9s, while the currently output segment is 2min15s, it means that the environmental expression in the space where the current display device is located lags behind the image content corresponding to the currently output segment.

[0127] At this time, it is necessary to select at least one frame from the continuous multiple frame images from the target segment to the current output segment as the target image. The selected target image will be used to instruct the display device to output according to the first strategy. The specific output process will be described in detail in subsequent embodiments.

[0128] Step 502: If the synchronization result indicates that the time of the target segment is synchronized with the time of the currently output segment, it is determined that at least one frame of the target image pointed to by the second information is at least one frame of the image in the currently output segment, and the display device continues to output the currently output segment.

[0129] In this example, when the synchronization result shows that the time of the target segment is completely synchronized with the time of the current output segment, it means that the environmental expression in the space where the current display device is located is consistent with or the same as the image content of the current output segment. At this time, since the target image is already included in the current output segment, the display device will continue to output the current segment in the current manner without any changes.

[0130] Step 503: If the synchronization result indicates that the time of the target segment is later than the time of the currently output segment, determine that the at least one frame of target image pointed to by the second information is at least one frame of the continuous multiple frames of images from the currently output segment to the target segment, so that the display device outputs the target image according to the second strategy.

[0131] In this example, when the synchronization result shows that the time of the target segment is later than the segment currently being output by the display device, for example, the time of the target segment is 2 minutes and 20 seconds, while the currently output segment is 2 minutes and 15 seconds, it means that the environmental expression in the space where the current display device is located is ahead of the image content corresponding to the currently output segment.

[0132] At this time, it is necessary to select at least one frame from the continuous multiple frame images from the current output segment to the target segment as the target image. The selected target image will be used to instruct the display device to output according to the second strategy. The specific output process will be described in detail in subsequent embodiments.

[0133] In one example, when the display device outputs at least one frame of the target image pointed to by the second information, Figure 6 As shown, the method also includes:

[0134] Step 601: Obtain the user's eye movement information.

[0135] In this example, the user's eye movement information is obtained through eye tracking technology to identify whether the user is watching the display device. The judgment result includes that the eye movement information indicates that the user is not staring at the display device or the user is staring at the display device.

[0136] It should be noted that the user watching the display device can also be the target object. For example, in a certain operation scenario, the user is watching a video while operating. When the display device obtains the user's operation characteristics, it also obtains the user's eye movement information.

[0137] Further, causing the display device to output the target image according to the first strategy includes:

[0138] Step 602: If the eye movement information indicates that the user is not gazing at the display device, the display device is caused to pause outputting the currently output segment, or output first prompt information for prompting the user of a delay.

[0139] In this example, if the eye movement information indicates that the user is not staring at the display device, it means that the user is distracted or staring elsewhere. Therefore, the display device pauses outputting the currently output segment or outputs the first prompt information for prompting the user to lag behind, so as to wait or prompt the user to stare at the display device again. The first prompt information can prompt the user in any way, such as a prompt sound or a prompt light, etc., which is not limited by the present disclosure.

[0140] Step 603: If the eye movement information indicates that the user is gazing at the display device, the display device is caused to output at least one frame of the target image pointed to by the second information.

[0141] In this example, if the eye movement information represents that the user is gazing at the display device, it means that the user is watching or re-watching the display device after being prompted. Since the first strategy corresponds to the output strategy of the display device when the time of the target segment is earlier than the time of the currently output segment, the at least one frame of the target image pointed to by the output second information is at least one frame of the continuous multiple frames from the target segment to the currently output segment. For example, the time of the target segment is 2min9s, and the time of the currently output segment is 2min15s. Therefore, the target image output by the display device is at least one frame of the segment from 2min9s to 2min15s.

[0142] Further, causing the display device to output an image according to the second strategy includes:

[0143] Step 604: if the eye movement information indicates that the user is not gazing at the display device, the display device is enabled to continue outputting the currently output segment, or to accelerate the output of at least one frame of the target image pointed to by the second information.

[0144] In this example, if the eye movement information indicates that the user is not staring at the display device, it means that the user is distracted or staring elsewhere. Since the second strategy corresponds to the output strategy of the display device when the time of the target segment is later than the time of the currently output segment, the display device can continue to output the currently output segment, or accelerate the output of at least one frame of the target image pointed to by the second information.

[0145] The target image for accelerated output is at least one frame of the continuous multiple frames from the current output segment to the target segment. For example, the time of the target segment is 2min20s, and the time of the current output segment is 2min15s. Therefore, the target image output by the display device is at least one frame of the segment from 2min15s to 2min20s.

[0146] Step 605: If the eye movement information indicates that the user is gazing at the display device, the display device is fast-forwarded to a specified frame image in the target segment.

[0147] In this example, if the eye movement information indicates that the user is staring at the display device, it means that the user is watching or re-watching the display device after being prompted. At this time, according to the second strategy, the display device needs to fast forward to a specified frame image in the target segment. The specified frame image is a frame image within the corresponding time period of the target segment, for example, the specified frame image can be a frame image in 1s corresponding to the target segment of 2min20s.

[0148] The present disclosure also provides a control device, such as Figure 7 As shown, the device comprises:

[0149] An acquisition module 701 is used to obtain first information; the first information includes an environmental expression of a space where a display device is located;

[0150] The determination module 702 is used to determine the second information based on the first information so that the display device outputs at least one frame of target image pointed to by the second information; the second information includes image content related to environmental expression in multiple frames of images continuously presented within a target time period.

[0151] In one example, the acquisition module 701 includes a collection unit 7011, which is used to collect relevant information within the space where the display device is located, and obtain first sub-information of the first target object and / or second sub-information of the second target object; the first information includes first sub-information and / or second sub-information; the first sub-information represents the operational characteristics of the first target object, and the second sub-information represents the spatial characteristics of the second target object.

[0152] In one example, the acquisition module 701 also includes a processing unit 7012, which is used to process the collected information; identify a first target object; identify at least one action of the first target object to obtain an operation feature that characterizes the action; identify at least one object associated with the action to obtain other target objects; the first information also includes features of other target objects.

[0153] In one example, the determination module 702 includes a first determination unit 7021, which is used to divide a plurality of frames of images presented continuously within a target time period into a plurality of segments, and obtain key information of each segment; match feature information of the target object with key information of each segment; determine the segment corresponding to the successfully matched key information as the target segment; determine whether the target segment is synchronized with the currently output segment based on the segment currently output by the display device, and obtain a synchronization result; and determine, based on the synchronization result, at least one frame of target image pointed to by the second information from the plurality of frames of images presented continuously within the target time period, so that the display device outputs at least one frame of target image pointed to by the second information.

[0154] In one example, the first determination unit 7021 is also used to obtain specified first information within a specified time period when there are multiple fragments corresponding to the key information that is successfully matched, and the specified time period is the previous time period of the time period corresponding to the currently acquired first information; determine the corresponding specified second information based on the specified first information, and the specified second information includes image content related to the specified feature information of the target object; and re-match the key information of each fragment by the current feature information and the specified feature information of the target object to determine the target fragment.

[0155] In one example, the determination module 702 further includes a second determination unit 7022, configured to:

[0156] If the synchronization result indicates that the time of the target segment is earlier than the time of the currently output segment, determine that the at least one frame of the target image pointed to by the second information is at least one frame of the continuous multiple frames of images from the target segment to the currently output segment, so that the display device outputs the target image according to the first strategy;

[0157] If the synchronization result indicates that the time of the target segment is synchronized with the time of the currently output segment, it is determined that the at least one frame of the target image pointed to by the second information is at least one frame of the image in the currently output segment, and the display device continues to output the currently output segment;

[0158] If the synchronization result indicates that the time of the target segment is later than the time of the currently output segment, determine that the at least one frame of target image pointed to by the second information is at least one frame of the continuous multiple frames of images from the currently output segment to the target segment, so that the display device outputs the target image according to the second strategy.

[0159] In one example, the determination module 702 also includes an output unit 7023, which is used to obtain the user's eye movement information; enable the display device to output the target image according to the first strategy, including: if the eye movement information indicates that the user is not gazing at the display device, the display device pauses outputting the currently output segment, or outputs the first prompt information for prompting the user of the delay; if the eye movement information indicates that the user is gazing at the display device, the display device outputs at least one frame of the target image pointed to by the second information; enable the display device to output the image according to the second strategy, including: if the eye movement information indicates that the user is not gazing at the display device, the display device continues to output the currently output segment, or accelerates the output of at least one frame of the target image pointed to by the second information; if the eye movement information indicates that the user is gazing at the display device, the display device fast forwards to a specified frame image in the target segment.

[0160] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device and a readable storage medium.

[0161] Figure 8 A schematic block diagram of an example electronic device 800 that can be used to implement an embodiment of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or required herein.

[0162] like Figure 8 As shown, the electronic device 800 includes a computing unit 801, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 802 or a computer program loaded from a storage unit 808 into a random access memory (RAM) 803. In the RAM 803, various programs and data required for the operation of the device 800 can also be stored. The computing unit 801, the ROM 802, and the RAM 803 are connected to each other via a bus 804. An input / output (I / O) interface 805 is also connected to the bus 804.

[0163] A number of components in the device 800 are connected to the I / O interface 805, including: an input unit 806, such as a keyboard, a mouse, etc.; an output unit 807, such as various types of displays, speakers, etc.; a storage unit 808, such as a disk, an optical disk, etc.; and a communication unit 809, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 809 allows the device 800 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0164] The computing unit 801 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 801 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 801 performs the various methods and processes described above, such as control methods. For example, in some embodiments, the control method may be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as a storage unit 808. In some embodiments, part or all of the computer program may be loaded and / or installed on the device 800 via ROM 802 and / or communication unit 809. When the computer program is loaded into RAM 803 and executed by the computing unit 801, one or more steps of the control method described above may be performed. Alternatively, in other embodiments, the computing unit 801 may be configured to perform the control method in any other appropriate manner (e.g., by means of firmware).

[0165] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0166] The program code for implementing the method of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0167] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0168] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0169] The systems and techniques described herein may be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), and the Internet.

[0170] A computer system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The relationship of client and server is generated by computer programs running on respective computers and having a client-server relationship with each other. The server may be a cloud server, a server of a distributed system, or a server combined with a blockchain.

[0171] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this disclosure can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved, and this document does not limit this.

[0172] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of the features. In the description of the present disclosure, the meaning of "plurality" is two or more, unless otherwise clearly and specifically defined.

[0173] The above is only a specific embodiment of the present disclosure, but the protection scope of the present disclosure is not limited thereto. Any person skilled in the art who is familiar with the technical field can easily think of changes or substitutions within the technical scope disclosed in the present disclosure, which should be included in the protection scope of the present disclosure. Therefore, the protection scope of the present disclosure should be based on the protection scope of the claims.

Claims

1. A control method, the method comprising: Get first information; The first information includes an environmental expression of a space where the display device is located; The second information is determined based on the first information, so that the display device outputs at least one frame of target image pointed to by the second information; the second information includes image content related to the environmental expression in multiple frames of images presented continuously within a target time period.

2. The method according to claim 1, obtaining the first information comprises: Collecting relevant information in the space where the display device is located to obtain first sub-information of the first target object and / or second sub-information of the second target object; The first information includes first sub-information and / or second sub-information; The first sub-information represents an operational feature of the first target object, and the second sub-information represents a spatial feature of the second target object.

3. The method according to claim 2, obtaining the first sub-information of the first target object comprises: Process the collected information; identifying the first target object; identifying at least one action of the first target object and obtaining an operation feature characterizing the action; The method further comprises: identifying at least one object associated with the action to obtain other target objects; The first information also includes features of other target objects.

4. The method according to claim 2, further comprising: Dividing the multiple frames of images presented continuously within the target time period into multiple segments, and obtaining key information of each segment; Matching the characteristic information of the target object with the key information of each segment; Determine the segment corresponding to the key information that has been successfully matched as the target segment; According to the segment currently output by the display device, determining whether the target segment is synchronized with the segment currently output, and obtaining a synchronization result; According to the synchronization result, at least one frame of target image pointed to by the second information is determined from multiple frames of images continuously presented within the target time period, so that the display device outputs the at least one frame of target image pointed to by the second information.

5. The method according to claim 4, when the successfully matched key information corresponds to multiple segments, determining the target segment comprises: Acquire specified first information within a specified time period, where the specified time period is a time period before the time period corresponding to the currently acquired first information; Determining corresponding designated second information based on the designated first information, wherein the designated second information includes image content related to designated feature information of the target object; The target segment is determined by re-matching the current feature information of the target object and the specified feature information with the key information of each segment.

6. The method according to claim 4, wherein determining, based on the synchronization result, at least one frame of target image pointed to by the second information from a plurality of frames of images continuously presented within the target time period comprises: If the synchronization result indicates that the time of the target segment is earlier than the time of the currently output segment, determine that the at least one frame of the target image pointed to by the second information is at least one frame of the continuous multiple frames of images from the target segment to the currently output segment, so that the display device outputs the target image according to the first strategy; If the synchronization result indicates that the time of the target segment is synchronized with the time of the currently output segment, it is determined that the at least one frame of the target image pointed to by the second information is at least one frame of the image in the currently output segment, and the display device continues to output the currently output segment; If the synchronization result represents that the time of the target segment is later than the time of the currently output segment, determine that the at least one frame of target image pointed to by the second information is at least one frame of the continuous multiple frames of images from the currently output segment to the target segment, so that the display device outputs the target image according to the second strategy.

7. The method according to claim 6, when the display device outputs at least one frame of target image pointed to by the second information, the method further comprises: Get the user's eye movement information; The step of causing the display device to output the target image according to the first strategy includes: If the eye movement information indicates that the user is not gazing at the display device, causing the display device to pause outputting the currently output segment, or outputting first prompt information for prompting the user of lag; If the eye movement information indicates that the user is gazing at the display device, causing the display device to output at least one frame of target image pointed to by the second information; The causing the display device to output the image according to the second strategy includes: If the eye movement information indicates that the user is not gazing at the display device, causing the display device to continue outputting the currently output segment, or to accelerate output of at least one frame of target image pointed to by the second information; If the eye movement information indicates that the user is gazing at the display device, the display device is fast-forwarded to a specified frame image in the target segment.

8. A control device, comprising: An acquisition module, used for acquiring first information; The first information includes an environmental expression of a space where the display device is located; A determination module is used to determine second information based on the first information, so that the display device outputs at least one frame of target image pointed to by the second information; the second information includes image content related to the environmental expression in multiple frames of images continuously presented within a target time period.

9. An electronic device, comprising: display device, at least one processor, and a memory communicatively connected to the at least one processor; wherein, The memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the following steps: obtaining first information; the first information includes an environmental expression of a space where the display device is located; determining second information based on the first information; the second information includes image content related to the environmental expression in a plurality of frames of images continuously presented within a target time period; The display device is used to output at least one frame of target image pointed to by the second information.

10. A non-transitory computer-readable storage medium storing computer instructions, the computer instructions being configured to cause the computer to perform the following steps: Obtaining first information; the first information includes an environmental expression of a space where a display device is located; The second information is determined based on the first information, so that the display device outputs at least one frame of target image pointed to by the second information; the second information includes image content related to the environmental expression in multiple frames of images presented continuously within a target time period.

Images