Vibration control method and device, electronic equipment and storage medium

Abstract

Embodiments of the present application provide a vibration control method and device, electronic equipment and readable medium. The method comprises: acquiring a plurality of continuous frames of images; each frame of image in the plurality of frames of images contains a vibration element that can trigger vibration; performing partition processing on each frame of image in the plurality of frames of images according to the same partition rule to obtain a plurality of image regions contained in each frame of image; determining a target vibration region in the image according to a region static index corresponding to the image region in the same position in the plurality of frames of images; the region static index is used to represent the similarity or difference between the image regions in the same position in the plurality of frames of images; identifying the type and state change of the vibration element contained in the target vibration region in the video stream corresponding to the plurality of frames of images; and generating a vibration control instruction matched with the type of the vibration element in response to the state change reaching a first preset condition.

Description

Technical Field

[0001] This application relates to the field of vibration technology, and in particular to a vibration control method, device, electronic device, and storage medium. Background Technology

[0002] With the continuous development of smart terminal technology, gaming software has become an important component of terminal applications. For gaming applications, the addition of vibration feedback can increase user immersion and enhance the human-computer interaction experience; it can also be used to confirm user actions. However, the confirmation of vibration scenarios in related technologies is often based on pixel information in the image or instructions received by the image recognition application programming interface (API), which not only requires significant maintenance and management costs but also cannot flexibly apply to more scenarios. Summary of the Invention

[0003] In view of this, this application provides a vibration control method, apparatus, electronic device, and storage medium.

[0004] On one hand, embodiments of this application provide a vibration control method, including:

[0005] Acquire multiple consecutive frames of images; each frame of the multiple frames contains a target vibration element that may trigger vibration;

[0006] Each frame of the multi-frame image is partitioned according to the same partitioning rules to obtain multiple image regions contained in each frame image;

[0007] The target vibration region in the image is determined based on the region static index corresponding to the image regions at the same position in the multi-frame images; the region static index is used to characterize the degree of similarity or difference between the image regions at the same position in the multi-frame images.

[0008] Identify the type and state changes of the target vibration elements contained in the target vibration region in the video stream corresponding to the multi-frame images;

[0009] In response to the state change reaching a first preset condition, a vibration control command matching the type of the target vibration element is generated.

[0010] In the above scheme, determining the target vibration region in the image based on the regional static index corresponding to the image regions at the same position in the multi-frame images includes:

[0011] The image region whose static indicators satisfy the second preset condition is identified.

[0012] The target vibration region is determined based on the image region whose static indicators satisfy the second preset condition.

[0013] The condition that the regional static index satisfies the second preset condition includes: when the regional static index represents the similarity between image regions at the same location, the regional static index is greater than or equal to a first threshold; when the regional static index represents the difference between image regions at the same location, the regional static index is less than or equal to a second threshold.

[0014] In the above scheme, determining the target vibration region based on the image region whose regional static index meets the second preset condition includes: directly determining the image region whose regional static index meets the second preset condition as the target vibration region;

[0015] Alternatively, the image region whose static indicators meet the second preset condition can be adjusted to obtain the target vibration region whose pixel area is greater than or smaller than the image region whose static indicators meet the second preset condition.

[0016] In the above scheme, identifying the type of the target vibration element contained in the target vibration region in the video stream corresponding to the multi-frame images includes:

[0017] A vibration element template library and a neural network model are obtained; the vibration element template library includes various vibration elements and the corresponding types of each vibration element; the neural network model is obtained by learning and training with different vibration elements and the corresponding types of each vibration element.

[0018] The target vibration element in the target vibration region is detected based on the vibration element template library or the neural network model, and the type of the target vibration element is determined.

[0019] In the above scheme, identifying the state changes of the target vibration elements contained in the target vibration region within the video stream corresponding to the multi-frame images includes:

[0020] By comparing the pixels of the target vibration element in two consecutive frames of the multi-frame image, the state change of the vibration element is determined.

[0021] In the above scheme, the state change reaching the first preset condition includes: at least one attribute value of the basic attributes of the pixels contained in the target vibration element has a comparison value in two consecutive frames of images that exceeds a third threshold.

[0022] The basic attributes include color, brightness, contrast, saturation, and sharpening.

[0023] In the above scheme, before partitioning each frame of the multi-frame images according to the same partitioning rule, the method further includes:

[0024] Each frame of the multi-frame image is preprocessed, wherein the preprocessing includes at least one of the following: smoothing, thresholding, sampling, and grayscale processing.

[0025] On the other hand, embodiments of this application also provide a vibration control device, including:

[0026] An acquisition module is used to acquire multiple consecutive frames of images; each frame of the multiple frames of images contains a target vibration element that may trigger vibration.

[0027] The partitioning module is used to partition each frame of the multi-frame images according to the same partitioning rules to obtain multiple image regions contained in each frame image.

[0028] The determination module is used to determine the target vibration region in the image based on the region static index corresponding to the image regions at the same position in the multi-frame images; the region static index is used to characterize the degree of similarity or difference between the image regions at the same position in the multi-frame images.

[0029] The identification module is used to identify the type and state changes of the target vibration elements contained in the target vibration region in the video stream corresponding to the multi-frame images;

[0030] The generation module is used to generate a vibration control command that matches the type of the target vibration element in response to the state change reaching a first preset condition.

[0031] In another aspect, embodiments of this application also provide an electronic device, the electronic device comprising: a processor and a memory for storing a computer program capable of running on the processor, wherein the processor, when running the computer program, performs the steps of the above method.

[0032] In another aspect, embodiments of this application also provide a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the steps of the above-described method.

[0033] This application provides a vibration control method, apparatus, electronic device, and storage medium. The method includes: acquiring multiple consecutive frames of images; each frame of the multiple frames contains a target vibration element that may trigger vibration; partitioning each frame of the multiple frames according to the same partitioning rule to obtain multiple image regions contained in each frame; determining a target vibration region in the image based on a region static index corresponding to the image regions at the same position in the multiple frames; the region static index is used to characterize the similarity or difference between the image regions at the same position in the multiple frames; identifying the type and state changes of the target vibration element contained in the target vibration region in the video stream corresponding to the multiple frames; and generating a vibration control command matching the type of the target vibration element in response to the state change reaching a first preset condition. The vibration detection method provided in this application acquires multiple consecutive frames of images in real time and then analyzes the state changes of the vibration elements contained in the multiple frames to determine whether vibration is needed in the scene. This vibration control method is applicable to the determination of vibration scenarios in any scene and has good universality. Attached Figure Description

[0034] Figure 1 This is an optional flowchart illustrating the vibration control method provided in the embodiments of this application. Figure 1 ;

[0035] Figure 2 This is an example illustration of two frames of images partitioned according to the same partitioning rules provided in this application embodiment;

[0036] Figure 3 This is a flowchart illustrating a method for determining regional static indicators, as provided in an embodiment of this application.

[0037] Figure 4 This is an example illustration of three frames of images provided in this application being partitioned according to the same partitioning rules;

[0038] Figure 5 This is a schematic diagram of a process for obtaining a target vibration region provided in an embodiment of this application. Figure 1 ;

[0039] Figure 6 This is an exemplary illustration of enlarging or shrinking an image region provided in an embodiment of this application;

[0040] Figure 7 This is a schematic diagram of a process for determining a vibration element and its type, provided in an embodiment of this application.

[0041] Figure 8 An optional flowchart of the vibration control method provided in this application embodiment. Figure 2 ;

[0042] Figure 9 This is a schematic diagram of a process for obtaining a target vibration region provided in an embodiment of this application;

[0043] Figure 10 This is a schematic diagram of a template matching process for obtaining vibration elements and types, provided in an embodiment of this application.

[0044] Figure 11 This is a schematic diagram of a machine learning process for obtaining vibration elements and types, provided in an embodiment of this application.

[0045] Figure 12 This is a schematic diagram of a process for determining the state changes of a vibrating element and triggering the issuance of a vibration control command based on the basic attributes of a pixel, provided in an embodiment of this application.

[0046] Figure 13 This is a schematic diagram of a process for determining the state change of a vibrating element based on a color change and triggering the issuance of a vibration control command, provided in an embodiment of this application.

[0047] Figure 14 This is a schematic diagram of the structure of a vibration control device provided in an embodiment of this application. Detailed Implementation

[0048] With the continuous development of smart terminal technology, entertainment applications such as games and video playback have become an important part of terminal applications. In the use of these applications, haptic feedback (or vibration feedback) plays an indispensable role in improving the user experience, especially in gaming applications. For gaming applications, vibration feedback is an important function for enhancing human-computer interaction and confirming user actions. Related technologies mainly provide vibration control methods for game items or game scenes, including identifying pixel information in a fixed target area or identifying drawing instructions from the Graphical Application Programming Interface (API) to determine the scene requiring vibration. The method of identifying pixel information in a fixed target area to determine the vibration scene mainly relies on image recognition technology to identify the scene and determine whether the current scene meets the vibration triggering conditions and the vibration type. In the proposed solutions, the coordinates of a fixed area need to be preset, and the pixel features within the preset fixed area need to be extracted to achieve the purpose of identifying a specific scene. However, the selection of the preset fixed area limits the scope of application of this technology, and also means that when adapting to new scenes (game screen updates, resolution adjustments, or custom buttons) or applications, the solution needs to be readjusted, increasing maintenance and management costs. The method of determining vibration scenes by recognizing drawing instructions from a graphical application programming interface (API) primarily involves identifying the current scene based on these instructions to determine whether it meets the vibration triggering conditions and the type of vibration. However, this method has several significant drawbacks. First, it requires the pre-extraction of a large amount of information, such as the object model information of the object to be identified and the correspondence between the object model information and the scene. Acquiring and processing this information requires complex preparatory work, increasing the system's complexity and development costs. Second, different recognition algorithms and rules need to be customized for different APIs, making generalization and standardization difficult. Finally, since this method relies heavily on the parsing and analysis of graphical drawing instructions, it may impose a performance burden, especially when dealing with large-scale or real-time scenes. Therefore, while the method of determining vibration scenes by recognizing graphical API drawing instructions may be effective in some specific scenarios, many technical and implementation challenges need to be overcome to achieve success in a wider range of applications.

[0049] Based on this, in order to solve one or more of the above problems, this application provides a vibration control method. By analyzing images of multiple consecutive frames, the method determines the type and state changes of vibration elements contained in the current scene, so as to determine whether the current scene needs vibration. This method is highly versatile, scalable and portable. It no longer relies on fixed area pixel information and can automatically detect and extract vibration elements that need to be detected for motion changes from the screen according to the characteristics of the video / game itself.

[0050] Specifically, see Figure 1 This illustrates an optional flowchart of the vibration control method provided in an embodiment of this application. Figure 1 As shown, the vibration control method may include:

[0051] Step 101: Acquire multiple consecutive frames of images; each frame of the multiple frames contains a target vibration element that may trigger vibration.

[0052] It should be noted that this method is applicable to the detection and vibration control of vibration elements in various types of video or game streams. The video streams mentioned can include, but are not limited to, movies, TV series, live TV broadcasts, and other watchable video streams. The game stream can be the video screen displayed on the display component during the runtime of a game application, referred to simply as a game. Regardless of the type of video, the steps of the vibration control method provided in this application embodiment are similar. The implementation of this application embodiment will be described below using the control of vibration in a game video stream as an example. That is, unless otherwise specified, the video streams and icons described below refer to games and vibration elements within games. Specifically, the vibration element can be an element in the game whose state can change and may trigger vibration when it changes to a certain state. For example, in a game video, the vibration element could be a skill icon, item icon, or start icon of a game character, etc.

[0053] In practical applications, a video stream over a period of time consists of frames arranged chronologically. In step 101, the "continuous multiple frames" can refer to multiple temporally consecutive frames in a game video stream, where each frame contains a target vibration element that may trigger vibration. The size of the image can be the size of the display screen of the terminal running the game, or it can be set according to the specific size of the icon to be detected or other conditions. This application does not limit the image size.

[0054] Step 102: Divide each frame of the multi-frame image into multiple image regions according to the same partitioning rules to obtain multiple image regions contained in each frame.

[0055] It should be noted that the so-called identical partitioning rule refers to applying the same processing to each frame of a multi-frame image, resulting in each frame being divided into multiple image regions. Specifically, the number of image regions divided into which different frames of the same image are divided is the same, and the size of the corresponding image regions in different frames is the same. The size of the image region can be measured using its pixel area. A larger pixel area results in a larger image region, and a smaller pixel area results in a smaller image region. It should be understood that the size of the pixel area can be measured by the number of pixels; the more pixels an image region contains, the larger its pixel area; the fewer pixels an image region contains, the smaller its pixel area. In this application, there are no restrictions on the partitioning method or the number of partitions. That is, the partitioning rule can be any method that can partition an image, as long as the same partitioning rule is applied to each frame.

[0056] For example, such as Figure 2 As shown, assuming the multi-frame image includes frame A and frame B, the partitioning process according to the same partitioning rule means that frame A is divided into four equal image regions (i.e., multiple image regions corresponding to frame A), namely A1, A2, A3, and A4. Similarly, frame B is also divided into four equal image regions (i.e., multiple image regions corresponding to frame B), namely B1, B2, B3, and B4. Furthermore, A1 and B1 are image regions located at the same position in both frames A and B. Likewise, A2 and B2, A3 and B3, and A4 and B4 are also image regions located at the same position in different frames. Based on the previous description of pixel area, the pixel areas of image regions located at the same position in frames A and B processed according to the same partitioning rule are equal, for example, A1 and B1 have equal pixel areas, A2 and B2 have equal pixel areas, A3 and B3 have equal pixel areas, and A4 and B4 have equal pixel areas.

[0057] In some embodiments, before partitioning each frame of the multi-frame images according to the same partitioning rule, the method may further include:

[0058] Each frame of the multi-frame image is preprocessed, wherein the preprocessing includes at least one of the following: smoothing, thresholding, sampling, and grayscale processing.

[0059] It should be noted that, in order to reduce the complexity of subsequent calculations and / or highlight specific regions (which may contain the icons to be detected), after acquiring multiple consecutive frames of images, each frame is pre-processed. This pre-processing may include at least one of the following: smoothing, thresholding, downsampling, or greyscale processing. In other words, each frame can undergo at least one of these processes to simplify subsequent calculations. Smoothing and thresholding can reduce noise in the image to highlight specific regions and simplify subsequent image processing. Downsampling can reduce image resolution to reduce computational complexity. Greyscale processing, also known as greyscale conversion, converts images in red, green, and blue (RGB) or hue, saturation, and value (HSV) modes into greyscale images to reduce the number of color channels and thus reduce computational complexity.

[0060] Step 103: Determine the target vibration region in the image based on the regional static index corresponding to the image regions at the same position in the multi-frame images; the regional static index is used to characterize the similarity or difference between the image regions at the same position in the multi-frame images.

[0061] It should be noted that the aforementioned regional static index is used to reflect the degree of similarity or difference between image regions located at the same position in different frames of the multi-frame image dataset. For example, Figure 2 The A-frame and B-frame images shown represent the static indicators of this region. Figure 2 The degree of similarity and difference between image regions in the same position in the A-frame and B-frame images shown (e.g., A1 and B1, A2 and B2, A3 and B3, A4 and B4).

[0062] Regarding the determination of regional static indicators, in some embodiments, such as Figure 3 As shown, the method may further include:

[0063] Step 301: Determine the weighting factors corresponding to the image regions at the same position in the multi-frame images;

[0064] Step 302: Determine the similarity or difference between image regions at the same position in the multi-frame images;

[0065] Step 303: Obtain the regional static index based on the similarity and the weighting factor, or obtain the regional static index based on the difference and the weighting factor.

[0066] It should be noted that this weighting factor reflects the probability of a vibration element in a certain image region. A larger weighting factor indicates a higher probability that the corresponding image region contains a vibration element, and a smaller weighting factor indicates a lower probability. In one scenario, the probability of a vibration element in a particular image region is roughly the same across multiple image regions; in this case, the weighting factor can be the same for each image region. In another scenario, when the probability of a vibration element in a particular image region is high, the weighting factor for that image region can be set higher, while the weighting factors for the other image regions can be set relatively lower. For example, in a multiplayer online battle arena (MOBA) game, a vibration element could be a character's skill icon. Skill icons are typically located in the lower right corner of the image; therefore, the image region in the lower right corner will be assigned a higher weighting factor, while other image regions will be assigned lower weighting factors, thus reducing interference from other image regions on the detection results. Here, the weighting factor for image regions in the same position across different frames in the multi-frame image is the same; for example, the weighting factors for A1 and B1 mentioned above are the same.

[0067] After setting the weight factor for each image region, the similarity between image regions at the same position in the multi-frame images is determined. Here, the image regions at the same position, as described earlier, refer to image regions in the same position across different frames. For example, A1 and B1, or A2 and B2, or A3 and B3, or A4 and B4, all described earlier, belong to image regions at the same position. The similarity between these image regions can refer to similarities such as those between A1 and B1, A2 and B2, A3 and B3, and A4 and B4.

[0068] In practical applications, the similarity referred to can be the Structural Similarity Index (SSIM) between image regions, or the degree of auto-correlation between image regions. It should be noted that since the acquired consecutive multi-frame images can include two or more frames, the calculation of this similarity can be divided into two cases. In one case, the multi-frame images include, for example... Figure 2In the two image frames shown, the similarity between image regions at the same location can refer to the similarity between the two image regions. In specific calculations, the SSIM index can be directly calculated using the SSIM algorithm to reflect the similarity between image regions at the same location, or the autocorrelation coefficient between image regions can be calculated using image processing methods such as autocorrelation algorithms. In another scenario, where the multi-frame image contains more than two images, the similarity between image regions at the same location can refer to the structural similarity index or autocorrelation degree between image regions in consecutive frames after certain operations (such as addition or multiplication). That is, when the multi-frame image contains more than two images, the similarity between image regions at the same location is not simply the SSIM index or autocorrelation degree calculated directly between two image regions. It requires calculating the SSIM index or autocorrelation degree between all two image regions in consecutive frames, and then processing all the calculated SSIM indexes or autocorrelation degrees to obtain the value of the value after certain operations.

[0069] See Figure 4 As shown, taking a multi-frame image containing three frames, each frame divided into four image regions, as an example, this illustrates the calculation of similarity between image regions at the same location. Figure 4 As shown, in front Figure 2Based on the previous image, an additional image C is added, which is divided into four image regions, C1, C2, C3, and C4, according to the same partitioning rules. A1, B1, and C1 are image regions at the same location; A2, B2, and C2 are image regions at the same location; A3, B3, and C3 are image regions at the same location; and A4, B4, and C4 are image regions at the same location. The calculation of the similarity between A1, B1, and C1 is used as an example. First, the autocorrelation degree (1) between A1 and B1 is calculated; then, the autocorrelation degree (2) between B1 and C1 is calculated; subsequently, one method is to add the autocorrelation degree (1) and the autocorrelation degree (2) together and then take the average. This average value is the similarity between A1, B1, and C1. The calculation of the similarity between the remaining image regions at the same location is similar. It should be noted that in the above example of 3 frames, the calculation of the autocorrelation degree of image regions at the same position is as follows: calculate the autocorrelation degree between A1 and B1 (1); then calculate the autocorrelation degree between B1 and C1 (2), and then obtain the autocorrelation degree corresponding to the image regions at the same position. In another embodiment, for more than three frames of images, such as 4 frames of images, the image regions at the same position are A1, B1, C1, and D1, the calculation of the autocorrelation degree of image regions at the same position can also be as follows: calculate the autocorrelation degree between A1 and B1 (1); then calculate the autocorrelation degree between C1 and D1 (3), and then obtain the autocorrelation degree between the image regions at the same position based on the autocorrelation degree (1) and the autocorrelation degree (3).

[0070] After obtaining the similarity between image regions at the same location and the weight factors corresponding to image regions at the same location among multiple image regions, the static index of the corresponding image region at the same location is determined based on the corresponding weight factor and the corresponding similarity. Specifically, the corresponding similarity is multiplied by the corresponding weight factor, and the product is the static index of the image region at the same location. This process is repeated to obtain the static index of each image region at the same location. Here, the static index of each image region at the same location is the same; the weight factor of each image region at the same location is also the same.

[0071] For example, such as Figure 2The two images, frame A and frame B, are divided into four image regions according to an equal division rule. After obtaining the similarity between A1 and B1 and their corresponding weighting factors, the static index of region A1 or B1 can be obtained; after obtaining the similarity between A2 and B2 and their corresponding weighting factors, the static index of region A2 or B2 can be obtained; after obtaining the similarity between A3 and B3 and their corresponding weighting factors, the static index of region A3 or B3 can be obtained; and after obtaining the similarity between A4 and B4 and their corresponding weighting factors, the static index of region A4 or B4 can be obtained.

[0072] In some embodiments, the degree of difference may refer to the difference between image regions. For example... Figure 2 The image shows the difference between image regions A1 and B1 at the same location in the A-frame and B-frame images. Specifically, this difference can be calculated using image processing techniques such as frame differencing between image regions at the same location in multiple frames. Similarly, depending on the number of images contained in the multiple frames, there are two cases. The first case is where the multiple frames contain images such as... Figure 2 The two images shown are used to calculate the difference image between image regions at the same location. This difference image is then used to evaluate the degree of difference between these regions. Here, the difference image can refer to the image region obtained by calculating the difference between the pixel values ​​of corresponding pixels in the image regions at the same location in the two images. In the second case, multiple images contain, for example,... Figure 4 For images consisting of two or more frames, multiple difference images are calculated between image regions at the same location in the multiple frames. These difference images are then used to evaluate the degree of difference between image regions at the same location in two consecutive frames. Finally, these degree differences are processed to obtain the overall difference score between image regions at the same location in the multiple frames. The "two consecutive frames" can refer to adjacent images or non-adjacent images. The specific selection method can be determined based on the actual situation and will not be elaborated further here.

[0073] Next, after obtaining the difference between image regions at the same location and the weight factor corresponding to each of the multiple image regions, the corresponding static index of the image region is determined based on the weight factor and the difference of each image region. Specifically, the difference of each image region is multiplied by the weight factor of the corresponding image region, and the product is the static index of that image region. This process is repeated to obtain the static index of each image region. Here, the static index of image regions at the same location is the same.

[0074] After determining the regional static indicators corresponding to each image region, one possible approach to acquiring the target vibration region is as follows: Figure 5 As shown, determining the target vibration region in the image based on the static index of the image regions at the same position in the multi-frame images may include:

[0075] Step 501: Determine the image region whose static indicators satisfy the second preset condition;

[0076] Step 502: Determine the target vibration region based on the image region whose static indicators meet the second preset condition;

[0077] The condition that the regional static index satisfies the second preset condition includes: when the regional static index represents the similarity between image regions at the same location, the regional static index is greater than or equal to a first threshold; when the regional static index represents the difference between image regions at the same location, the regional static index is less than or equal to a second threshold.

[0078] It should be noted that step 501 is the initial screening of image regions; step 502 is to obtain the target vibration region based on the initially screened image regions. In step 501, the region static index satisfying the second preset condition may include: when the region static index represents the similarity between image regions at the same location, the region static index is greater than or equal to a first threshold; or when the region static index represents the difference between image regions at the same location, the region static index is less than or equal to a second threshold. That is, the initial screening conditions for image regions differ depending on the region static index used. Specifically, when the region static index represents the similarity between image regions at the same location, the initial screening is performed according to the previously calculated similarity between image regions at the same location, i.e., the region static index (similarity) is greater than or equal to the first threshold. The first threshold can be selected based on the actual situation. When the region static index represents the difference between image regions at the same location, the initial screening is performed according to the previously calculated difference between image regions at the same location, i.e., the region static index (difference) is less than or equal to the second threshold, where the second threshold can also be selected based on the actual situation.

[0079] After selecting the image regions that meet the conditions, step 502 may specifically include: directly determining the image regions whose static indicators meet the second preset conditions as the target vibration regions;

[0080] Alternatively, the image region whose static indicators meet the second preset condition can be adjusted to obtain the target vibration region whose pixel area is greater than or smaller than the image region whose static indicators meet the second preset condition.

[0081] It should be noted that, for determining the target vibration region, one possible approach is to directly use the selected image region as the target vibration region. This approach is suitable when the pixel area of ​​the vibrating element is unknown. Another approach is to analyze the pixel area difference between the vibrating element and the image region, and then determine the target vibration region based on this difference.

[0082] For the second scenario, the specific process may include: First, analyzing the pixel area difference between the icon to be detected and the image region; then, if the pixel area difference is such that the pixel area of ​​the icon to be detected is less than the pixel area of ​​the image region, reducing the pixel area of ​​the image region corresponding to the static index of the region meeting the set conditions; determining the reduced image region as the target vibration region; or, if the pixel area difference is such that the pixel area of ​​the icon to be detected is greater than the pixel area of ​​the image region, increasing the pixel area of ​​the image region corresponding to the static index of the region meeting the set conditions; determining the increased image region as the target vibration region; or, if the pixel area difference is such that the pixel area of ​​the icon to be detected is equal to the pixel area of ​​the image region, determining the image region whose static index of the region meets the second preset condition as the target vibration region.

[0083] It should be noted that this situation applies when the size of the vibrating element is known. The relationship between image size and pixel area has already been described. When partitioning each frame of the image, the pixel area of ​​each partitioned image region can be determined. There are three cases regarding the difference in pixel area between the vibrating element and the image region. In one case, the pixel areas of the vibrating element and the image region are equal; in this case, the target vibration region is also the selected image region. In another case, the vibrating element is smaller than the pixel area of ​​the image region; in this case, based on the image region whose static indicators meet the set conditions, its pixel area is reduced, and the reduced image region is determined as the target vibration region. For a detailed understanding, see [reference needed]. Figure 6 As shown, based on the selected A2, its pixel area is reduced to form A2'( Figure 6 The circular region in the image (A2') is the target vibration region. In one case, the vibration element is larger than the pixel area between image regions. In this case, based on the image region whose static indicators meet the set conditions, its pixel area is increased, and the increased image region is determined as the target vibration region. For a detailed understanding, see [example missing]. Figure 6 As shown, based on the selected A1, its pixel area is increased to form A1'( Figure 7 The shaded area in the diagram), A1' is the target vibration area. It should be noted that... Figure 6 This only shows the target vibration region in one frame out of a multi-frame image. Since the multiple frames are extracted from the same video stream, the target vibration region is contained in each frame at the same pixel coordinates.

[0084] Step 104: Identify the type and state changes of the target vibration elements contained in the target vibration region in the video stream corresponding to the multi-frame images.

[0085] Specifically, such as Figure 7 As shown, identifying the type of the target vibration element contained in the target vibration region in the video stream corresponding to the multi-frame images may include:

[0086] Step 701: Obtain the vibration element template library and the neural network model; the vibration element template library includes various vibration elements and the corresponding type of each vibration element; the neural network model is obtained by learning and training with different vibration elements and the corresponding type of each vibration element;

[0087] Step 702: Detect the target vibration element in the target vibration region according to the vibration element template library or the neural network model, and determine the type of the target vibration element.

[0088] It should be noted that step 104 involves detecting vibration elements in the target vibration region and determining the type of the detected vibration and its state changes. Specifically, steps 701 and 702 involve identifying target vibration elements and determining the type of target vibration elements in the target vibration region.

[0089] Specifically, in step 701, before identifying the target vibration region, a vibration element template library or a neural network model needs to be obtained. The vibration element template library can be a pre-stored database of multiple template images and their corresponding vibration elements and types. The neural network model can be pre-trained using a large number of different vibration elements and their corresponding types.

[0090] Specifically, for step 702, identifying the type of target vibration element in the target vibration region using a vibration element template library can be done as follows: Using template matching technology, the pixels of the target vibration region are traversed, and the pixel region with the highest pixel matching value with the template image in the vibration element template library is obtained as the target vibration element. Simultaneously, the type of the target vibration element is determined based on the type of vibration element in the template image. For example, if the type of vibration element in the template image is a skill icon, then the matched target vibration element will also be a skill icon. More precise identification is possible, such as identifying the type of the skill icon, such as continuous attack skills, jump skills, etc. In practical applications, the vibration element template library can be selected or designed according to the actual situation. It should be noted that the template images in the vibration element template library can only contain images of vibration elements. When using this vibration element template library to identify vibration elements in the target vibration region, the images in the vibration element template library can be directly matched with the pixels in the target vibration region to determine the target vibration element and its type. Alternatively, the vibration elements in the target vibration region can be extracted, and their images can be matched with images in a vibration element template library to determine the target vibration element and its type. Another optional implementation method is as follows: Machine learning, such as using a convolutional neural network (CNN), can be used to train a vibration element detection model. The trained CNN model can then be used to predict the type of vibration element from the target vibration region, or to predict the type of vibration element from the vibration region and locate the specific position of the vibration element within the target vibration region.

[0091] In some embodiments, identifying the state change of the target vibration element contained in the target vibration region in the video stream corresponding to the multi-frame images includes: comparing the pixels of the target vibration element contained in two consecutive frames of the multi-frame images to determine the state change of the vibration element. The two consecutive frames can be two frames sequentially arranged in time. That is, comparing the vibration elements in the target vibration region of two consecutive frames of the multi-frame images to determine whether the state of the vibration element has changed. It should be noted that in this embodiment, there are multiple ways to compare the vibration elements in the target vibration region of two consecutive frames of the multi-frame images. One optional method is to compare the state of the vibration element in every two consecutive frames of the multi-frame images to determine whether the state of the vibration element in the multi-frame images has changed. For example, Figure 4The three image frames shown, assuming the target vibration regions are A1, B1, and C1, require comparison of the vibration elements between A1 and B1, and between B1 and C1 to determine if their states have changed. In practical applications, another possible approach is to compare the states of vibration elements between consecutive frames in a chronological order to determine if the states of vibration elements across multiple frames have changed. For example, if the image set contains four frames, and the target vibration regions are A1, B1, C1, and D1, the comparison of the vibration elements between A1 and B1, and between C1 and D1, is necessary to determine if their states have changed. The specific comparison method can be chosen based on the actual situation.

[0092] Step 105: In response to the state change reaching the first preset condition, generate a vibration control command that matches the type of the target vibration element.

[0093] The state change reaching the first preset condition includes:

[0094] The target vibration element contains at least one attribute value among the basic attributes of the pixels, and the contrast value of the pixel in the two frames exceeds a third threshold; wherein the basic attributes include color, brightness, contrast, saturation, and sharpness.

[0095] It should be noted that step 105 is the step of determining the timing and type of generating vibration control commands. That is, when the state change of the target vibration element reaches a first preset condition, a vibration control command matching the type of the target vibration element is generated. Meeting the first preset condition may include: at least one attribute value among the basic attributes of the pixels contained in the target vibration element has a contrast value exceeding a third threshold in two frames of images. The basic attributes may include color, brightness, contrast, saturation, sharpening, etc.

[0096] Wherein, at least one attribute value of the basic attributes of the pixels contained in the target vibration element exceeds a third threshold in the comparison value between two frames of images. For example, when the ratio of the average brightness of the pixels contained in the vibration element in the two frames exceeds the third threshold, the vibration triggering condition is met. At this time, a corresponding vibration control command should be generated, and a vibration control command matching the target vibration element should also be generated.

[0097] The change in color of the target vibrating element can refer to a change in the color of the vibrating element across two frames of images. Specifically, the vibrating element (which is essentially still an image, only its pixel area is the same as the vibrating element's pixel area) can be extracted from the two frames of images and converted to different color spaces (e.g., Hue, Saturation, Value-HSV color spaces). Thresholding can then be used to determine if the color has changed between the two frames. More specifically, two color spaces are defined according to the usage scenario and denoted as R1 and R2 respectively. Using masking techniques, the number of pixels (num1) of the vibrating element in one frame within R1 and the number of pixels (num2) of the vibrating element in the other frame within R2 are calculated. It is then determined whether num1 is less than color threshold 1 and / or whether num2 is greater than color threshold 2. When num1 is less than color threshold 1 and / or num2 is greater than color threshold 2, the vibration triggering condition is met, and a corresponding vibration control command is generated. The third threshold, color threshold 1, and color threshold 2 can all be set according to actual conditions. The cases for other basic attributes are similar and can be understood without further elaboration. Here, satisfying the first preset condition can mean that the contrast value of at least one of the basic attributes of the pixels contained in the target vibration element in the two consecutive frames exceeds the third threshold. This can refer to at least one of color, brightness, contrast, saturation, and sharpness exceeding the third threshold in the two frames. It should be understood that each attribute value should have a corresponding third threshold. That is, the third thresholds corresponding to different attribute values ​​may be the same or different.

[0098] In this embodiment, generating a vibration control command that matches the type of the target vibration element may refer to determining a vibration control command that matches the type of the target vibration element based on a pre-designed mapping relationship between the types of vibration elements and vibration control commands. Then, the vibration control command is sent to the vibration control system of a terminal or electronic device to drive a motor to vibrate, producing the desired vibration effect.

[0099] The vibration control method provided in this application acquires multiple consecutive frames of images, analyzes these frames to determine the types and state changes of the vibration elements contained within them, and thus determines whether vibration is required in the current scene. This method is highly versatile, scalable, and portable. It no longer relies on fixed area pixel information and can automatically detect and extract vibration elements requiring motion change detection from the image based on the characteristics of the video / game itself, thereby triggering vibration.

[0100] In order to understand this application, Figures 8 to 13 The detection process for the icon to be detected is described below. Figures 8 to 13In this process, the acquired multi-frame images include N frames; each frame is a full-screen image. Based on this, the vibration control process can be specifically as follows: Figure 8 As shown, it may include:

[0101] 1. Obtain n consecutive frames of images from the target game application video, and perform corresponding pre-processing on each frame.

[0102] Specifically, preprocessing includes at least one of the following:

[0103] A) Including but not limited to conventional image processing techniques such as smoothing and thresholding, the above preprocessing methods are used to reduce noise in the image and highlight specific areas to simplify image processing tasks.

[0104] B) Including but not limited to down-sampling, used to reduce image resolution to reduce computational complexity.

[0105] C) Including but not limited to grayscale conversion, converting RGB or HSV images to grayscale images, reducing the number of color channels, thereby reducing computational complexity.

[0106] It should be noted that n is greater than or equal to 2.

[0107] 2. Vibration area detection.

[0108] Here, the region in an image containing vibrating elements is called a vibration region. In game applications, vibrating elements often exhibit high similarity within a certain number of consecutive frames. We partition the full-screen image and compare consecutive frames to find regions with high similarity or low difference, thereby detecting the possible regions where vibrating elements may reside. Specifically, for example... Figure 9 The diagram illustrates a flowchart for vibration region detection. The specific process may include: partitioning the preprocessed n consecutive full-screen images into segments and assigning weights; calculating the similarity between sub-image segments at the same location (i.e., the image regions mentioned above); and selecting sub-image intervals that meet the similarity criteria as the target vibration region, i.e., image regions with a similarity greater than or equal to a first threshold.

[0109] Here, assuming we partition the full-screen image R, we represent the partitions as {a1, a2, ..., a...} n}, and assign weights {w1, w2, ..., w} to each sub-image interval according to the specific task. n}, where R = a1 + a2 + ... + a n ,w1+w2+…+w n=1. Then, for image regions at the same position in n frames (the partitioned sub-image intervals), image processing techniques such as frame differencing or auto-correlation are used to calculate the similarity {s1,s2,…,s} of each image partition within n consecutive frames. n} or the degree of difference {d1,d2,…,d} n By multiplying the similarity or difference by the weight of each sub-image interval (also known as the image region), we can obtain the regional static index {s1*w1,s2*w2,…,s} for each sub-image interval. n *w n} or {d1*w1,d2*w2,…,d n *w n The process involves identifying sub-image intervals where the similarity is greater than a certain preset threshold or the difference is less than a certain preset threshold. Then, the target vibration region is obtained based on the initially screened sub-image intervals (the specific method has been described previously and will not be repeated here).

[0110] Here, the process of obtaining the target vibration region described above is illustrated with an example. For instance, the full-screen image is evenly divided into 6-8 sub-image intervals, and each sub-image interval is assigned the same weight, that is, the weight of each sub-image interval is 1 / n. The difference degree in each region is calculated using the image difference method, and then the sub-image interval with the difference degree less than a certain threshold is found as our target vibration region.

[0111] For example, the full-screen image can be evenly divided into 6-8 sub-image intervals, and each sub-image interval can be assigned a different weight {w1, w2, ..., w} depending on the actual application scenario. n}, calculate the autocorrelation degree of each sub-image interval {cor1,cor2,…,cor n Finally, for each sub-image interval, the weight is multiplied by the autocorrelation degree to obtain the similarity value of the sub-image interval, and then the sub-image interval with a similarity greater than a certain threshold is found as our target vibration region.

[0112] 3. Vibration Element Detection. This involves detecting target vibration elements within the aforementioned target vibration area. For example, in game applications, this involves identifying skill icons, aiming to accurately locate and identify the desired target vibration element within the target vibration area. For instance, in a Multiplayer Online Battle Arena (MOBA) video, based on the aforementioned target vibration area, the skill icon of the current hero is selected as the target vibration element to be detected. The goal is to detect the position of the hero skill icon and its corresponding type tag ID. Two vibration element identification methods are illustrated below, including template matching. Template matching, as a general target identification method, involves the following steps: traversing the template image, by traversing the vibration area pixels, locating the pixel area with the highest matching value based on the template image as the successfully detected target vibration element. Simultaneously, the type of successfully detected target vibration element is determined by the template image. The process is shown below. Figure 10 .

[0113] Here, vibration element detection methods can also include machine learning. Machine learning methods have become the mainstream approach for image processing tasks in recent years. Taking the detection of skill icons in game videos as an example, we can use a convolutional neural network as a vibration element detection model. The trained CNN model can predict the type of vibration element and locate the specific position of the target vibration element within the vibration area. The process is shown below. Figure 11 .

[0114] 4. Vibration Element State Change Detection. For successfully detected target vibration elements, we compare the pixels of the target vibration element in two consecutive frames of the game video and detect its state changes. If the state of the target vibration element changes and meets the predefined vibration trigger conditions, a corresponding vibration control command is issued. During gameplay, changes in the state of vibration elements (such as hero skills) usually represent corresponding user actions. Therefore, vibration element state detection aims to detect whether the vibration trigger conditions are met in the current frame. For example, in a multiplayer online battle arena game video, based on the aforementioned successfully detected vibration elements (i.e., hero skills), we analyze the vibration element area in the game video frame by frame to detect changes in the hero skill icon's state. Here, we can divide the hero skill state into unreleased state, released state, and cooldown state. State change detection can detect the changes between any two states as needed. For some heroes, due to the special nature of their skills, there may be multiple released states or multiple unreleased states, which can be adjusted according to specific needs. Based on the type of change between different states, a corresponding state change ID is assigned. The following examples illustrate two methods for detecting changes in the state of vibrating elements. These methods include, but are not limited to, comparing the brightness, contrast, saturation, and sharpness of basic image attributes of vibrating elements in two consecutive frames. After calculating the value of a certain attribute, a threshold method is used to determine whether a preset vibration trigger condition (i.e., the first preset condition) has been met. Another method for detecting changes in the state of vibrating elements includes, but is not limited to, comparing whether the vibrating elements have undergone color changes in two consecutive frames. The vibrating element image can be converted to a different color space (e.g., Hue, Saturation, Value-HSV color space), and a threshold method can also be used to determine whether the color change of the vibrating element image meets the preset vibration trigger condition.

[0115] For the first method of detecting changes in the state of vibration elements, the specific process may include: extracting the target vibration element from two consecutive frames of images and calculating the average brightness value. The average brightness of the pixels of the target vibration element in the two consecutive frames is denoted as b. i-1 ,b i If the ratio of the two is b i / b i-1 If the brightness is below a certain threshold (i.e., a third threshold), it proves that the state of the target vibration element in the two consecutive frames has significantly darkened, and its state change can be considered to meet the vibration triggering condition. The process is as follows: Figure 12 .

[0116] The specific process for detecting changes in the state of vibration elements using the second method described above can include: setting ranges within two HSV color spaces, extracting the target vibration element images from two consecutive frames, using masking techniques to calculate and count the number of pixels in the target vibration element images within the two color ranges, and determining whether the vibration triggering condition is met if the number of pixels in color range 1 drops to a certain preset threshold and / or the number of pixels in color range 2 rises to another preset threshold. The process is described in [link to process details]. Figure 13 The calculation and judgment steps have been explained in detail above, and will not be repeated here.

[0117] 5. Issue corresponding vibration control commands based on the type of vibrating element and its state changes. Based on the type of vibrating element and its state changes, match the corresponding vibration control command to control the motor vibration. For example, in a multiplayer online battle arena game video, based on the successfully detected hero skill ID and state change ID, match the corresponding preset vibration control command ID and issue this preset vibration control command to control the motor to generate a preset vibration sensation (i.e., vibration effect).

[0118] This application also provides an icon detection device, such as... Figure 14 As shown, the device 140 may include:

[0119] The acquisition module 1401 is used to acquire multiple consecutive frames of images; each frame of the multiple frames of images contains a target vibration element that may trigger vibration.

[0120] The partitioning module 1402 is used to partition each frame of the multi-frame images according to the same partitioning rules to obtain multiple image regions contained in each frame image.

[0121] The determination module 1403 is used to determine the target vibration region in the image based on the region static index corresponding to the image regions at the same position in the multi-frame images; the region static index is used to characterize the degree of similarity or difference between the image regions at the same position in the multi-frame images.

[0122] The identification module 1404 is used to identify the type and state changes of the target vibration element contained in the target vibration region in the video stream corresponding to the multi-frame images;

[0123] The generation module 1405 is used to generate a vibration control command that matches the type of the target vibration element in response to the state change reaching a first preset condition.

[0124] In some embodiments, the apparatus further includes a preprocessing module for preprocessing each frame of the multi-frame images, wherein the preprocessing includes at least one of the following: smoothing, thresholding, sampling, and grayscale processing.

[0125] In some embodiments, the determining module is further configured to: determine the weight factor corresponding to the image regions at the same position in the multi-frame images; determine the similarity or difference between the image regions at the same position in the multi-frame images; and obtain the region static index based on the similarity and the weight factor, or obtain the region static index based on the difference and the weight factor.

[0126] In some embodiments, the determining module is further configured to: determine an image region whose regional static index meets a second preset condition; and determine the target vibration region based on the image region whose regional static index meets the second preset condition; wherein, the regional static index meeting the second preset condition includes: when the regional static index represents the similarity between image regions at the same position, the regional static index is greater than or equal to a first threshold; when the regional static index represents the difference between image regions at the same position, the regional static index is less than or equal to a second threshold.

[0127] In some embodiments, the determining module is further configured to: directly determine the image region whose regional static index meets the second preset condition as the target vibration region; or, adjust the image region whose regional static index meets the second preset condition to obtain the target vibration region whose pixel area is greater than or less than the image region whose regional static index meets the second preset condition.

[0128] In some embodiments, the identification module is configured to: obtain a vibration element template library and a neural network model; the vibration element template library includes a variety of vibration elements and the type corresponding to each vibration element; the neural network model is obtained by learning and training through different vibration elements and the type corresponding to each vibration element; and detect target vibration elements in the target vibration region according to the vibration element template library or the neural network model, and determine the type of the target vibration element.

[0129] In some embodiments, the identification module is further configured to: compare the pixels of the target vibration element contained in two consecutive frames of the multi-frame image to determine the state change of the vibration element.

[0130] It should be noted that the description of the apparatus in this application embodiment is similar to the description of the method embodiment described above, and has similar beneficial effects as the method embodiment; therefore, it will not be repeated. For technical details not disclosed in this apparatus embodiment, please refer to the description of the method embodiment of this application for understanding.

[0131] It should be noted that, in the embodiments of this application, if the above-mentioned vibration control method is implemented as a software functional module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the embodiments of this application, or the part that contributes to the related technology, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a terminal to execute all or part of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, mobile hard drives, read-only memory (ROM), magnetic disks, or optical disks. Thus, the embodiments of this application are not limited to any specific hardware and software combination.

[0132] This application also provides an electronic device, which includes a processor and a memory for storing a computer program that can run on the processor, wherein the processor executes the steps of the method described above when running the computer program.

[0133] The processor typically controls the overall operation of the electronic device. The memory is configured to store instructions and applications executable by the processor, and can also cache data to be processed or already processed by the processor and various modules in the electronic device. It can be implemented using flash memory or random access memory (RAM).

[0134] This application provides a storage medium storing executable instructions, wherein a computer program is stored. When the computer program is executed by a processor, it will cause the processor to execute the vibration control method provided in this application.

[0135] In some embodiments, the storage medium may be a computer-readable storage medium, such as a ferromagnetic random access memory (FRAM), a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), flash memory, magnetic surface memory, optical disc, or a compact disk-read-only memory (CD-ROM); or it may be a device that includes one or any combination of the above-mentioned memories.

[0136] In some embodiments, executable instructions may take the form of a program, software, software module, script, or code, written in any form of programming language (including compiled or interpreted languages, or declarative or procedural languages), and may be deployed in any form, including as a standalone program or as a module, component, subroutine, or other unit suitable for use in a computing environment.

[0137] As an example, executable instructions may, but do not necessarily, correspond to files in a file system. They may be stored as part of a file that holds other programs or data, for example, in one or more scripts within a Hyper Text Markup Language (HTML) document, in a single file dedicated to the program in question, or in multiple co-located files (e.g., files storing one or more modules, subroutines, or code sections). As an example, executable instructions may be deployed to execute on a single computing device, or on multiple computing devices located in one location, or on multiple computing devices distributed across multiple locations and interconnected via a communication network.

[0138] It should be noted that the technical features described in the embodiments of this application can be combined arbitrarily without conflict.

[0139] It should be understood that the phrase "one embodiment" or "an embodiment" throughout the specification means that a specific feature, structure, or characteristic related to the embodiment is included in at least one embodiment of this application. Therefore, phrases such as "in some embodiments" or "in one embodiment" appearing throughout the specification do not necessarily refer to the same embodiment. Furthermore, these specific features, structures, or characteristics can be combined in any suitable manner in one or more embodiments. It should be understood that in the various embodiments of this application, the sequence numbers of the above-described processes do not imply a sequential order of execution; the execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application. The sequence numbers of the above-described embodiments are merely descriptive and do not represent the superiority or inferiority of the embodiments.

[0140] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element. In the several embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. The device embodiments described above are merely illustrative; for example, the division of units is merely a logical functional division, and in actual implementation, there may be other division methods, such as: multiple units or components may be combined, or integrated into another system, or some features may be ignored or not performed. The above descriptions are merely embodiments of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of protection of the claims.

Claims

1. A vibration control method, characterized in that, include: Acquire multiple consecutive frames of images; each frame of the multiple frames contains a target vibration element that may trigger vibration; Each frame of the multi-frame image is partitioned according to the same partitioning rules to obtain multiple image regions contained in each frame image; The target vibration region in the image is determined based on the region static index corresponding to the image regions at the same position in the multi-frame images; the region static index is used to characterize the degree of similarity or difference between the image regions at the same position in the multi-frame images. Identify the type and state changes of the target vibration elements contained in the target vibration region in the video stream corresponding to the multi-frame images; In response to the state change reaching a first preset condition, a vibration control command matching the type of the target vibration element is generated.

2. The method according to claim 1, characterized in that, Determining the target vibration region in the image based on the region static index corresponding to the image regions at the same position in the multi-frame images includes: The image region whose static indicators satisfy the second preset condition is identified. The target vibration region is determined based on the image region whose static indicators satisfy the second preset condition. The condition that the regional static index satisfies the second preset condition includes: when the regional static index represents the similarity between image regions at the same location, the regional static index is greater than or equal to a first threshold; when the regional static index represents the difference between image regions at the same location, the regional static index is less than or equal to a second threshold.

3. The method according to claim 2, characterized in that, Determining the target vibration region based on the image region whose regional static index meets the second preset condition includes: directly determining the image region whose regional static index meets the second preset condition as the target vibration region; Alternatively, the image region whose static indicators meet the second preset condition can be adjusted to obtain the target vibration region whose pixel area is greater than or smaller than the image region whose static indicators meet the second preset condition.

4. The method according to claim 1, characterized in that, The identification of the type of target vibration element contained in the target vibration region in the video stream corresponding to the multi-frame images includes: A vibration element template library and a neural network model are obtained; the vibration element template library includes various vibration elements and the corresponding types of each vibration element; the neural network model is obtained by learning and training with different vibration elements and the corresponding types of each vibration element. The target vibration element in the target vibration region is detected based on the vibration element template library or the neural network model, and the type of the target vibration element is determined.

5. The method according to claim 1 or 4, characterized in that, Identifying the state changes of the target vibration elements contained in the target vibration region within the video stream corresponding to the multi-frame images includes: By comparing the pixels of the target vibration element in two consecutive frames of the multi-frame image, the state change of the vibration element is determined.

6. The method according to claim 5, characterized in that, The state change reaching the first preset condition includes: at least one attribute value of the basic attributes of the pixels contained in the target vibration element has a contrast value exceeding a third threshold in the two consecutive frames of images. The basic attributes include color, brightness, contrast, saturation, and sharpening.

7. The method according to claim 1, characterized in that, Before partitioning each frame of the multi-frame images according to the same partitioning rule, the method further includes: Each frame of the multi-frame image is preprocessed, wherein the preprocessing includes at least one of the following: smoothing, thresholding, sampling, and grayscale processing.

8. A vibration control device, characterized in that, include: An acquisition module is used to acquire multiple consecutive frames of images; each frame of the multiple frames of images contains a target vibration element that may trigger vibration. The partitioning module is used to partition each frame of the multi-frame images according to the same partitioning rules to obtain multiple image regions contained in each frame image. The determination module is used to determine the target vibration region in the image based on the region static index corresponding to the image regions at the same position in the multi-frame images; the region static index is used to characterize the degree of similarity or difference between the image regions at the same position in the multi-frame images. The identification module is used to identify the type and state changes of the target vibration elements contained in the target vibration region in the video stream corresponding to the multi-frame images; The generation module is used to generate a vibration control command that matches the type of the target vibration element in response to the state change reaching a first preset condition.

9. An electronic device, characterized in that, The electronic device includes: a processor and a memory for storing a computer program capable of running on the processor, wherein, when the processor runs the computer program, it performs the steps of the method according to any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 7.

Images