Vibration control method and device, mobile terminal and storage medium

Abstract

The application discloses a vibration control method and device, a mobile terminal and a storage medium. The vibration control method is applied to the mobile terminal, and the vibration control method comprises the following steps: in a game running process, in response to a drawing instruction of a graphic application programming interface (API), drawing data of a game interface to be drawn is acquired; based on the drawing data, content information of a specified region in the game interface is determined; and if the content information satisfies a vibration triggering condition, the mobile terminal is controlled to vibrate. The method can reduce power consumption during vibration control during the game.

Description

Technical Field

[0001] This application relates to the field of mobile terminal technology, and more specifically, to a vibration control method, device, mobile terminal, and storage medium. Background Technology

[0002] With the rapid advancement of technology and living standards, mobile devices (such as smartphones and tablets) are widely used, and as their configurations improve, more and more users are using them for gaming. To enhance the gaming experience, 4D (4-dimensional) vibration is often provided. Vibration control solutions for mobile devices typically employ artificial intelligence (AI) to determine the scenarios requiring vibration. However, using AI algorithms to identify vibration scenarios leads to higher power consumption on mobile devices. Summary of the Invention

[0003] In view of the above problems, this application proposes a vibration control method, device, mobile terminal and storage medium.

[0004] In a first aspect, embodiments of this application provide a vibration control method applied to a mobile terminal. The method includes: during game operation, in response to drawing instructions from a graphics application programming interface (API), acquiring drawing data of a game interface to be drawn; determining content information of a specified area in the game interface based on the drawing data; and controlling the mobile terminal to vibrate if the content information meets vibration triggering conditions.

[0005] Secondly, this application provides a vibration control device applied to a mobile terminal. The device includes a data acquisition module, a content acquisition module, and a vibration triggering module. The data acquisition module is used to acquire drawing data of the game interface to be drawn in response to drawing instructions from a graphics application programming interface (API) during game operation. The content acquisition module is used to determine content information of a specified area in the game interface based on the drawing data. The vibration triggering module is used to control the vibration of the mobile terminal if the content information meets the vibration triggering conditions.

[0006] Thirdly, embodiments of this application provide a mobile terminal, including: one or more processors; a memory; and one or more applications, wherein the one or more applications are stored in the memory and configured to be executed by the one or more processors, and the one or more applications are configured to perform the vibration control method provided in the first aspect above.

[0007] Fourthly, embodiments of this application provide a computer-readable storage medium storing program code, which can be invoked by a processor to execute the vibration control method provided in the first aspect above.

[0008] The solution provided in this application obtains the drawing data of the game interface to be drawn by responding to the drawing instructions of the graphics API during game operation. Based on the drawing data, the content information of a specified area in the game interface is determined. If the content information meets the vibration trigger condition, the vibration of the mobile terminal is controlled. In this way, when drawing the game interface to be displayed, the content information of a specified area is determined according to the drawing data of the game interface, and then the vibration during the game is controlled. This avoids the problem of high power consumption caused by using AI algorithms to identify vibration scenes and improves the battery life of the mobile terminal. Attached Figure Description

[0009] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0010] Figure 1 A flowchart of a vibration control method according to an embodiment of this application is shown.

[0011] Figure 2 A schematic diagram illustrating the OpenGL rendering principle provided in one embodiment of this application is shown.

[0012] Figure 3 A flowchart of a vibration control method according to another embodiment of this application is shown.

[0013] Figure 4 A flowchart of a vibration control method according to yet another embodiment of this application is shown.

[0014] Figure 5 This illustration shows a schematic diagram of a software architecture for implementing the vibration control method in this embodiment of the present application.

[0015] Figure 6 A flowchart of a vibration control method according to another embodiment of this application is shown.

[0016] Figure 7 A schematic diagram of an interface provided in an embodiment of this application is shown.

[0017] Figure 8 This illustration shows another interface diagram provided by an embodiment of this application.

[0018] Figure 9 This illustration shows yet another interface diagram provided in an embodiment of this application.

[0019] Figure 10 This illustration shows another interface diagram provided by an embodiment of the present application.

[0020] Figure 11 This illustration shows yet another interface diagram provided by an embodiment of this application.

[0021] Figure 12 This illustration shows yet another interface diagram provided in an embodiment of this application.

[0022] Figure 13 This illustration shows another interface diagram provided by an embodiment of the present application.

[0023] Figure 14 A block diagram of a vibration control device according to an embodiment of this application is shown.

[0024] Figure 15 This is a block diagram of a mobile terminal for executing the vibration control method according to an embodiment of this application.

[0025] Figure 16 This is a storage unit in this application embodiment for storing or carrying program code that implements the vibration control method according to this application embodiment. Detailed Implementation

[0026] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings.

[0027] As mobile devices become increasingly powerful, more and more users are choosing to play games on them. When gaming on mobile devices, some solutions incorporate in-game content, such as shooting, being shot, and killing, using motor vibrations to provide 4D vibration feedback. This enhances player immersion and improves the user experience.

[0028] In related technologies, mobile terminals typically identify scenarios that require vibration by acquiring game sounds and capturing game screenshots, then using AI algorithms and additional hardware acceleration to identify the scenarios that require vibration and then triggering the vibration.

[0029] After a long period of research, the inventors discovered that using AI algorithms or adding extra processing chips to identify user interface (UI) or game audio to identify the scene that triggers vibration has two problems. First, because of screenshots, using AI algorithms or chips for processing leads to high power consumption and increased costs. Second, there are various sound interferences in games, and users can customize the position of the UI interface, resulting in low recognition accuracy.

[0030] To address the aforementioned issues, the inventors have proposed a vibration control method, device, mobile terminal, and storage medium as described in this application. These methods enable the determination of content information in a designated area based on the rendering data of the game interface during gameplay, thereby controlling vibration during gameplay. This avoids the high power consumption issues associated with using AI algorithms to identify vibration scenarios, thus improving the battery life of the mobile terminal. The specific vibration control method will be described in detail in subsequent embodiments.

[0031] Please see Figure 1 , Figure 1 A schematic flowchart of a vibration control method according to an embodiment of this application is shown. In a specific embodiment, the vibration control method is applied to, for example... Figure 13 The vibration control device 400 shown and the mobile terminal 100 equipped with the vibration control device 400 are also shown. Figure 14 The following will use a mobile terminal as an example to illustrate the specific process of this embodiment. Of course, it is understood that the mobile terminal used in this embodiment can be a smartphone, tablet computer, etc., and is not limited thereto. The following will focus on... Figure 1 The process shown is described in detail. The vibration control method may specifically include the following steps:

[0032] Step S110: During game operation, in response to the drawing instructions of the graphics application programming interface (API), obtain the drawing data of the game interface to be drawn.

[0033] In this embodiment of the application, the mobile terminal can monitor the calls to the graphics API during the operation of the game, so as to obtain the drawing data for drawing the game interface when the drawing instruction of the graphics API is detected, and then determine the content information of a specified area in the game interface based on the drawing data.

[0034] In some implementations, the above graphics APIs may include OpenGL, OpenGL ES, OpenCV, Vulkan, DirectX, Metal, etc. Of course, the specific graphics API used in the mobile terminal is not limited.

[0035] In some implementations, the mobile terminal may include a CPU (Central Processing Unit) and a GPU (Graphics Processing Unit). When rendering a game interface, the mobile terminal can determine the graphics API used to render the game screen, and then call the determined graphics API to communicate with the driver of the underlying hardware (e.g., the GPU), thereby activating the corresponding GPU functions to render the game screen. The determination of the graphics API to be called can rely on the CPU, while the calling of the graphics API to achieve the corresponding rendering can rely on the GPU.

[0036] The following example uses OpenGL to illustrate how mobile devices render game interfaces.

[0037] The CPU can determine the vertex information (vertex position, normal direction, vertex color, texture coordinates, etc.) and texture information needed for rendering based on the data from the game interface, and then provide them to the GPU. After determining the required graphics API instructions, the CPU submits them to the GPU, which then executes the corresponding graphics API instructions sequentially according to the OpenGL pipeline to complete the rendering of the game interface. A schematic diagram of the OpenGL pipeline is shown below. Figure 2 As shown.

[0038] The vertex array buffer is the vertex information provided by the CPU to the GPU, including vertex position, vertex color (only the color of the vertex, unrelated to the texture color), texture coordinates (used for texture mapping), and other vertex information. The vertex shader is the program that processes the vertex information provided by the vertex array buffer. The vertex shader is executed once for each vertex provided by the vertex array buffer. The next stage of the vertex shader is primitive assembly. Primitives are geometric objects such as triangles, lines, or point sprites. In this stage, the vertices output by the vertex shader are combined into primitives.

[0039] Next, rasterization is performed. Rasterization is the process of converting primitives into a set of two-dimensional fragments, which are then processed by the fragment shader (the input to the fragment shader). These two-dimensional fragments represent pixels that can be drawn on the screen. The mechanism used to generate each fragment value from the vertex shader output assigned to each primitive vertex is called interpolation. That is, the scattered vertex information provided by the CPU is transformed into dense pixels on the screen. After primitive assembly, the vertices can be understood as becoming graphics. During rasterization, the pixels (texture coordinates, color, etc.) of that graphics region are interpolated based on the shape of the graphics. It is worth noting that the pixels at this point are not pixels on the screen and do not have color. The coloring is done next in the fragment shader.

[0040] Fragment shading is a procedure in OpenGL used to calculate the color of fragments (pixels). Fragment shading is a pixel-by-pixel operation, meaning the fragment shader is executed once for each pixel, and it operates in parallel. The fragment shader has three main data inputs: uniform variables, vertex shader input variables (also known as varying variables), and samplers. The values ​​of uniform variables are consistent across the vertex and fragment shaders within the same OpenGL shader program. Vertex shader input variables are generally different for each pixel; their values ​​are determined by interpolating the values ​​from the vertex shaders corresponding to the vertices that make up the primitive, based on the pixel positions. Samplers are used to obtain the pixel colors of a predefined texture.

[0041] After fragment shading is completed, fragment-by-fragment operations are performed, which mainly include: pixel assignment testing, clipping testing, stencil and depth testing, blending, and dithering. Finally, the generated fragments are placed in the frame buffer, and the screen draws the fragments in the buffer to complete the rendering and display of the game interface.

[0042] During the rendering and display of a game interface on a mobile terminal, the CPU needs to provide corresponding graphics API drawing instructions to complete the GPU's rendering process. The submission of these graphics API drawing instructions by the CPU can be monitored, allowing the CPU to obtain the drawing data used to render the game interface based on these instructions.

[0043] In some implementations, the rendering data may include at least vertex information and texture information. Understandingly, vertex information can be used to determine the location of each primitive in the game interface, and thus can be used to determine the content of the area of ​​interest in the game interface. Texture information can be used to determine the specific content information within the primitives, and therefore, the content information of a specified area in the game interface can be subsequently determined based on the vertex information and texture information.

[0044] In some implementations, mobile terminals can intercept graphics API instructions submitted by the CPU to the GPU by hooking the graphics API to determine the rendering data. Specifically, mobile terminals can use hook functions to monitor the API and analyze the rendering instructions, thereby intercepting the corresponding rendering instructions and capturing the rendering data used to render the game interface to be displayed.

[0045] Step S120: Based on the drawing data, determine the content information of a specified area in the game interface.

[0046] In this embodiment, after determining the drawing data of the game interface to be drawn, the mobile terminal can determine the content information of a specified area in the game interface based on the drawing data. The content information to be determined can be set according to the scene settings in the game that require vibration.

[0047] In some implementations, the content information may include the content type of the drawn content, the drawn content itself, and information about changes to the drawn content. Specifically, the content type of the drawn content can be the type corresponding to the content in a specified area of ​​the game interface. For example, if the content in the specified area is a weapon icon, the content type can be weapon type; or if the content in the specified area is a kill count indicator, the content type can be indicators corresponding to different kill counts. The drawn content can be the content in a specified area of ​​the game interface, such as kill count, ammo count, or health. The information about changes to the drawn content can be changes to the content in the specified area, including whether the content has changed or not, and the specific changes when changes occur. Of course, the specific content information that needs to be determined is not limited.

[0048] In some implementations, the designated area in the game interface can be a pre-set area that needs to be monitored, so that vibration is triggered when the content information in the designated area meets certain conditions. The specific location of the designated area in the game interface is not limited; for example, the designated area can be a display area for the number of bullets, a display area for the weapon icon, a display area for the number of kills, etc.

[0049] It's important to note that when a mobile terminal responds to graphics API drawing commands, acquires drawing data, and determines the content information of a specified area based on that data, the normal rendering of the game interface is not affected. In other words, acquiring drawing data and determining the content information of a specified area based on that data, along with responding to graphics API drawing commands and rendering, occur simultaneously. That is, while the CPU submits the corresponding graphics drawing commands to the GPU, the CPU also acquires the drawing data for the game interface to be drawn based on these commands, and determines the content information of a specified area within the game interface based on that data.

[0050] Step S130: If the content information meets the vibration triggering conditions, control the mobile terminal to vibrate.

[0051] In this embodiment, after determining the content information, the mobile terminal can match the content information with the vibration triggering conditions. If the content information meets the vibration triggering conditions, vibration is required, and the mobile terminal can be controlled to vibrate. If the content information does not meet the vibration triggering conditions, the mobile terminal can be deactivated or stopped from vibrating. This achieves the monitoring of whether the content information in a specified area meets the vibration triggering conditions based on the rendering data of the game interface during game operation, and triggers vibration when the conditions are met. This effectively reduces the computational load of the mobile terminal in monitoring vibration scenes during gameplay and avoids the power consumption caused by adding extra processing chips.

[0052] In some implementations, vibration triggering conditions may include: changes in content information relative to previously defined content information in a specified area, or a match between the content information and preset content information. That is, when the content information of the currently determined designated area to be drawn changes relative to previously defined content information in that area, or when the currently determined content information matches preset content information, the mobile terminal is controlled to vibrate. Understandably, in some scenarios, changes in the content information of a designated area may necessitate vibration. For example, if the designated area represents the remaining bullet count of a gun, a change indicates that a shot has been fired, thus triggering vibration. Similarly, if the designated area represents a health bar, a change in the health bar indicates that the user has been attacked (e.g., shot), thus triggering vibration. In other scenarios, a match between the content of a designated area and set content may also necessitate vibration. For example, if the designated area represents the kill count, a kill count reaching a certain value matches a set value, thus triggering vibration. Of course, the specific vibration triggering conditions can be determined based on the actual game scenario.

[0053] It is understood that the vibration control method provided in this application determines the content information of a specified area by acquiring drawing data when an API drawing instruction is detected, and decides whether to vibrate based on the content information. This process is performed before the game interface to be drawn is actually displayed, that is, when the drawing instruction is submitted to the GPU for drawing. Therefore, the real-time performance of vibration control can be improved, avoiding the problem of untimely vibration control in the solution of controlling vibration by taking a screenshot of the game interface and using AI algorithm recognition.

[0054] In some implementations, since the determination of content information for a specified area by acquiring drawing data and deciding whether to vibrate based on the content information when an API drawing instruction is detected occurs during the submission of the drawing instruction to the GPU for drawing, the vibration of the mobile terminal can be controlled simultaneously with the subsequent display of the game interface to be drawn. This ensures consistency between the game interface and the mobile terminal's vibration, improving the accuracy of vibration control. Optionally, the game has a corresponding display frame rate, and each frame of the game screen is displayed according to the display frame rate. The mobile terminal controls the total duration of rendering and display according to the display frame rate. Therefore, the mobile terminal can determine the display time of the game interface to be drawn based on the total rendering and display time, and then control the mobile terminal to vibrate at that display time, achieving precise vibration control.

[0055] The vibration control method provided in this application enables the determination of content information in a designated area based on the rendering data of the game interface during gameplay. This avoids the high power consumption issue associated with using AI algorithms to identify vibration scenes, thus improving the battery life of mobile terminals and avoiding the cost of adding additional chips. Furthermore, since it eliminates the need for screenshots or extracting game sounds to determine the required vibration scene, the accuracy of vibration control is improved. Moreover, because the determination of content information in a designated area by acquiring rendering data and deciding whether to vibrate based on this information occurs when the API rendering command is detected, and this process is performed when the rendering command is submitted to the GPU for rendering, the real-time performance of vibration control is improved. This avoids the problem of untimely vibration control in solutions that rely on screenshots of the game interface and AI algorithms for vibration identification.

[0056] Please see Figure 3 , Figure 3 A flowchart illustrating a vibration control method according to another embodiment of this application is shown. This vibration control method is applied to the aforementioned mobile terminal, and will be discussed below. Figure 3 The process shown is described in detail. The vibration control method may specifically include the following steps:

[0057] Step S210: During game operation, in response to the specified type of graphics API instruction in the graphics API drawing instructions, determine the drawing data of the game interface to be drawn.

[0058] In this embodiment, during game execution, drawing instructions from the graphics API can be monitored to determine the drawing data for the game interface to be drawn when a specific type of instruction from the graphics API is detected. Understandably, rendering the game interface requires submitting numerous graphics API instructions to the GPU to complete the rendering process. However, some instructions are unrelated to the drawing data that needs to be determined, such as the instruction `GLbitfield mask` for clearing specified buffer data and the instruction `GLclampf red` for clearing color values. Therefore, when monitoring the graphics API drawing instructions, specific types of graphics API instructions can be extracted. In other words, unnecessary graphics API instructions are filtered out, and only instructions of the specified type are responded to.

[0059] In some implementations, a specified type of graphics drawing instruction may include a first graphics API instruction related to a data buffer and a second graphics API instruction related to a texture. The first graphics API instruction related to the buffer involves writing data used to draw the game interface into the buffer; therefore, by intercepting the first graphics API instruction, the data required for drawing can be obtained. The second graphics API instruction related to the texture involves texture information; therefore, by intercepting the second graphics API instruction, texture-related data can be obtained. Thus, by responding to both the first and second graphics API instructions, the required drawing data can be obtained.

[0060] In one possible implementation, the drawing data needed to draw the game interface may include vertex data and texture identification tags. Vertex data includes vertex position, vertex color, and texture coordinates; it may also include normal directions. Texture identification tags are used to determine the content of the texture. Optionally, the identification tag can be a Label or a Hash. Understandably, when the same texture object is created at different times, its corresponding object address or identity document (id) may be different, but the identification tag is the same. Therefore, the identification tag can be used to identify the content of the texture, and thus determine the content information of a specified area. For example, in a shooting game, if the identification tag of a texture object corresponding to a certain gun is Icon_WEP_SCAR_W, then the gun can be determined based on this identification tag. When a texture object does not have an identification tag, a content-generated Hash can be used as the identification tag to distinguish different texture objects.

[0061] The mobile terminal can respond to the first graphics API instruction to obtain vertex data for drawing the game interface, wherein the vertex data includes vertex position, vertex color and texture coordinates; the mobile terminal can also respond to the second graphics API instruction to obtain the identity tag corresponding to each texture for drawing the game interface.

[0062] Optionally, in response to a first graphics API instruction, the mobile terminal can obtain vertex data for drawing the game interface from the vertex buffer. Specifically, the mobile terminal can copy vertex data from the vertex buffer to a cache to obtain vertex data for drawing the game interface, so as to subsequently determine the content information of a specified area. Taking OpenGL as an example, the vertex buffer can be a VBO / VAO (Vertex Buffer Object or Vertex Array Object). Since the vertex buffer provides vertex data such as vertex position, vertex color, texture coordinates, and normal direction, the required vertex data can be obtained.

[0063] For example, a mobile terminal can obtain vertex data from a buffer containing vertex attributes and vertex indices. The graphics API commands for obtaining vertex attributes can be glBindBuffer(GL_ARRAY_BUFFER, VBO) and glBufferData(GL_ARRAY_BUFFER, sizeof(vertices), vertices, GL_STATIC_DRAW).

[0064] The graphics API commands for obtaining vertex indices can be glBindBuffer(GL_ELEMENT_ARRAY_BUFFER, EBO), glBufferData(GL_ELEMENT_ARRAY_BUFFER, sizeof(indices), indices, GL_STATIC_DRAW), etc. Of course, the above are just examples and do not represent a limitation on specific graphics API commands.

[0065] Optionally, when the mobile terminal responds to the second graphics API instruction to obtain the texture's identity tag, it can find the texture's identity tag through the texture ID and a mapping table. The second graphics API instruction may include glActiveTexture, glBindTexture, glDrawElements, etc. Of course, the above are just examples and do not represent a limitation on specific first graphics API instructions.

[0066] Step S220: Based on the drawing data, determine the content information of a specified area in the game interface.

[0067] Step S230: If the content information meets the vibration triggering conditions, control the mobile terminal to vibrate.

[0068] In the embodiments of this application, steps S220 and S230 can be referred to the contents of other embodiments, and will not be repeated here.

[0069] The vibration control method provided in this application obtains the required rendering data of the game interface to be rendered by responding to a specified type of graphics API instruction. Then, it determines the content information of a specified area using the rendering data, thereby controlling the vibration during gameplay. This avoids the high power consumption problem caused by using AI algorithms to identify vibration scenes, improves the battery life of mobile terminals, and avoids the cost of adding additional chips. Furthermore, by filtering graphics API instructions and responding only to specified types of graphics API instructions, the processor's processing load can be reduced, thus lowering the power consumption of the mobile terminal.

[0070] Please see Figure 4 , Figure 4 A flowchart illustrating a vibration control method according to another embodiment of this application is shown. This vibration control method is applied to the aforementioned mobile terminal, and will be discussed below. Figure 4 The process shown is described in detail. The vibration control method may specifically include the following steps:

[0071] Step S310: In response to a first graphics API instruction associated with the data buffer, obtain vertex data for drawing the game interface, the vertex data including vertex position, vertex color and texture coordinates.

[0072] Step S320: In response to a second graphics API instruction related to textures, obtain the identity tag corresponding to each texture used to draw the game interface.

[0073] In the embodiments of this application, steps S310 and S320 can be referred to the content of the foregoing embodiments, and will not be repeated here.

[0074] Step S330: In response to a third graphics API instruction related to drawing, obtain the vertex data and texture identity label corresponding to the currently drawn primitive based on the vertex data and identity label.

[0075] In this embodiment, a game interface typically contains multiple primitives. When rendering the game interface, these primitives may be divided into multiple drawing instructions, each responsible for drawing a portion of the primitives. When the CPU submits graphics API drawing instructions to the GPU, it can submit the relevant graphics API instructions for the currently drawn primitives. This allows for multiple submissions of drawing instructions to complete the rendering of all primitives, ultimately rendering and displaying the game interface. Therefore, the mobile terminal can also respond to third-party graphics API instructions related to drawing and, based on vertex data and identity tags, obtain the vertex data and texture identity tags corresponding to the currently drawn primitives. For example, in OpenGL, third-party graphics API instructions related to drawing could include `drawTriangles(Textures, Buffers)`, etc.

[0076] Step S340: Based on the vertex data corresponding to the currently drawn primitives, determine whether the content of the specified area is being drawn.

[0077] In this embodiment of the application, after the mobile terminal obtains the vertex data and identity tag corresponding to the currently drawn primitive, it can determine the position of the currently drawn primitive in the game interface based on the vertex position in the obtained vertex data, and then determine whether the currently drawn primitive is located in the specified area based on the determined position; if it is located in the specified area, it means that the content of the specified area is being drawn; if it is not located in the specified area, it means that the content of the specified area is not being drawn.

[0078] Step S350: If the content of the specified area is currently being drawn, the content information of the specified area is determined based on the vertex data corresponding to the currently drawn primitive and the texture identity label.

[0079] In this embodiment, if it is determined that content is currently being drawn in a specified area, the mobile terminal can determine the content information of the specified area based on the vertex data and texture identity tags corresponding to the currently drawn graphic, so as to determine whether vibration is required based on the content information of the specified area. If it is determined that content is not currently being drawn in a specified area, it means that the drawn content does not belong to the area associated with the vibration scene, so subsequent processing can be omitted.

[0080] Step S360: If the content information meets the vibration triggering conditions, control the mobile terminal to vibrate.

[0081] In this embodiment, step S360 can be found in other embodiments and will not be repeated here.

[0082] In some implementations, please refer to Figure 5The mobile terminal can run an instruction interception module 111, an instruction processing module 112, and a logic processing module 113. The instruction processing module 112 includes a first instruction processing unit 1121, a second instruction processing unit 1122, and a third instruction processing unit 1123. The instruction interception module 111 can intercept GPU graphics API instructions via hooking. After the intercepted instructions are filtered, the first graphics API instruction, the second graphics API instruction, and the third graphics API instruction are selected and sent to the instruction processing module 112. The first instruction processing unit 1121 in the instruction processing module 112 can respond to the first graphics API instruction to obtain vertex data used to draw the game interface. The second instruction processing unit 1122 in the instruction processing module 112 can respond to the second graphics API instruction to obtain the identity tag corresponding to each texture used to draw the game interface. The first instruction processing unit 1121 in the instruction processing module 112 can respond to the second graphics API instruction to obtain the identity tag corresponding to each texture used to draw the game interface. The three-instruction processing unit 1123 can respond to third graphics API instructions and obtain the vertex data and texture identity tags corresponding to the currently drawn primitives based on vertex data and identity tags. After processing the selected graphics API instructions, the instruction processing module 112 passes the processing result to the logic processing module 113. The logic processing module 113 determines the content information of the specified area based on the processing result of the instruction processing module 112, and then determines whether to trigger vibration. This achieves the determination of whether to trigger vibration based on the instructions and data during graphics drawing, avoiding the high power consumption problem caused by using AI algorithms to identify vibration scenes, and improving the battery life of mobile terminals. The specific processing procedures implemented in the instruction interception module 111, instruction processing module 112, and logic processing module 113 can be found in the above content and will not be repeated here.

[0083] The vibration control method provided in this application determines whether a specified area is being drawn based on the obtained vertex data when the game interface is completed by multiple drawing commands. If the specified area is being drawn, the vibration during the game can be controlled based on the content information. This avoids the high power consumption problem caused by using AI algorithms to identify vibration scenes, improves the battery life of mobile terminals, and avoids the cost of adding additional chips.

[0084] Please see Figure 6 , Figure 6 A flowchart illustrating a vibration control method according to another embodiment of this application is shown. This vibration control method is applied to the aforementioned mobile terminal, and will be discussed below. Figure 6 The process shown is described in detail. The vibration control method may specifically include the following steps:

[0085] Step S410: During game operation, in response to the drawing instructions of the graphics application programming interface (API), obtain the drawing data of the game interface to be drawn.

[0086] In this embodiment, step S310 can be referred to the content of the foregoing embodiments, and will not be repeated here.

[0087] Step S420: Based on the drawing data, obtain the texture corresponding to the specified area in the game interface as the target texture.

[0088] In this embodiment of the application, when determining the content information of a specified area based on the acquired drawing data, the texture corresponding to the specified area can be determined according to the acquired drawing data, so as to determine the content information of the specified area based on the texture of the specified area.

[0089] In some implementations, the mobile terminal can determine the primitives displayed in the game interface as a specified area based on the vertex positions in the rendering data; then, it can obtain the texture corresponding to the determined primitives as the target texture. Understandably, vertex positions represent the positions of primitives in the game interface, thus allowing the determination of primitives within a specified area, and the texture corresponding to the primitives is the texture corresponding to the specified area.

[0090] In other implementations, if the acquisition of drawing data is performed by determining whether the content of the specified area is being drawn in the previous embodiment, then the texture of the currently drawn primitive can be directly used as the texture of the specified area.

[0091] Step S430: Based on the target texture, determine the content information of a specified area in the game interface.

[0092] In this embodiment of the application, after determining the target texture corresponding to the specified area, since the texture is associated with the content in the game interface, the content information of the specified area in the game interface can be determined based on the target texture.

[0093] In one possible implementation, the acquired rendering data may include an identity tag corresponding to the texture, and the content information to be determined may include the content type of the rendered content in the specified area. The mobile terminal can acquire the identity tag corresponding to the target texture in the rendering data; then, based on the correspondence between the identity tag and the content type, it can determine the content type of the rendered content in the specified area. For example, in a shooting game, if the specified area is the area of ​​the weapon icon, and the identity tag of the texture object corresponding to a certain gun is Icon_WEP_SCAR_W, then the type of gun can be determined based on this identity tag.

[0094] In one possible implementation, the drawing data may include the texture coordinates of a texture, and the content information may include the drawing content of a specified area. The mobile terminal can obtain the texture coordinates of the target texture from the drawing data as the target texture coordinates; based on the correspondence between texture coordinates and drawing content, and the target texture coordinates, the drawing content corresponding to the target texture coordinates is determined. The correspondence between texture coordinates and drawing content can be pre-identified offline using a recognition tool (e.g., RenderDoc). For example, if the specified area is a text display area related to text such as remaining bullet count, kill count, or death count, the correspondence between text and texture coordinates can be identified offline. Understandably, texture coordinates reside in texture space. When a texture is applied to a primitive, its texture pixel address must be mapped to the object coordinate system and then translated to the screen coordinate system or pixel position. Each texture pixel in the texture can be declared using its coordinates; therefore, when the texture content changes, the texture coordinates also change. The texture coordinates obtainable from the drawing data can be used to determine the drawing content.

[0095] In one possible implementation, the acquired rendering data may include texture coordinates, and the content information to be acquired may include change information of the content in a specified area. The mobile terminal can acquire the texture coordinates corresponding to the texture marked in the rendering data; based on the texture coordinates corresponding to the target texture, it can determine the change information of the content in the specified area, including whether the content of the specified area has changed. Understandably, when the texture content changes, the texture coordinates also change; therefore, whether the content of the specified area has changed can be determined based on whether the texture coordinates have changed. Of course, the above change information may also include the magnitude of the change, for example, the magnitude of the change in texture content can be determined based on the magnitude of the change in texture coordinates, and thus the magnitude of the change in the content of the specified area can be determined.

[0096] It should be understood that the content information that needs to be determined may include one or more of the following: content type, drawn content, and change information. Of course, the specific content information that the mobile terminal needs to determine is not limited and can be set according to the vibration scenarios that need to be focused on in the actual game.

[0097] Step S440: If the content information meets the vibration triggering conditions, control the mobile terminal to vibrate.

[0098] In some implementations, when the content information includes the content type of drawn content in a specified area, it can be determined whether the content type is the specified content type, thereby determining whether the content information meets the vibration condition. When the content type is the specified content type, the mobile terminal can be controlled to vibrate. Specifically, if the content type is the specified content type, it can be determined that the content information in the specified area meets the vibration trigger condition; if the content type is not the specified content type, it can be determined that the content information in the specified area does not meet the vibration trigger condition. The specified content type can be determined based on the actual scenario requiring vibration.

[0099] For example, the game scenario could be a MOBA game; please refer to [link / reference]. Figure 7 The designated area can be kill notification area A1. Different kill notifications correspond to different notification content types. The content type in this case is the notification content type, which can be the notification content types corresponding to triple kill, quadra kill, penta kill, godlike, etc. For example, Figure 7 When the notification content type is "Pentakill", the mobile device's vibration can be controlled to improve the user's gaming experience.

[0100] For example, the game scenario could be an FPS (First-Person Shooter) game; see [link / reference]. Figure 8 The designated area can be area A2, where the gun icon is located. In this case, the content type is "gun type," and the specified content type can be a specific gun type. For example, when the gun type is a specific gun type, the mobile device's vibration can be controlled to achieve a vibration effect when using a specific gun, thus enhancing the user's gaming experience.

[0101] In some implementations, when the content information includes drawn content in a specified area, it can be determined whether the drawn content matches the specified content. If the drawn content matches the specified content, it is determined that the content information meets the vibration triggering condition; if the drawn content does not match the specified content, it is determined that the content information does not meet the vibration triggering condition. The specified content can be determined based on the actual scenario requiring vibration.

[0102] For example, the game scenario could be a MOBA game; please refer to [link / reference]. Figure 9 The designated area can be the kill count display area A3. The content drawn in this case is the kill count, and the designated content is the specified kill count. The specified count can be the number of times you want to control the vibration of the mobile device. For example, Figure 8 When the number of kills reaches a specified number "8", the mobile terminal can be controlled to vibrate, thus enhancing the user's gaming experience.

[0103] In some implementations, when the content information includes change information of the content in a specified area, it can be determined whether the content in the specified area has changed based on the change information. If the content in the specified area has changed, it is determined that the content information meets the vibration triggering condition, and the mobile terminal is controlled to vibrate. If the content in the specified area has not changed, it is determined that the content information does not meet the vibration triggering condition.

[0104] For example, the game scenario is an FPS (First-Person Shooter) game; please refer to [link / reference]. Figure 10 The designated area can be area A4, where the health bar is located. Please also refer to... Figure 10 and Figure 11 When the health bar A4 changes, it indicates that the character has been attacked, and therefore vibration can be generated, thus providing users with a more realistic gaming experience.

[0105] For example, the game scenario is an FPS (First-Person Shooter) game; please refer to [link / reference]. Figure 12 The designated area is area A5, which displays the remaining number of bullets. Please also refer to... Figure 12 and Figure 13 When the number of remaining bullets changes, it indicates that a shot has been fired, and therefore vibration can be generated, thus providing users with a realistic gaming experience.

[0106] Of course, the game scenarios and vibration scenes mentioned above are only examples to illustrate how to determine whether to vibrate based on content information.

[0107] In some implementations, when the mobile terminal determines that the content information meets the vibration triggering conditions and controls the mobile terminal to vibrate, it can also control at least one of the vibration intensity and frequency according to different content information. Specifically, the mobile terminal can control the vibration intensity of the vibration motor to control the vibration intensity during gameplay; the mobile terminal can also control the vibration frequency of the vibration motor to control the vibration frequency during gameplay.

[0108] In one possible implementation, when the content information includes drawn content, vibration intensities and / or frequencies can be applied based on the different drawn content. For example, if the designated area represents the number of kills, different vibration intensities can be applied for different designated kill counts. The vibration intensity can be proportional to the number of kills. For instance, if the designated kill counts include 8, 10, 15, and 20, the vibration intensity corresponding to kill counts of 8, 10, 15, and 20 increases sequentially, thereby providing the user with a more realistic gaming experience.

[0109] In one possible implementation, when the content information includes a content type, different vibration intensities and / or frequencies can be applied based on the different content types. For example, a designated area can be a kill notification area, and different types of kill notifications can correspond to different vibration intensities. For instance, if kill notifications include triple kills, quadruple kills, and pentakills, the vibration intensities corresponding to triple kills, quadruple kills, and pentakills can increase sequentially, thereby providing users with a more realistic gaming experience.

[0110] In one possible implementation, when the content information includes change information, and the change information includes whether the content of a specified area has changed and the degree of change, the mobile terminal can also vibrate with different vibration intensities and / or frequencies according to different degrees of change when it is determined that the content of the specified area has changed. For example, the specified area is the area where the health bar is located. Different changes in health can correspond to different vibration intensities. The vibration intensity can be proportional to the magnitude of the health change. For instance, in shooting games, when subjected to severe attacks, the vibration intensity is also greater, thereby providing the user with a more realistic gaming experience.

[0111] The above implementation methods can also be combined. That is, when the content information includes multiple pieces of information, they can be combined to control the vibration intensity and frequency. For example, the designated area includes a gun icon area, with the corresponding content information being the gun type, and the designated area also includes a bullet quantity display area, with the corresponding content information being the bullet quantity. In this case, when a change in the bullet quantity is detected, different vibration intensities can be applied according to different gun types. For example, the vibration intensity can be proportional to the damage capability of the gun type, so the greater the damage capability of the gun, the greater the vibration intensity, providing the user with a more realistic gaming experience.

[0112] The vibration control method provided in this application obtains the required rendering data of the game interface to be rendered by responding to a specified type of graphics API instruction. Then, based on the rendering data, it determines the target texture corresponding to a specified area. Next, based on the target texture, it determines the content type, rendering content, and change information to obtain content information. Finally, it controls the vibration during gameplay based on this content information. This avoids the high power consumption problem associated with using AI algorithms to identify vibration scenes, thus improving the battery life of mobile terminals and avoiding the cost of adding additional chips. Furthermore, it provides a way to control game vibration based on different content information, which can enhance the user's gaming experience.

[0113] Please see Figure 14This diagram illustrates a structural block diagram of a vibration control device 800 provided in an embodiment of this application. The vibration control device 800 utilizes the aforementioned mobile terminal and includes: a data acquisition module 810, a content acquisition module 820, and a vibration triggering module 830. Specifically, the data acquisition module 810, during game operation, responds to drawing instructions from a graphics application programming interface (API) to acquire drawing data of the game interface to be drawn; the content acquisition module 820, based on the drawing data, determines content information of a specified area within the game interface; and the vibration triggering module 830, if the content information meets vibration triggering conditions, controls the mobile terminal to vibrate.

[0114] In some implementations, the content acquisition module 820 includes a texture acquisition unit and an information determination unit. The texture acquisition unit is used to acquire the texture corresponding to a specified area in the game interface as a target texture based on the rendering data; the information determination unit is used to determine the content information of the specified area in the game interface based on the target texture.

[0115] In one possible implementation, the rendering data includes an identity tag corresponding to the texture, and the content information includes the content type of the rendering content in the specified area. The information determination unit can be used to: obtain the identity tag corresponding to the target texture in the rendering data; and determine the content type of the rendering content in the specified area based on the correspondence between the identity tag and the content type.

[0116] In one possible implementation, the drawing data includes texture coordinates of a texture, and the content information includes the drawing content of the specified area. The information determination unit can be used to: obtain the texture coordinates of the target texture in the drawing data as the target texture coordinates; and determine the drawing content corresponding to the target texture coordinates based on the correspondence between texture coordinates and drawing content, and the target texture coordinates.

[0117] In one possible implementation, the rendering data includes texture coordinates of a texture, and the content information includes change information of the content of the specified region. The information determination unit can be used to: obtain the texture coordinates corresponding to the target texture in the rendering data; and, based on the texture coordinates corresponding to the target texture, determine the change information of the content of the specified region, wherein the change information includes whether the content of the specified region has changed.

[0118] In one possible implementation, the drawing data includes vertex positions. The texture acquisition unit can be used to: determine the primitives displayed in the game interface as the specified area based on the vertex positions in the drawing data; and acquire the texture corresponding to the primitives as the target texture.

[0119] In some implementations, the data acquisition module 810 can be used to: determine the drawing data of the game interface to be drawn in response to a specified type of graphics API instruction in the drawing instructions of the graphics API.

[0120] In one possible implementation, the specified type of graphics drawing instructions includes a first graphics API instruction related to the data buffer and a second graphics API instruction related to textures. The drawing data includes vertex data and identity tags corresponding to the textures. The data acquisition module 810 may include a first data acquisition unit and a second data acquisition unit. The first data acquisition unit is used to acquire vertex data for drawing the game interface in response to the first graphics API instruction. The vertex data includes vertex position, vertex color, and texture coordinates. The second data acquisition unit is used to acquire identity tags corresponding to each texture for drawing the game interface in response to the second graphics API instruction.

[0121] In one possible implementation, the specified type of graphics drawing instruction further includes third graphics API instructions related to drawing. The data acquisition module 810 may also include a third data acquisition unit. The third data acquisition unit is used, in response to the third graphics API instruction, to acquire the vertex data and texture identity tags corresponding to the currently drawn primitives, based on the vertex data and identity tags.

[0122] In this implementation, the content acquisition module 820 can be used to: determine whether the content of the specified area is being drawn based on the vertex data corresponding to the currently drawn primitive; if the content of the specified area is being drawn, determine the content information of the specified area based on the vertex data corresponding to the currently drawn primitive and the texture identity tag.

[0123] In one possible implementation, the first data acquisition unit may be used to: in response to the first graphics API instruction, acquire vertex data from the vertex buffer for drawing the game interface.

[0124] In some implementations, the vibration triggering module 830 can be used to control the mobile terminal to vibrate if the content information changes relative to the previously specified area content information, or if the content information matches preset content information.

[0125] In some implementations, the vibration trigger module 830 can be used to control the mobile terminal to vibrate while displaying the game interface.

[0126] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working process of the above-described device and module can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0127] In the several embodiments provided in this application, the coupling between modules can be electrical, mechanical, or other forms of coupling.

[0128] Furthermore, the functional modules in the various embodiments of this application can be integrated into one processing module, or each module can exist physically separately, or two or more modules can be integrated into one module. The integrated modules described above can be implemented in hardware or as software functional modules.

[0129] In summary, the solution provided in this application obtains the drawing data of the game interface to be drawn by responding to the drawing instructions of the graphics API during game operation. Based on the drawing data, it determines the content information of a specified area in the game interface. If the content information meets the vibration triggering conditions, it controls the vibration of the mobile terminal. This allows the determination of the content information of a specified area based on the drawing data of the game interface when drawing the game interface to be displayed, thereby controlling the vibration during the game. This avoids the problem of high power consumption caused by using AI algorithms to identify vibration scenes and improves the battery life of the mobile terminal.

[0130] Please refer to Figure 15 This diagram illustrates a structural block diagram of a mobile terminal according to an embodiment of this application. The mobile terminal 100 can be an electronic device such as a smartphone or tablet computer capable of running applications. The mobile terminal 100 in this application may include one or more of the following components: a processor 110, a memory 120, a vibration motor 130, and one or more applications. The one or more applications may be stored in the memory 120 and configured to be executed by one or more processors 110. The one or more applications are configured to perform the methods described in the foregoing method embodiments.

[0131] Processor 110 may include one or more processing cores. Processor 110 connects to various parts within the mobile terminal 100 using various interfaces and lines, and performs various functions and processes data of the mobile terminal 100 by running or executing instructions, programs, code sets, or instruction sets stored in memory 120, and by calling data stored in memory 120. Optionally, processor 110 may be implemented using at least one hardware form of Digital Signal Processing (DSP), Field-Programmable Gate Array (FPGA), or Programmable Logic Array (PLA). Processor 110 may integrate one or a combination of several of the following: Central Processing Unit (CPU), Graphics Processing Unit (GPU), and modem. The CPU primarily handles the operating system, user interface, and applications; the GPU is responsible for rendering and drawing the displayed content; and the modem handles wireless communication. It is understood that the modem may also not be integrated into processor 110 and may be implemented separately using a communication chip.

[0132] The memory 120 may include random access memory (RAM) or read-only memory (ROM). The memory 120 can be used to store instructions, programs, code, code sets, or instruction sets. The memory 120 may include a program storage area and a data storage area. The program storage area may store instructions for implementing an operating system, instructions for implementing at least one function (such as touch functionality, sound playback functionality, image playback functionality, etc.), and instructions for implementing the various method embodiments described below. The data storage area may also store data created by the mobile terminal 100 during use (such as phonebook data, audio and video data, chat log data, etc.).

[0133] The vibration motor 130 is used by the mobile terminal to control the vibration motor 130 to operate when a scenario requiring vibration is determined, thereby generating vibration. For example, in the vibration control method provided in the embodiments of this application, the vibration motor 130 can be used to generate vibration during game operation.

[0134] Please refer to Figure 16 This diagram illustrates a structural block diagram of a computer-readable storage medium provided in an embodiment of this application. The computer-readable medium 800 stores program code that can be called by a processor to execute the methods described in the above method embodiments.

[0135] The computer-readable storage medium 800 may be an electronic memory such as flash memory, EEPROM (Electrically Erasable Programmable Read-Only Memory), EPROM, hard disk, or ROM. Optionally, the computer-readable storage medium 800 includes a non-transitory computer-readable storage medium. The computer-readable storage medium 800 has storage space for program code 810 that performs any of the method steps described above. This program code can be read from or written to one or more computer program products. The program code 810 may be compressed, for example, in a suitable form.

[0136] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application.

Claims

1. A vibration control method, characterized in that, Applied to a mobile terminal, the method includes: During game operation, in response to drawing instructions from the graphics application programming interface (API), the drawing data of the game interface to be drawn is obtained; Based on the drawing data, determine the content information of a specified area in the game interface; If the content information changes relative to the previously specified area, or if the content information matches preset content information, the mobile terminal will vibrate while the game interface is displayed.

2. The method according to claim 1, characterized in that, The step of determining the content information of a specified area in the game interface based on the drawing data includes: Based on the drawing data, the texture corresponding to a specified area in the game interface is obtained as the target texture; Based on the target texture, the content information of a specified area in the game interface is determined.

3. The method according to claim 2, characterized in that, The drawing data includes the identity tag corresponding to the texture, and the content information includes the content type of the drawn content in the specified area. Determining the content information of the specified area in the game interface based on the target texture includes: Obtain the identity tag corresponding to the target texture in the rendering data; Based on the correspondence between the identity tags and content types, the content type of the drawn content in the specified area is determined.

4. The method according to claim 2, characterized in that, The drawing data includes texture coordinates, the content information includes the drawing content of the specified area, and determining the content information of the specified area in the game interface based on the target texture includes: Obtain the texture coordinates of the target texture from the drawing data, and use them as the target texture coordinates; Based on the correspondence between texture coordinates and the content to be drawn, and the target texture coordinates, the content to be drawn corresponding to the target texture coordinates is determined.

5. The method according to claim 2, characterized in that, The drawing data includes texture coordinates, the content information includes content change information of the specified area, and determining the content information of the specified area in the game interface based on the target texture includes: Obtain the texture coordinates corresponding to the target texture in the drawing data; Based on the texture coordinates corresponding to the target texture, the change information of the content of the specified area is determined, including whether the content of the specified area has changed.

6. The method according to claim 2, characterized in that, The drawing data includes vertex positions, and the step of obtaining the texture corresponding to a specified area in the game interface as the target texture based on the drawing data includes: Based on the vertex positions in the drawing data, the primitives displayed in the game interface as the specified area are determined; Obtain the texture corresponding to the primitive and use it as the target texture.

7. The method according to claim 1, characterized in that, The process of obtaining drawing data for the game interface to be drawn in response to drawing instructions from the Graphics Application Programming Interface (API) includes: In response to a specified type of graphics API instruction in the graphics API drawing instructions, determine the drawing data for the game interface to be drawn.

8. The method according to claim 7, characterized in that, The specified type of graphics API instruction includes a first graphics API instruction related to data buffering and a second graphics API instruction related to textures. The drawing data includes vertex data and identity tags corresponding to textures. The method of determining the drawing data of the game interface to be drawn in response to the specified type of graphics API instruction in the graphics API drawing instructions includes: In response to the first graphics API instruction, vertex data for drawing the game interface is obtained, the vertex data including vertex position, vertex color and texture coordinates; In response to the second graphics API instruction, obtain the identity tag corresponding to each texture used to draw the game interface.

9. The method according to claim 8, characterized in that, The specified type of graphics drawing instruction also includes third graphics API instructions related to drawing. The step of determining the drawing data of the game interface to be drawn in response to the specified type of graphics API instruction in the graphics API drawing instruction further includes: In response to the third graphics API instruction, based on the vertex data and identity tags, obtain the vertex data and texture identity tags corresponding to the currently drawn primitive; The step of determining the content information of a specified area in the game interface based on the drawing data includes: Based on the vertex data corresponding to the currently drawn primitives, determine whether the content of the specified area is being drawn; If the content of the specified region is currently being drawn, the content information of the specified region is determined based on the vertex data corresponding to the currently drawn primitives and the texture's identity label.

10. The method according to claim 8, characterized in that, The step of obtaining vertex data for drawing the game interface in response to the first graphics API instruction includes: In response to the first graphics API instruction, vertex data for drawing the game interface is obtained from the vertex buffer.

11. A vibration control device, characterized in that, Applied to mobile terminals, the device includes: a data acquisition module, a content acquisition module, and a vibration triggering module, wherein... The data acquisition module is used to acquire the drawing data of the game interface to be drawn in response to the drawing instructions of the graphics application programming interface (API) during the game operation. The content acquisition module is used to determine the content information of a specified area in the game interface based on the drawing data; The vibration trigger module is used to control the mobile terminal to vibrate while displaying the game interface if the content information changes relative to the content information corresponding to the previously specified area, or if the content information matches the preset content information.

12. A mobile terminal, characterized in that, include: One or more processors; Memory; One or more applications, wherein the one or more applications are stored in the memory and configured to be executed by the one or more processors, the one or more applications being configured to perform the method as described in any one of claims 1-10.

13. A computer-readable storage medium, characterized in that, The computer-readable storage medium contains program code that can be invoked by a processor to execute the method as described in any one of claims 1-10.

Images