Data coding method and device, electronic equipment, storage medium and product

Abstract

The invention provides a data coding method and device, electronic equipment, a storage medium and a product. The method comprises the following steps: acquiring operation data of a user on a first target image in a current time period; identifying a target operation scene corresponding to the operation data from a plurality of operation scenes; based on the operation data and the target operation scene, predicting a to-be-operated area corresponding to the user from a second target image corresponding to the next time period; the target coding resource corresponding to the to-be-operated area is determined from the coding resource of the second target image, and the target coding resource is used for coding the to-be-operated area, so that the definition of the to-be-operated area is improved, and meanwhile, waste of coding resources of areas except the to-be-operated area is avoided.

Description

Technical Field

[0001] This disclosure relates to the field of data processing technology, and in particular to a data encoding method, apparatus, electronic device, storage medium, and product. Background Technology

[0002] In relevant data encoding schemes, encoding is usually based on the region of interest (ROI) acquired by the camera. However, the ROI acquired by the camera is only applicable to the current time period and cannot effectively predict changes in the ROI of the user's next action. This results in low clarity of the area of ​​interest that the user is interested in and a waste of encoding resources. Summary of the Invention

[0003] This disclosure provides a data encoding method, apparatus, electronic device, storage medium, and product to solve the problem in related technologies where changes in the region of interest for a user's next action cannot be effectively predicted, leading to a waste of encoding resources.

[0004] A first aspect of this disclosure provides a data encoding method, the method comprising: Obtain user operation data on the first target image within the current time period; Identify the target operation scenario corresponding to the operation data from multiple operation scenarios; Based on the operation data and the target operation scenario, the user's corresponding area to be operated is predicted from the second target image corresponding to the next time period; The target coding resources corresponding to the region to be operated on are determined from the coding resources of the second target image. The target coding resources are used to encode the region to be operated on.

[0005] A second aspect of this disclosure provides a data encoding apparatus, the apparatus comprising: The acquisition unit is used to acquire user operation data on the first target image within the current time period; The identification unit is used to identify the target operation scenario corresponding to the operation data from multiple operation scenarios; The prediction unit is used to predict the user's corresponding area to be operated from the second target image corresponding to the next time period based on the operation data and the target operation scenario. The encoding unit is used to determine the target encoding resource corresponding to the region to be operated on from the encoding resources of the second target image. The target encoding resource is used to encode the region to be operated on.

[0006] A third aspect of this disclosure provides an electronic device comprising: At least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor to enable the at least one processor to perform the methods described in the first aspect of this disclosure.

[0007] A fourth aspect of this disclosure provides a non-transitory computer-readable storage medium storing computer instructions, wherein the computer instructions are used to cause a computer to perform the methods described in the first aspect of this disclosure.

[0008] A fifth aspect of this disclosure provides a computer program product including a computer program that, when executed by a processor, implements the methods described in the first aspect of this disclosure.

[0009] In summary, this disclosure proposes a data encoding method, which includes: acquiring user operation data on a first target image within the current time period; identifying the target operation scenario corresponding to the operation data from multiple operation scenarios; predicting the user's corresponding operation area from a second target image corresponding to the next time period based on the operation data and the target operation scenario; and determining the target encoding resource corresponding to the operation area from the encoding resources of the second target image, wherein the target encoding resource is used to encode the operation area.

[0010] According to the solution provided in this disclosure, by analyzing the user's operation data on the first target image within the current time period, the target operation scenario is determined, and the area to be operated is dynamically adjusted based on the target operation scenario. Encoding resources are allocated to the area to be operated, which improves the clarity of the area to be operated and avoids the waste of encoding resources in areas outside the area to be operated.

[0011] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this disclosure. Attached Figure Description

[0012] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure, and are not intended to unduly limit this disclosure.

[0013] Figure 1 A flowchart illustrating the data encoding method provided in this embodiment of the disclosure; Figure 2 A flowchart illustrating a method for determining an area to be operated according to an embodiment of this disclosure; Figure 3 A flowchart illustrating the data encoding method provided in this embodiment of the disclosure; Figure 4A schematic diagram illustrating the data encoding method provided as an application example of this disclosure; Figure 5 This is a schematic diagram of the structure of the data encoding device provided in the embodiments of this disclosure; Figure 6 This is a schematic diagram of the hardware composition structure of an electronic device provided in an embodiment of this disclosure. Detailed Implementation

[0014] Embodiments of this disclosure are described in detail below. Examples of these embodiments are illustrated in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and intended to explain this disclosure, and should not be construed as limiting this disclosure.

[0015] Distributed coding is a technique for encoding and processing data in a distributed system. One of its key features is that the encoding process can be performed in parallel on multiple nodes.

[0016] In existing data encoding schemes, encoding resources are evenly distributed across the display screen area. This method assigns the same bitrate to every area of ​​the entire screen, failing to consider the differences in user attention levels and resulting in wasted encoding resources for areas that users are not interested in. Because it does not consider real-time user behavior and changes in areas of interest, the method of evenly dividing the screen cannot adapt to changes in dynamic user behavior, making it impossible to adjust the encoding strategy in a timely manner when users move their viewpoint or focus on different areas.

[0017] Furthermore, the data encoding schemes that rely on external cameras or other devices to acquire the user's visual region of interest (the area to be operated on) not only increase hardware costs but may also be limited by hardware constraints, resulting in insufficient flexibility. The region of interest acquired via a camera is only applicable to the current time period and cannot effectively predict the user's next action or changes in interest, showing insensitivity to dynamic changes in future field of vision.

[0018] To address the shortcomings of related technologies, this disclosure acquires user operation data on a first target image within the current time period; identifies the target operation scenario corresponding to the operation data from multiple operation scenarios; predicts the user's corresponding operation area from a second target image corresponding to the next time period based on the operation data and the target operation scenario; and determines the target encoding resource corresponding to the operation area from the encoding resources of the second target image. The target encoding resource is used to encode the operation area, thereby improving the clarity of the operation area and avoiding the waste of encoding resources in areas outside the operation area.

[0019] The present disclosure will now be described in further detail with reference to the accompanying drawings and specific embodiments.

[0020] The present disclosure provides a data encoding method that can be applied to virtual reality, augmented reality, security monitoring, and remote rendering of game scenes. The method can be executed by a cloud server or a terminal device.

[0021] like Figure 1 As shown, Figure 1 This is a flowchart illustrating the data encoding method provided in this embodiment of the disclosure. The data encoding method provided in this embodiment includes the following steps: Step 101: Obtain user operation data on the first target image within the current time period; In one embodiment, the first target image refers to the image frame that the user is currently viewing or operating within the current time period.

[0022] In one embodiment, the operation data can be hand operation data, arm operation data, or foot behavior data. This application uses hand operation data as an example. Hand operation data includes at least click location, movement direction, movement speed, gesture type, click frequency, and operation duration, used to analyze user intent. Specifically, click location records the coordinates of each click; movement direction records the direction of hand movement, typically represented by a vector; movement speed records the speed of hand movement; gesture type identifies the user's gesture actions, such as grasping, clicking, or swiping; click frequency counts the number of clicks made by the user within a certain time period; and operation duration records the duration of each gesture action.

[0023] In one embodiment, user operation data can be collected in real time using sensors (such as cameras) on the client side.

[0024] In one embodiment, when the current time period is relatively short, it can be the current moment, but this application does not limit this.

[0025] Step 102: Identify the target operation scenario corresponding to the operation data from multiple operation scenarios; In one embodiment, the multiple operating scenarios can be predefined or redefined.

[0026] In one embodiment, the target operation scenario is used to indicate the operation scenario corresponding to the user's operation data on the first target image within the current time period.

[0027] In one embodiment, the operation scenario refers to different operation states defined based on the user's behavior, such as static, interactive, moving, collision, exploration, fine operation, alert, and complex operation.

[0028] In one embodiment, a static scene refers to a situation where the user's hand is in a relatively static state, possibly viewing a fixed point or static content, or waiting for loading. The corresponding operation characteristics include: click position: fixed or unchanged; click frequency: low (<0.5 times / second); hand movement direction: no significant directional change (vector change less than 5 degrees / second); movement speed: low (<1cm / s); gesture type: no activity or simple hovering; operation duration: long (>3 seconds).

[0029] In one embodiment, the interaction scenario refers to a user's hand gestures with a clear intent, typically involving interaction with a virtual object, such as clicking a button or grasping a virtual object. The corresponding operation data includes: click position: close to the interactive object, with the click position changing within 10 cm; click frequency: medium (1-2 times / second); hand movement direction: changes within a small range, with the direction change amplitude <30 degrees; movement speed: medium (1-5 cm / s); gesture type: clear actions such as clicking or grasping; and operation duration: short to medium (<2 seconds).

[0030] In one embodiment, the motion scenario refers to the continuous movement of the user's hand, which may be for navigation or browsing a virtual environment. The hand movements indicate the intention to move, and the corresponding operation data includes: click position: no fixed position, large movement range (>20cm); click frequency: high (>2 times / second); hand movement direction: continuously changing, direction change frequency >1 time / second; movement speed: high (>5cm / s); gesture type: swipe, drag, etc.; operation duration: long (>3 seconds).

[0031] In one embodiment, a collision scenario refers to a user's hand suddenly interrupting an operation, with the action exhibiting characteristics of an abrupt stop or a sharp change in direction. This typically indicates that the user has encountered an accident in the virtual environment or needs to quickly adjust their actions. The corresponding operation data includes: click location: no fixed location, usually during movement; click frequency: variable, may suddenly decrease or increase; hand movement direction: sudden and sharp change, with a change in direction greater than 90 degrees; movement speed: sharp change (possibly from >5cm / s to 0); gesture type: sudden stop or sharp change in direction; operation duration: short to medium (<2 seconds).

[0032] In one embodiment, exploring a scene refers to a user's diverse hand movements and frequent changes in direction, indicating behavior of exploring new paths or finding new targets in a virtual environment. The corresponding operation data includes: click location: scattered, no specific area, click location variation range > 20 cm; click frequency: medium (1-2 times / second); hand movement direction: frequently changing, direction change frequency > 2 times / second; movement speed: medium (1-5 cm / s); gesture type: exploratory swiping, dragging, etc.; operation duration: medium (> 2 seconds).

[0033] In one embodiment, a fine-grained operation scenario refers to a user's concentrated and meticulous hand movements, typically involving the processing of intricate tasks or precise operations on small targets, such as adjusting virtual devices or performing detailed operations. The corresponding operation data includes: click location: fixed or within a small range, with a click location variation range of <5 cm; click frequency: high (>2 times / second); hand movement direction: minute changes, with a directional change amplitude of <10 degrees; movement speed: low (<1 cm / s); gesture type: fine adjustment, precise click; operation duration: long (>3 seconds).

[0034] In one embodiment, an alert scenario refers to a user's hand movements exhibiting rapid and sudden changes, possibly in response to an emergency in a virtual environment or to make a quick reaction. The corresponding operation data includes: click location: variable, rapidly changing, click location change range >15cm; click frequency: high (>2 times / second); hand movement direction: rapidly changing, direction change frequency >3 times / second; movement speed: high (>5 cm / s); gesture type: quick reaction, rapid swipe; operation duration: short (<1 second).

[0035] In one embodiment, a complex operation scenario refers to a user's hand displaying multiple complex operation types, possibly involving multi-step tasks or processing multiple objectives, demonstrating operational complexity. The corresponding operation data includes: click location: variable, scattered, click location variation range >10cm; click frequency: high (>2 times / second); hand movement direction: multi-directional changes, direction change frequency >2 times / second; movement speed: medium (1-5 cm / s); gesture type: multiple operations alternating; operation duration: medium (>2 seconds).

[0036] In one embodiment, a machine learning model (such as a classifier) ​​can be used to match the current operation data with an operation scenario template to determine the most suitable operation scenario. For example, if the user clicks frequently and moves their hand quickly, it may be identified as an "alert scenario".

[0037] Step 103: Based on the operation data and the target operation scenario, predict the user's corresponding area to be operated from the second target image corresponding to the next time period; In one embodiment, the second target image refers to an image frame that the user may focus on or manipulate in the next time period.

[0038] In one embodiment, the area to be operated refers to an image area that the user may be interested in or operate on in the next time period, based on the current operation data and the target operation scenario.

[0039] In one embodiment, a trained motion prediction model can be used to predict the user's behavior in the next time period by combining current operation data and the target operation scene. Based on the prediction results, the user's gaze direction or hand position in the next time period is calculated to determine the area to be operated. If the user is currently in a moving scene, the system predicts that the user may continue to move in a certain direction and accordingly expands the field of view (FOV) to cover that area. The motion prediction model can be an LSTM model, Transformer model, or other similar models.

[0040] Step 104: Determine the target coding resource corresponding to the region to be operated from the coding resource of the second target image. The target coding resource is used to encode the region to be operated.

[0041] In one embodiment, encoding resources refer to resources used for video encoding, which typically include bitrate, quantization parameters, computing power, etc.

[0042] In one embodiment, the target coding resource refers to a specific coding resource allocated to the region to be operated on in order to ensure that the region has high visual quality.

[0043] In one embodiment, the encoding strategy is dynamically adjusted according to the importance of the region to be operated on. For the region to be operated on, more encoding resources (such as higher bit rate and lower quantization parameter) are allocated to ensure high-quality encoding, while for non-operational regions, fewer encoding resources are allocated to reduce visual quality.

[0044] This disclosure obtains user operation data on a first target image within the current time period; identifies the target operation scenario corresponding to the operation data from multiple operation scenarios; predicts the user's corresponding operation area from a second target image corresponding to the next time period based on the operation data and the target operation scenario; and determines the target encoding resource corresponding to the operation area from the encoding resources of the second target image. The target encoding resource is used to encode the operation area, which improves the clarity of the operation area and avoids wasting encoding resources in areas outside the operation area.

[0045] In one embodiment, such as Figure 2 As shown, based on the operation data and the target operation scenario, the user's corresponding area to be operated is predicted from the second target image corresponding to the next time period, including: Step 201: Based on the operation data, determine the user's location information in the next time period; In one embodiment, the location information for the next time period refers to the spatial location of the user's operation target at a future moment (the next frame or the next second) or time period, which is the target point that the user's hand will operate on.

[0046] In one embodiment, user hand gesture data can be input into a trained temporal prediction model. The model outputs the predicted position of the user at the next moment. For example, if the user is quickly swiping to the right, the model predicts that their gaze will focus on a certain area on the right side of the screen after 0.2 seconds.

[0047] Step 202: Obtain the direction vectors of the image acquisition device and the user's position information in the next time period; In one embodiment, the image acquisition device may be a camera, a camera built into a virtual reality headset, or other visual sensors used to determine the user's gaze direction.

[0048] In one embodiment, the direction vector refers to a unit vector pointing from the image acquisition device to the user's spatial location, representing the user's line of sight or direction of attention.

[0049] Step 203: Based on the direction vector, determine the user's initial region of interest in the next time period; In one embodiment, the initial region of interest is a screen area that the user may be interested in, initially delineated based on the direction vector and field of view. The initial region of interest is typically a centrally symmetrical rectangular or trapezoidal area.

[0050] In one embodiment, the user's initial region of interest in the next time period can be determined based on the direction vector and the region indicated by the field of view.

[0051] Step 204: Based on the initial region of interest and the target operation scenario, predict the user's corresponding area to be operated from the second target image corresponding to the next time period.

[0052] In one embodiment, a new field of view can be determined based on the target operation scenario. Combined with the initial region of interest, the user's corresponding area to be operated can be predicted from the second target image corresponding to the next time period. This allows for the adjustment of the area to be operated, and the user's area of ​​interest at the next moment can be predicted in advance, avoiding image blurring and waste of encoding resources due to response delay.

[0053] In one embodiment, such as Figure 3 As shown, Figure 3 The flowchart of a data encoding method provided in this application embodiment includes step 301, acquiring user hand behavior data in real time; step 302, predicting the user's position at the next moment; step 303, analyzing the hand behavior data and identifying the user's current operation scenario; step 304, dividing the initial region of interest; and step 305, dynamically adjusting the region of interest according to the specific scenario and dynamically segmenting the video encoding region, wherein the user's position at the next moment is used to determine the initial region of interest.

[0054] In one embodiment, determining the user's location information for the next time period based on operational data includes: Input the operation data into the action prediction model to obtain the user's movement information in the next time period; In one embodiment, the action prediction model is a machine learning or deep learning model (such as LSTM, GRU, Transformer, etc.), which, after training, can predict future action trends based on historical behavior sequences. In this application, the input to the action prediction model is operational data, and the output of the action prediction model is movement information.

[0055] In one embodiment, motion information refers to the user's movement trend over a future period of time, including at least the direction of movement (such as left, right front, up, down, etc.) and the speed of movement.

[0056] Based on mobile information and the user's location information within the current time period, determine the user's location information for the next time period.

[0057] In one embodiment, if the user's position in the current time period is (x, y, z), the predicted movement direction is (dx, dy, dz), the predicted speed is v, and the time interval is Δt (the time interval between the current and next time periods), the corresponding predicted movement distance is distance = v × Δt. The mathematical expression for the unit direction vector is:

[0058] in, Let x be the direction vector along the x-axis. Let be the direction vector along the y-axis. Let z be the direction vector along the z-axis. It is a unit direction vector.

[0059] The mathematical expressions for determining the user's location information in the next time period are: New location_x = x + unit direction vector_x × distance, New location_y = y + unit direction vector_y × distance, New location_z = z + unit direction vector_z × distance. Here, "new location" refers to the user's location in the next time period, "unit direction vector_x", "unit direction vector_y", and "unit direction vector_z" refer to the aforementioned direction vectors along the x-axis, y-axis, and z-axis, respectively, and "distance" refers to the predicted movement distance.

[0060] In one embodiment, by determining the user's location information in the next time period based on mobility information and the user's location information in the current time period, the predicted location can be used to determine the area to be operated on in the image of the next time period, thereby dynamically allocating coding resources and reducing the waste of coding resources.

[0061] In one embodiment, determining the user's initial region of interest in the next time period based on the direction vector includes: Based on the direction vector, rotate towards the target direction according to the angle of the first field of view to obtain the target boundary vector, and determine the user's projection position in the display screen area in the next time period based on the projection matrix of the image acquisition device and the user's position information in the next time period. In one embodiment, the field of view (FAV) represents the range of angles that an observer (such as a camera or eye) can see in a specific direction. In three-dimensional space, the FAV typically describes the range of angles that an observer can see in both the horizontal and vertical directions. Based on the phenomenon that the sensitive area of ​​human vision is approximately 20 degrees, an initial FAV is determined to be 20 degrees, representing the field of view that the observer is primarily concerned with, used to initially divide the area into regions of interest and regions of non-interest. In this application, the angle of the first FAV is determined to be half of the initial FAV, i.e., fov_half = 10 degrees.

[0062] In one embodiment, the target direction typically includes up, down, left, and right. A rotation matrix is ​​used to rotate the direction vector by 10 degrees upwards, downwards, leftwards, and rightwards respectively, obtaining the target boundary vector. This target boundary vector includes an upper boundary vector, a lower boundary vector, a left boundary vector, and a right boundary vector. Extending a certain distance from the observer's position along these four boundary directions determines four boundary points in space. These four boundary points mark the range of the observer's field of view in three-dimensional space. The mathematical expressions for the left and right boundary vectors are as follows: L = rotate_vector(d, fov_half), R = rotate_vector(d, -fov_half). The rotate_vector function can be implemented using the rotation matrix of the image acquisition device to simulate the boundary perception of human vision. The mathematical expression for d is as follows:

[0063] C represents the camera position C=(Cx, Cy, Cz), T represents the target point, i.e. the user's position in the next time period T=(Tx, Ty, Tz), i.e. the aforementioned new position, and d represents the direction vector.

[0064] In one embodiment, the image acquisition device takes a camera as an example. The projection matrix of the image acquisition device refers to the intrinsic parameter matrix of the camera (including focal length, principal point, distortion, etc.), which is used to map three-dimensional spatial points to a two-dimensional image coordinate system (coordinates on the display screen).

[0065] In one embodiment, the display screen area refers to the screen area where the second target image is finally output, which is usually rectangular and used to define the boundaries of the initial region of interest and the region to be operated on.

[0066] Based on the target boundary vector and projection position, the user's initial region of interest in the next time period is determined.

[0067] In one embodiment, since the user sees a two-dimensional screen, it is necessary to convert these four three-dimensional spatial boundary points into their positions on the two-dimensional screen. The conversion process refers to taking a picture with a camera, "capturing" the three-dimensional world into a two-dimensional image. After conversion, the four points on the screen define a rectangular or trapezoidal area, which is the area of ​​the image that the user is currently interested in.

[0068] In one embodiment, the target boundary vector is obtained by rotating the target direction according to the first field of view angle based on the direction vector. Based on the projection matrix of the image acquisition device and the user's position information in the next time period, the projection position of the user in the display screen area in the next time period is determined. Based on the target boundary vector and the projection position, the initial region of interest of the user in the next time period is determined. The generated initial region of interest can be used as the basis for subsequent adjustment of the field of view angle according to the operation scenario, so as to achieve flexible control.

[0069] In one embodiment, based on an initial region of interest and a target operation scenario, predicting the user's target area from a second target image corresponding to the next time period includes: Based on the target operation scenario, the second field of view is determined from the mapping relationship between the operation scenario and the field of view. In one embodiment, the mapping relationship between the operating scene and the field of view can be represented in the form of a mapping relationship table or in the form of a mapping relationship array. This application does not limit this.

[0070] In one embodiment, a second field of view is used to replace the initial field of view, thereby achieving a region of interest adjustment that better meets the user's needs.

[0071] The initial region of interest is updated based on the second field of view, and the user's area to be operated on in the next time period is determined from the second target image corresponding to the next time period.

[0072] In one embodiment, if the target operation scenario is a static scenario, the field of view (FOV) is maintained at 20 degrees, focusing on the static content or fixed point currently viewed by the user. If the target operation scenario is an interactive scenario, the FOV is adjusted to 40°, corresponding to a rotation angle of half 40 degrees, i.e., 20 degrees, ensuring that the interactive object and its surrounding environment are within the field of view. If the target operation scenario is a moving scenario, the FOV is expanded to 60°, corresponding to a rotation angle of half 60 degrees, i.e., 30 degrees, covering the area the user may move to, supporting navigation and environmental browsing. If the target operation scenario is a collision scenario, the FOV is expanded to 50°, increasing attention to the surrounding environment to support the user's rapid reaction and adjustment. If the target operation scenario is an exploration scenario, the FOV is expanded to 80°, covering a wide field of view, supporting the user's free exploration and multi-path selection. If the target operation scenario is a fine-grained operation scenario, the FOV is maintained at 20°, focusing on the specific location and detailed operations of fine control. If the target operation scenario is an alert scenario, the FOV is rapidly expanded to 90°, providing a wide field of view to support the user's rapid decision-making and action in emergency situations. If the target operation scenario is complex, adjust the FOV to 70° to support the effective management and execution of multiple operation objectives or steps.

[0073] In one embodiment, by updating the initial region of interest based on the second field of view, the user's area to be operated on in the next time period can be determined from the second target image corresponding to the next time period. This eliminates the need to use a fixed field of view and allows the focus range to be automatically adjusted according to the user's behavioral intention, thereby improving the system's intelligence level.

[0074] In one embodiment, a target coding resource corresponding to the region to be operated on is determined from the coding resources of the second target image. The target coding resource is used to encode the region to be operated on, including: The coding resources of the second target image are divided into target coding resources and peripheral coding resources; In one embodiment, peripheral coding resources refer to coding resources allocated to regions other than the region to be operated on in the second target image.

[0075] In the process of encoding the second target image, the target coding resource is used to encode the region to be operated on, and the peripheral coding resource is used to encode other regions in the second target image besides the region to be operated on.

[0076] In one embodiment, the encoding region of the second target image corresponding to the video is dynamically segmented, the bitrate of the region to be operated on, i.e. the region of interest to the user, is increased, while the region of no interest to the user, i.e. other regions, is appropriately degraded, and then distributed to different encoding nodes in the distributed system for processing.

[0077] In one embodiment, by using target coding resources to encode the region to be operated on and using peripheral coding resources to encode other regions in the second target image besides the region to be operated on during the encoding process of the second target image, the waste of coding resources for non-interested regions is avoided, and the coding resources are concentrated on the regions of interest to the user.

[0078] In one embodiment, in a game scenario, the user needs to evade pursuing monsters in a maze. The game optimizes the user experience by dynamically adjusting the field of view. This includes real-time acquisition of user hand behavior data. Specifically, the user's hand operation data is collected from the client in real time, including: click location: (2.0, 1.8, 0.5), click frequency: 120 times / minute, indicating high-frequency operation; hand movement direction: (0.5, -0.5, 0.1), indicating frequent changes in hand direction during escape; movement speed: 10 cm / s, indicating an emergency state; gesture type: frequent waving and sudden stops, indicating emergency operations; operation duration: each operation is short, only 0.2 seconds. Next-moment position prediction includes: Based on the current user's hand data, the system predicts the next position. If the current hand position is (x, y, z) = (2.0, 1.8, 0.5), the direction of movement is (dx, dy, dz) = (0.5, -0.5, 0.1), the speed is 10 cm / s, and the time interval is 1 second. Calculate the movement distance: distance = v × Δt = 10 × 1 = 10 cm. Calculate the unit direction vector: unit direction vector ≈ (0.707, -0.707, 0.141). Calculate the new position: new position x = 2.0 + 0.707 × 10 ≈ 9.07, new position y = 1.8 - 0.707 × 10 ≈ 11.93, new position z = 0.5 + 0.141 × 10 ≈ 1.64, resulting in the user's next position being (9.07, 11.93, 1.64). Analyzing hand data identifies the user's current operational scenario, such as: Click location: rapid changes in location indicate frequent adjustments by the user, consistent with rapid changes within a range exceeding 15 centimeters. Click frequency: 2 times per second, consistent with high-frequency clicks, indicating high-frequency interaction in an emergency. Hand movement direction: frequent changes in direction, meeting the requirement of a frequency exceeding 3 times per second, showing the user rapidly adjusting direction to respond to changes. Movement speed: 10 cm / s, higher than the 5 cm / s required for an alert scenario, indicating a rapid reaction. Gesture type: frequent waving and sudden stops, consistent with rapid reaction and rapid swiping, indicating high-intensity operations in an emergency. Operation duration: 0.2 seconds, a short duration, meeting the requirement of less than 1 second in an alert scenario, showing a rapid reaction. Based on hand behavior characteristics, the system identifies the operational scenario as an alert scenario. Initial division of the region of interest: The initial field of view is set to 20 degrees, suitable for players to concentrate on observing the area in front. The system divides the region of interest as follows: Direction vector from camera to target point: Camera position: (0,0,0), Target point: (2.0,1.8,0.5), Direction vector: (2.0,1.8,0.5), Direction vector calculation: Direction vector = (2.0,1.8,0.5). Calculation of half of the field of view: Half of the field of view = 10.Calculate the boundary vector: Rotate the direction vector by 10 degrees using a rotation matrix to obtain the target boundary vector. Calculate the projection area boundary: Project the target point onto the screen using the camera's projection matrix to obtain the projection position. Determine the initial region of interest (FOV): Using the projection position and boundary vector, determine the four boundary points of the projection area, where the left boundary vector = rotation(direction vector, -10), and the right boundary vector = rotation(direction vector, 10). Finally, dynamically divide the FOV according to the specific scene. In alert scenarios, the FOV needs to be significantly expanded to ensure that players can quickly identify threats and paths in the surrounding environment. The current FOV is 20, and the adjusted FOV is 90. The adjusted FOV expands from 20 degrees to 90 degrees, covering a wide field of view. The field of view boundary is recalculated using the expanded FOV to ensure that the entire field of view is visible to the player. The system allocates video encoding resources based on the dynamically adjusted FOV. For the area to be operated on, i.e., the area covered by the expanded FOV, the bitrate is increased to improve video clarity so that players can perceive and react faster. For areas of non-interest, i.e., areas outside the area to be operated on, the quality is degraded to reduce encoding resource consumption.

[0079] In summary, the solution provided in this public disclosure is as follows: First, user operation data on the first target image within the current time period is acquired; the target operation scenario corresponding to the operation data is identified from multiple operation scenarios; based on the operation data and the target operation scenario, the user's corresponding operation area is predicted from the second target image corresponding to the next time period; the target encoding resource corresponding to the operation area is determined from the encoding resource of the second target image. The target encoding resource is used to encode the operation area, which improves the clarity of the operation area and avoids wasting encoding resources in areas outside the operation area.

[0080] Secondly, the size and direction of the field of view are dynamically adjusted according to different operating scenarios (such as static, interactive, and moving scenarios). For example, in static scenarios, the field of view is narrowed to focus on displaying local details; in moving or alert scenarios, the field of view is widened to cover a larger field of view.

[0081] Furthermore, in addition to considering the current user's operation data, a behavior prediction model is used to predict the user's operation location and scenario in the next time period. High bitrate allocation is also applied to the predicted future areas, making video encoding more proactive, improving response efficiency and video quality in real-world scenarios, and ensuring users receive a consistently high-quality visual experience even in rapidly changing environments.

[0082] Furthermore, by acquiring real-time user hand gesture data and behavioral characteristics, the system dynamically adjusts its encoding strategy based on algorithms, eliminating the need for pre-defined complex rule sets. The system can optimize the allocation of regions of interest and encoding resources in real-time under different operational scenarios (such as static, interactive, and dynamic states), improving its flexibility and adaptability, and enabling the encoding strategy to dynamically match the user's actual needs.

[0083] The following application example further illustrates the data encoding method provided in this disclosure: like Figure 4 As shown, Figure 4 This is a flowchart illustrating the data encoding method provided in this application example. The data encoding method provided in this application example includes the following steps: Step 401: Obtain user operation data on the first target image within the current time period; Step 402: Identify the target operation scenario corresponding to the operation data from multiple operation scenarios; Step 403: Input the operation data into the action prediction model to obtain the user's movement information in the next time period; Step 404: Based on the mobile information and the user's location information in the current time period, determine the user's location information in the next time period; Step 405: Obtain the direction vectors of the image acquisition device and the user's position information in the next time period; Step 406: Based on the direction vector, rotate towards the target direction according to the angle of the first field of view to obtain the target boundary vector, and determine the user's projection position in the display screen area in the next time period based on the projection matrix of the image acquisition device and the user's position information in the next time period. Step 407: Based on the target boundary vector and projection position, determine the user's initial region of interest in the next time period; Step 408: Based on the target operation scene, determine the second field of view from the mapping relationship between the operation scene and the field of view. Step 409: Update the initial region of interest based on the second field of view, and determine the user's area to be operated on in the next time period from the second target image corresponding to the next time period; Step 410: Divide the coding resources of the second target image into target coding resources and peripheral coding resources; Step 411: In the process of encoding the second target image, the target encoding resource is used to encode the region to be operated on, and the peripheral encoding resource is used to encode other regions in the second target image other than the region to be operated on.

[0084] To implement the data encoding method provided in this disclosure, this disclosure also provides a data encoding apparatus, such as... Figure 5As shown. Figure 5 This is a schematic diagram of the structure of a data encoding device provided in an embodiment of the present disclosure. The data encoding device 500 includes: The acquisition unit 501 is used to acquire user operation data on the first target image within the current time period; The identification unit 502 is used to identify the target operation scenario corresponding to the operation data from multiple operation scenarios; The prediction unit 503 is used to predict the user's corresponding area to be operated from the second target image corresponding to the next time period based on the operation data and the target operation scenario. The encoding unit 504 is used to determine the target encoding resource corresponding to the region to be operated from the encoding resource of the second target image. The target encoding resource is used to encode the region to be operated.

[0085] In one embodiment, the prediction unit 503 is specifically used for: Based on the operational data, determine the user's location information in the next time period; Obtain the direction vectors of the image acquisition device and the user's position information in the next time period; Based on the direction vector, determine the user's initial region of interest in the next time period; Based on the initial region of interest and the target operation scenario, the user's corresponding area to be operated is predicted from the second target image corresponding to the next time period.

[0086] In one embodiment, the prediction unit 503 is specifically used for: Input the operation data into the action prediction model to obtain the user's movement information in the next time period; Based on mobile information and the user's location information within the current time period, determine the user's location information for the next time period.

[0087] In one embodiment, the prediction unit 503 is specifically used for: Based on the direction vector, rotate towards the target direction according to the angle of the first field of view to obtain the target boundary vector, and determine the user's projection position in the display screen area in the next time period based on the projection matrix of the image acquisition device and the user's position information in the next time period. Based on the target boundary vector and projection position, the user's initial region of interest in the next time period is determined.

[0088] In one embodiment, the prediction unit 503 is specifically used for: Based on the target operation scenario, the second field of view is determined from the mapping relationship between the operation scenario and the field of view. The initial region of interest is updated based on the second field of view, and the user's area to be operated on in the next time period is determined from the second target image corresponding to the next time period.

[0089] In one embodiment, the encoding unit 504 is specifically used for: The coding resources of the second target image are divided into target coding resources and peripheral coding resources; In the process of encoding the second target image, the target coding resource is used to encode the region to be operated on, and the peripheral coding resource is used to encode other regions in the second target image besides the region to be operated on.

[0090] This application obtains user operation data on a first target image within the current time period; identifies the target operation scenario corresponding to the operation data from multiple operation scenarios; predicts the user's corresponding operation area from a second target image corresponding to the next time period based on the operation data and the target operation scenario; and determines the target encoding resource corresponding to the operation area from the encoding resources of the second target image. The target encoding resource is used to encode the operation area, improving the clarity of the operation area and avoiding the waste of encoding resources in areas outside the operation area.

[0091] It should be noted that the data encoding device provided in the above embodiments is only illustrated by the division of the above program modules. In practical applications, the above processing can be assigned to different program modules as needed, that is, the internal structure of the data encoding device can be divided into different program modules to complete all or part of the processing described above. In addition, the data encoding device provided in the above embodiments and the data encoding method embodiments provided in this disclosure belong to the same concept, and the specific implementation process can be found in the method embodiments, which will not be repeated here.

[0092] Figure 6 This is a schematic diagram of the hardware composition structure of the electronic device provided in the embodiments of this disclosure, such as... Figure 6 As shown, the electronic device 600 includes at least one processor 602; and a memory 601 communicatively connected to the at least one processor 602; wherein the memory 601 stores instructions executable by the at least one processor 602, the instructions being executed by the at least one processor 602 to implement the steps of the data encoding method of the embodiments of this disclosure.

[0093] Optionally, the electronic device may specifically be a data encoding device in the embodiments of this application, and the electronic device may implement the corresponding processes implemented by the data encoding device in the various methods of the embodiments of this application. For the sake of brevity, it will not be described in detail here.

[0094] It is understood that the electronic device also includes a communication interface 603. Various components in the electronic device are coupled together via a bus system 604. It is understood that the bus system 604 is used to implement communication between these components. In addition to a data bus, the bus system 604 also includes a power bus, a control bus, and a status signal bus. However, for clarity, in... Figure 6 The general designated all buses as Bus System 604.

[0095] It is understood that memory 601 can be volatile memory or non-volatile memory, or both. Non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), ferromagnetic random access memory (FRAM), flash memory, magnetic surface memory, optical disc, or compact disc read-only memory (CD-ROM); magnetic surface memory can be disk storage or magnetic tape storage. Volatile memory can be random access memory (RAM), which is used as an external cache. By way of example, but not limitation, many forms of RAM are available, such as Static Random Access Memory (SRAM), Synchronous Static Random Access Memory (SSRAM), Dynamic Random Access Memory (DRAM), Synchronous Dynamic Random Access Memory (SDRAM), Double Data Rate Synchronous Dynamic Random Access Memory (DDRSDRAM), Enhanced Synchronous Dynamic Random Access Memory (ESDRAM), SyncLink Dynamic Random Access Memory (SLDRAM), and Direct Rambus Random Access Memory (DRRAM).The memory 601 described in this embodiment of the invention is intended to include, but is not limited to, these and any other suitable types of memory.

[0096] The methods disclosed in the above embodiments can be applied to or implemented by processor 602. Processor 602 may be an integrated circuit chip with signal processing capabilities. During implementation, each step of the above methods can be completed by integrated logic circuits in the hardware of processor 602 or by instructions in software form. Processor 602 may be a general-purpose processor, DSP, or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. Processor 602 can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of this invention. A general-purpose processor may be a microprocessor or any conventional processor, etc. The steps of the methods disclosed in the embodiments of this invention can be directly manifested as execution by a hardware decoding processor, or execution by a combination of hardware and software modules in the decoding processor. The software modules may be located in a storage medium, specifically memory 601. Processor 602 reads information from memory 601 and, in conjunction with its hardware, completes the steps of the aforementioned methods.

[0097] In an exemplary embodiment, the electronic device may be implemented by one or more application-specific integrated circuits (ASICs), DSPs, programmable logic devices (PLDs), complex programmable logic devices (CPLDs), FPGAs, general-purpose processors, controllers, MCUs, microprocessors, or other electronic components to perform the aforementioned method.

[0098] This disclosure also provides a non-transitory computer-readable storage medium storing computer instructions, which are used to cause a computer to execute the steps of the data encoding method of the present invention.

[0099] Optionally, the computer-readable storage medium can be applied to the data encoding device in the embodiments of this application, and the computer instructions cause the computer to execute the corresponding processes implemented by the data encoding device in the various methods of the embodiments of this application. For the sake of brevity, these will not be described in detail here.

[0100] This disclosure also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the data encoding method provided in this embodiment of the invention.

[0101] In the several embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. The device embodiments described above are merely illustrative. For example, the division of units is only a logical functional division, and in actual implementation, there may be other division methods, such as: multiple units or components can be combined, or integrated into another system, or some features can be ignored or not executed. In addition, the coupling, direct coupling, or communication connection between the various components shown or discussed can be through some interfaces, and the indirect coupling or communication connection between devices or units can be electrical, mechanical, or other forms.

[0102] The units described above as separate components may or may not be physically separate. The components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the units may be selected to achieve the purpose of this embodiment according to actual needs.

[0103] In addition, in the various embodiments of the present invention, each functional unit can be integrated into one processing unit, or each unit can be a separate unit, or two or more units can be integrated into one unit; the integrated unit can be implemented in hardware or in the form of hardware plus software functional units.

[0104] Those skilled in the art will understand that all or part of the steps of the above method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it performs the steps of the above method embodiments. The aforementioned storage medium includes various media that can store program code, such as mobile storage devices, ROM, RAM, magnetic disks, or optical disks.

[0105] Alternatively, if the integrated units of this invention are implemented as software functional modules and sold or used as independent products, they can also be stored in a computer-readable storage medium. Based on this understanding, the technical solutions of the embodiments of this invention, or the parts that contribute to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the methods of the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as mobile storage devices, ROM, RAM, magnetic disks, or optical disks.

[0106] The above are merely specific embodiments of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A data encoding method characterized by, The method comprises the following steps: obtaining operation data of a user on a first target image in a current time period; identifying a target operation scene corresponding to the operation data from a plurality of operation scenes; based on the operation data and the target operation scene, predicting a to-be-operated region of the user from a second target image corresponding to a next time period; determining a target encoding resource corresponding to the to-be-operated region from an encoding resource of the second target image, the target encoding resource being used for encoding the to-be-operated region.

2. The method of claim 1, wherein, The method comprises the following steps: based on the operation data, determining position information of the user in the next time period; obtaining a direction vector of an image acquisition device and the position information of the user in the next time period; based on the direction vector, determining an initial region of interest of the user in the next time period; based on the initial region of interest and the target operation scene, predicting the to-be-operated region of the user from the second target image corresponding to the next time period.

3. The method of claim 2, wherein, The method comprises the following steps: inputting the operation data into an action prediction model to obtain movement information of the user in the next time period; based on the movement information and position information of the user in the current time period, determining the position information of the user in the next time period.

4. The method of claim 2, wherein, The method comprises the following steps: based on the direction vector, rotating by an angle of a first field of view to a target direction to obtain a target boundary vector, and based on a projection matrix of the image acquisition device and the position information of the user in the next time period, determining a projection position of the user in a display screen region in the next time period; based on the target boundary vector and the projection position, determining the initial region of interest of the user in the next time period.

5. The method of claim 2, wherein, The method comprises the following steps: based on the target operation scene, determining a second field of view from a mapping relationship between operation scenes and fields of view; based on the second field of view, updating the initial region of interest to determine the to-be-operated region of the user in the next time period from the second target image corresponding to the next time period.

6. The method of claim 1, wherein, The method comprises the following steps: dividing the encoding resource of the second target image into the target encoding resource and a peripheral encoding resource; in the process of encoding the second target image, using the target encoding resource to encode the to-be-operated region, and using the peripheral encoding resource to encode other regions of the second target image except the to-be-operated region.

7. A data encoding device, characterized by comprising: The method comprises the following steps: an obtaining unit is configured to obtain operation data of a user on a first target image in a current time period; An identifying unit is configured to identify a target operation scenario corresponding to the operation data from a plurality of operation scenarios; A predicting unit is configured to predict a to-be-operated region corresponding to the user from a second target image corresponding to a next time period based on the operation data and the target operation scenario; An encoding unit is configured to determine a target encoding resource corresponding to the to-be-operated region from encoding resources of the second target image, and the target encoding resource is used for encoding the to-be-operated region.

8. An electronic device, comprising: Comprise: At least one processor; And The memory is in communication connection with the at least one processor; wherein, The memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method in any one of claims 1 to 6.

9. A non-transitory computer-readable storage medium having stored thereon computer instructions, wherein, The computer instructions are used to enable the computer to perform the method in any one of claims 1 to 6.

10. A computer program product comprising a computer program which, when executed by a processor, implements the method of any one of claims 1 to 6.

Images