Image frame processing method and related device

Abstract

The present disclosure provides an image frame processing method, comprising: receiving an image frame to be processed sent by a caller; setting and starting a waiting timer based on a pre-set waiting time; in response to determining that the waiting timer is timed out, performing image processing on the image frame to be processed to obtain a first image processing result, and calling back the first image processing result to the caller; or in response to determining that a second image processing result is received from a first processing unit before the waiting timer is timed out, terminating the waiting timer. Based on the above image frame processing method, the present disclosure further provides an image frame processing device, a storage medium and a program product.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to the technical field of computer vision, and particularly relates to an image frame processing method, an image frame processing apparatus, a storage medium and a program product. BACKGROUND

[0002] Extended Reality (XR) is a collective term for Augmented Reality (AR), Virtual Reality (VR), Mixed Reality (MR) and other technologies. XR can bring the "immersion" of seamless conversion between the virtual world and the real world to the experimenter by fusing the visual interaction technologies of the above three. At present, the visual interaction algorithms involved in XR usually include 6DoF tracking, gesture recognition, Eye Tracking (ET, also known as gaze tracking) and face tracking, etc.

[0003] The upper application of XR can guarantee the rendering frame rate through asynchronous execution and heterogeneous computing, etc. However, due to the preemption of different visual interaction algorithms to the heterogeneous processing units for executing the algorithms, and the processing units can only process the image frames when they are idle, the processing time of part of the image frames is too long. In the case that the processing units cannot return the image processing results in time, the rendering results of the upper application appear "jumping", which greatly affects the user experience. SUMMARY

[0004] Therefore, the embodiments of the present disclosure provide an image frame processing method, which can effectively solve the jumping problem caused by frame loss.

[0005] The image frame processing method provided by the embodiments of the present disclosure can include: receiving an image frame to be processed sent by a caller; setting and starting a waiting timer based on a pre-set waiting time; in response to determining that the waiting timer is timed out, performing image processing on the image frame to be processed to obtain a first image processing result, and calling back the first image processing result to the caller; or in response to determining that a second image processing result is received from a first processing unit before the waiting timer is timed out, terminating the waiting timer.

[0006] In the embodiments of the present disclosure, the image processing includes gaze tracking algorithm processing.

[0007] In embodiments of the present disclosure, the image processing on the to-be-processed image frame comprises: obtaining a cached third image processing result; wherein the third image processing result comprises: a pupil center of a previous image frame, an eye rotation center of the previous image frame, and a line-of-sight direction of the previous image frame; selecting a pupil region from the to-be-processed image frame based on the pupil center of the previous image frame; inputting the to-be-processed image frame and the selected pupil region into an inference engine to obtain an eye segmentation result of the to-be-processed image frame and a pupil center of the to-be-processed image frame; inputting a camera parameter, the third image processing result, the eye segmentation result of the to-be-processed image frame, and the pupil center of the to-be-processed image frame into a solver to obtain a line-of-sight landing point of the to-be-processed image frame; and determining the eye rotation center and the line-of-sight direction of the to-be-processed image frame based on the line-of-sight landing point of the to-be-processed image frame.

[0008] In embodiments of the present disclosure, the image processing comprises one or a combination of a 6DoF tracking algorithm processing, a gesture recognition algorithm processing, and a face tracking algorithm processing.

[0009] In embodiments of the present disclosure, the image frame processing method can further comprise: in response to determining that no second image processing result is received from the first processing unit, caching the first image processing result; or in response to determining that a second image processing result is received from the first processing unit, caching the second image processing result.

[0010] In embodiments of the present disclosure, the waiting timer is set to have a timing duration of the pre-set waiting time.

[0011] Embodiments of the present disclosure also provide an image frame processing method, comprising:

[0012] The invoker sends a to-be-processed image frame to a first processing unit and a second processing unit respectively;

[0013] After receiving the to-be-processed image frame, the first processing unit performs image processing on the to-be-processed image frame to obtain a second image processing result, and calls back the second image processing result to the invoker; and

[0014] The second processing unit sets and starts a waiting timer based on a pre-set waiting time; in response to determining that the waiting timer is timed out, performs image processing on the to-be-processed image frame to obtain a first image processing result, and calls back the first image processing result to the invoker; or in response to determining that the second image processing result is received from the first processing unit before the waiting timer is timed out, terminates the waiting timer.

[0015] Based on the image frame processing method, the embodiment of the present disclosure further provides an image frame processing device, comprising:

[0016] An image frame receiving module is configured to receive an image frame to be processed sent by a caller;

[0017] A timing control module is configured to set and start a waiting timer based on a pre-set waiting time;

[0018] An image processing module is configured to perform image processing on the image frame to be processed to obtain a first image processing result in response to determining that the waiting timer is timed out; or configured to terminate the waiting timer in response to determining that a second image processing result is received from a first processing unit before the waiting timer is timed out; and

[0019] A callback module is configured to call back the first image processing result to the caller.

[0020] In the embodiment of the present disclosure, the image frame processing device can further comprise an image processing result caching module configured to cache the first image processing result or the second image processing result.

[0021] In the embodiment of the present disclosure, the image processing module comprises:

[0022] An image processing result obtaining unit is configured to obtain a third image processing result cached; wherein the third image processing result comprises a pupil center of a previous image frame, an eye rotation center of the previous image frame and a line of sight direction of the previous image frame;

[0023] A preprocessing unit is configured to select a pupil region from the image frame to be processed based on the pupil center of the previous image frame;

[0024] An inference engine is configured to obtain an eye segmentation result of the image frame to be processed and a pupil center based on the image frame to be processed and the selected pupil region;

[0025] A solver is configured to perform calculation based on a camera parameter, the third image processing result, the eye segmentation result of the image frame to be processed and the pupil center to obtain a line of sight landing point of the image frame to be processed; and

[0026] An image processing result calculation unit is configured to determine an eye rotation center and a line of sight direction of the image frame to be processed based on the line of sight landing point.

[0027] The embodiment of the present disclosure further provides an electronic device, comprising a memory, a processor and a computer program stored in the memory and executable on the processor, wherein the processor implements the above-mentioned image frame processing method when executing the computer program.

[0028] Embodiments of the present disclosure also provide a non-transitory computer-readable storage medium storing computer instructions for causing a computer to execute the image frame processing method described above.

[0029] Embodiments of the present disclosure also provide a computer program product comprising computer program instructions for causing a computer to execute the image frame processing method described above when the computer program instructions are run on a computer.

[0030] In the image frame processing method and device described above, the second processing unit waits for a preset waiting time after receiving the image frame to be processed from the caller. If the image processing result of the first processing unit is received within the waiting time, it indicates that the first processing unit has not been preempted by the task of other callers and has completed the image processing in time. Therefore, the second processing unit does not need to repeat the image processing of the image frame to be processed. If the image processing result of the first processing unit is not received after the waiting time, it indicates that the first processing unit has been preempted by the task of other callers and cannot complete the image processing in time. Therefore, the second processing unit processes the image frame to be processed and returns the image processing result to the corresponding caller in time. It can be seen that the image frame processing method described above can ensure that each image frame can return the image processing result before the picture rendering, effectively avoiding the situation that the rendering result of the upper-layer application appears to "jump" due to the first processing unit being preempted and being unable to output the image processing result in time. Therefore, the solution described above can greatly improve the user experience. BRIEF DESCRIPTION OF DRAWINGS

[0031] In order to more clearly illustrate the technical solutions in the present disclosure or the related art, the following will briefly introduce the drawings needed to be used in the embodiments or the related art descriptions. Obviously, the drawings in the following description are only embodiments of the present disclosure, and for those skilled in the art, other drawings can also be obtained without creative labor based on these drawings.

[0032] Figure 1 An application scenario diagram of the existing image frame processing is shown;

[0033] Figure 2 An application scenario diagram of the image frame processing according to some embodiments of the present disclosure is shown;

[0034] Figure 3 An implementation flow of the image frame processing method according to some embodiments of the present disclosure is shown;

[0035] Figure 4An implementation flow of the image frame processing method performed by the second processing unit in the image frame processing method according to some embodiments of the present disclosure is shown.

[0036] Figure 5 An implementation flow of the eye movement tracking image processing of the image frame performed by the second processing unit according to an embodiment of the present disclosure is shown.

[0037] Figure 6 An internal structure of the image frame processing device according to some embodiments of the present disclosure is shown.

[0038] Figure 7 A more specific hardware structure schematic diagram of an electronic device according to the present embodiment is shown. DETAILED DESCRIPTION

[0039] In order to make the objectives, technical solutions and advantages of the present disclosure clearer, the present disclosure will be further described in detail below with reference to specific embodiments and drawings.

[0040] It should be noted that, unless otherwise defined, the technical terms or scientific terms used in the embodiments of the present disclosure should be understood as the common meanings understood by those skilled in the art to which the present disclosure belongs. The terms "first", "second" and similar terms used in the embodiments of the present disclosure do not represent any order, number or importance, but are only used to distinguish different components. The terms "include" or "contain" and similar terms mean that the elements or objects before the terms encompass the elements or objects listed after the terms and their equivalents, without excluding other elements or objects. The terms "connect" or "connected" and similar terms do not mean physical or mechanical connection, but can include electrical connection, whether direct or indirect. The terms "upper", "lower", "left", "right" and the like only represent relative positional relationships, and when the absolute positions of the described objects change, the relative positional relationships may also change accordingly.

[0041] It can be understood that, before using the technical solutions of the various embodiments of the present disclosure, the user will be informed of the type, use range, use scenario and the like of the personal information involved by appropriate means, and the authorization of the user will be obtained.

[0042] For example, in response to receiving the active request of the user, the user is sent prompt information to explicitly prompt the user that the operation requested to be performed will require the acquisition and use of the personal information of the user. Thus, the user can voluntarily choose whether to provide the personal information to the electronic device, application program, server or storage medium and the like software or hardware performing the operation of the technical solutions of the present disclosure according to the prompt information.

[0043] As an optional but non-limiting implementation, in response to receiving the active request of the user, the manner of sending the prompt information to the user may be, for example, a pop-up window manner in which the prompt information can be presented in a textual manner. In addition, the pop-up window can also carry a selection control for the user to select "agree" or "disagree" to provide personal information to the electronic device.

[0044] It can be understood that the above notification and user authorization obtaining process is only illustrative and does not limit the implementation of the present disclosure, and other manners meeting the relevant laws and regulations can also be applied to the implementation of the present disclosure.

[0045] As mentioned above, the visual interaction algorithms involved in the current XR include 6DoF tracking algorithm, gesture recognition algorithm, eye tracking algorithm, and face tracking algorithm, and the like. In order to bring the "immersion" of seamless conversion between the virtual world and the real world to the experimenter, the upper application of the XR usually needs to execute the above-mentioned multiple visual interaction algorithms at the same time. Usually, the upper application of the XR can establish multiple image processing threads in the processing unit to execute the above-mentioned multiple visual interaction algorithms respectively. For example, for each image frame, the upper application needs to establish an image processing thread in the processing unit for executing the 6DoF tracking algorithm on the image frame, and also needs to establish another image processing thread in the processing unit for executing the eye tracking algorithm on the image frame. The two image processing threads share the computing resources of the processing unit. In the embodiment of the present disclosure, each image processing thread corresponds to a caller in the upper application. That is, each caller corresponds to an image processing thread, that is, corresponds to a kind of visual interaction algorithm. When a certain visual interaction algorithm needs to be executed on a certain image frame, the caller corresponding to the visual interaction algorithm can send the image frame to be processed to the processing unit to execute the corresponding visual interaction algorithm on the image frame, receive the image processing result called back by the processing unit, and apply the called back image processing result to the rendering of the image.

[0046] Figure 1 An application scenario diagram of the existing image frame processing is shown. In the application scenario shown in the figure, there are multiple callers: caller 1, caller 2, caller 3, …, caller N. Each caller can correspond to a kind of visual interaction algorithm. For example, caller 1 can correspond to the 6DoF tracking algorithm; caller 2 can correspond to the eye tracking algorithm; caller 3 can correspond to the face tracking algorithm, and the like. Further, in the application scenario shown in the figure, the processing unit is shared by the multiple callers. That is, the processing unit is shared by the multiple image processing threads corresponding to the multiple callers. Figure 1 Figure 1 ​The application scenario shown also includes at least one first processing unit. The at least one first processing unit is configured to execute a corresponding visual interaction algorithm on the image frame under the call of a certain caller, and call back the processing result of the image frame to the caller. Specifically, in an embodiment of the present disclosure, the first processing unit can be an embedded neural network processor (NPU), a graphics processing unit (GPU), or a digital signal processor (DSP).

[0047] It can be understood that, in the application scenario shown, Figure 1 In the application scenario shown, different image processing threads may occupy the same processing unit. For example, caller 1 occupies a first processing unit to execute a 6DoF tracking algorithm on an image frame; at the same time, caller 2 also occupies the first processing unit to execute an eye tracking algorithm. Since the 6DoF tracking algorithm has a higher requirement on the XR rendering frame rate and has a higher priority in processing, the first processing unit will preferentially process the task of caller 1, and then execute the task of caller 2. This situation may cause the eye tracking processing time of the first processing unit on the image frame to be too long from the perspective of caller 2. In the case that the processing unit cannot return the processing result of the eye tracking in time, the upper-layer application needs to discard the data image frame in the eye tracking process, thereby causing the rendering result to jump, which greatly affects the user experience.

[0048] To solve the above problem, an embodiment of the present disclosure provides an image frame processing method. In the image frame processing method provided by the embodiment of the present disclosure, in addition to the first processing unit, a second processing unit is introduced as a backup of the first processing unit. Figure 2 An application scenario diagram of the image frame processing according to some embodiments of the present disclosure is shown. As shown in Figure 2 As shown, each caller can call the computing resources of the second processing unit to perform image processing on the image frame in addition to calling the computing resources of the first processing unit to perform image processing on the image frame. In an embodiment of the present disclosure, the second processing unit is configured to process the current image frame in the case that the processing resources of the first processing unit are preempted and cannot process the current image frame in time, so as to obtain the image processing result of the current image frame in time, and call back the image processing result of the current image frame to the corresponding caller. In an embodiment of the present disclosure, the second processing unit can be a central processing unit (CPU), an NPU, a GPU, or a DSP.

[0049] Figure 3 An implementation flow of the image frame processing method according to an embodiment of the present disclosure is shown. In an embodiment of the present disclosure, Figure 3The method is executed by the caller, the first processing unit and the second processing unit. As Figure 3 The image frame processing method can include the following steps:

[0050] At step 302, the caller sends the image frame to be processed to the first processing unit and the second processing unit respectively.

[0051] The embodiments of the present disclosure do not limit the order in which the caller sends the image frame to be processed to the first processing unit and the second processing unit. Generally, the caller can send the image frame to be processed to the first processing unit and the second processing unit at the same time.

[0052] In addition, it can be understood that before step 302, the caller needs to establish an image processing thread in the first processing unit and the second processing unit respectively, so as to realize image processing of the image frame to be processed by the first processing unit and / or the second processing unit.

[0053] At step 312, after receiving the image frame to be processed, the first processing unit processes the image frame to be processed to obtain a second image processing result.

[0054] In the embodiments of the present disclosure, the image processing can include one or a combination of image processing algorithms involved in XR interaction technologies, such as 6DoF tracking algorithm processing, gesture recognition algorithm processing, eye tracking algorithm processing and face tracking algorithm processing.

[0055] It can be understood that after the first processing unit establishes the image processing thread, the first processing unit will add the established image processing thread to its own task queue and execute the tasks in the task queue in order. In this way, when the first processing unit's task queue does not contain other tasks with higher priority, the first processing unit will process the image frame to be processed in time. When the first processing unit's task queue already contains other tasks with higher priority, the image processing thread needs to wait until the tasks are processed, so that the processing time is longer, which may cause the problem that the image processing result cannot be returned in time.

[0056] It can be understood that corresponding to different image processing types, the first processing unit can obtain different types of image processing results after corresponding image processing. For example, for 6DoF tracking algorithm processing, the rotation angle or displacement of the measured person in 6 degrees of freedom can be obtained after image processing of the image frame by the 6DoF tracking algorithm; for gesture recognition algorithm processing, the gesture type of the measured person can be obtained after image processing of the image frame by the gesture recognition algorithm; for eye movement tracking algorithm processing, the eye features of the measured person, such as the pupil center, eye rotation center, and line of sight direction, can be obtained after image processing of the image frame by the eye movement tracking algorithm; and for face tracking algorithm processing, the face features of the measured person can be obtained after image processing of the image frame by the face tracking algorithm.

[0057] In step 314, the first processing unit calls back the second image processing result to the caller.

[0058] In step 316, the first processing unit sends the second image processing result to the second processing unit.

[0059] It should be noted that the embodiments of the present disclosure do not limit the execution order of steps 214 and 216. The two steps can be executed in parallel or in any order.

[0060] In step 322, after receiving the image frame to be processed, the second processing unit starts a waiting timer based on the pre-set waiting time.

[0061] In the embodiments of the present disclosure, the timing duration of the waiting timer can be set as the pre-set waiting time. Specifically, in the embodiments of the present disclosure, the waiting time can be set according to the average duration required by the first processing unit for image processing of the image frame. In actual applications, the waiting time can be set as a fixed constant value, or the waiting time can be dynamically adjusted during execution. Specifically, the average duration of the first processing unit for image processing of the image frame can be monitored in real time within a pre-set period of time, and then the waiting time is dynamically adjusted based on the monitored average duration. It can be seen that the dynamic adjustment of the waiting time can improve the overall processing efficiency of image frame processing.

[0062] In step 324, in response to determining that the waiting timer is timed out, the second processing unit performs image processing on the image frame to be processed to obtain the first image processing result.

[0063] In step 326, the second processing unit calls back the first image processing result to the caller.

[0064] In step 324, the second processing unit can also cache the first image processing result obtained by the image processing.

[0065] Alternatively, further, in response to determining that the second image processing result is received from the first processing unit before the expiration of the waiting timer, the waiting timer is terminated (the step of terminating the waiting timer is not shown in the figure). Figure 3

[0066] In embodiments of the present disclosure, the image processing performed by the second processing unit can also be one of or a combination of 6DoF tracking algorithm processing, gesture recognition algorithm processing, eye tracking algorithm processing, and face tracking algorithm processing. However, it should be noted that the second processing unit will perform the same type of image processing as the first processing unit, and the image processing algorithm performed is the same as the visual interaction algorithm corresponding to the caller. For example, assuming that the first processing unit performs eye tracking algorithm processing on the image frame, the second processing unit will also perform eye tracking algorithm processing.

[0067] It should be noted that the processes of steps 312-316 and the processes of steps 322-326 are independently and in parallel executed by the first processing unit and the second processing unit, respectively.

[0068] In addition, the above processes can be executed in a loop during the execution of the upper-layer application until the upper-layer application ends. In this case, the caller can further notify the first processing unit and the second processing unit to end the established image processing thread, respectively.

[0069] In the above Figure 3 ​In the illustrated image frame processing method, the caller of the upper layer application starts an image processing thread in the first processing unit, and also starts an image processing thread in the second processing unit. The image processing thread started in the first processing unit is queued in the task queue of the first processing unit after receiving the image frame to be processed from the caller, and waits for processing of the first processing unit. Meanwhile, the second processing unit waits for a preset waiting time after receiving the image frame to be processed from the caller. If the image processing result of the first processing unit is received within the waiting time, it indicates that the first processing unit has not been preempted by tasks of other callers, and has completed image processing in time. Therefore, the second processing unit does not need to repeat image processing on the image frame to be processed. If the image processing result of the first processing unit is not received after the waiting time, it indicates that the first processing unit has been preempted by tasks of other callers, and cannot complete image processing in time. Therefore, the second processing unit processes the image frame to be processed, and returns the image processing result to the corresponding caller in time. It can be seen that, by using the above image processing method, the image processing result of each image frame can be returned before picture rendering, so that the situation that the rendering result of the upper layer application jumps due to the first processing unit being preempted and unable to output the image processing result in time can be avoided, and therefore, the user experience can be greatly improved.

[0070] In the embodiments of the present disclosure, the first processing unit is a device mainly used for executing visual interaction algorithms, and therefore, generally, the first processing unit has stronger computing capability than the second processing unit. For example, the first processing unit is implemented by using a DSP, and the second processing unit is implemented by using a CPU. In the above case, the accuracy and precision of the image processing result of the first processing unit are generally better than those of the image processing result of the second processing unit. Based on this, the above method can further include:

[0071] In step 328, after receiving the second image processing result from the first processing unit, the second processing unit caches the second image processing result from the first processing unit.

[0072] Before performing the above step 328, if the second processing unit has cached the first image processing result obtained by image processing of the second processing unit, the second image processing result from the first processing unit can be used to replace the previously cached first image processing result.

[0073] Based on the above method, the second image processing result from the first processing unit is applied to the processing process of subsequent image frames by the second processing unit. It can be seen that, by using the above steps, the accuracy and precision of image frame processing can be further improved.

[0074] Further, it can be understood that after the processing of all image frames is completed, the invoker also needs to separately notify the first processing unit and the second processing unit that the image processing thread on them is ended.

[0075] Figure 4 The specific flow of the image frame processing method performed by the second processing unit in the image frame processing method according to the embodiments of the present disclosure is shown. As shown in Figure 4 The image frame processing method performed by the second processing unit can specifically include the following steps:

[0076] In step 402, the image frame to be processed sent by the invoker is received.

[0077] In step 404, a waiting timer is set and started based on the pre-set waiting time.

[0078] As described above, in the embodiments of the present disclosure, the timing duration of the waiting timer can be set as the pre-set waiting time. Moreover, in actual application, the waiting time can be set as a fixed constant value, or the waiting time can be dynamically adjusted during execution.

[0079] In step 406, in response to determining that the waiting timer is timed out, the image frame to be processed is processed to obtain a first image processing result.

[0080] In step 408, the first image processing result is called back to the invoker.

[0081] The step 408 can further include caching the first image processing result.

[0082] In step 410, in response to determining that the second image processing result is received from the first processing unit before the waiting timer is timed out, the waiting timer is terminated.

[0083] In some other embodiments of the present disclosure, the method can further include, in response to determining that the second image processing result from the first processing unit is received, caching the second image processing result from the first processing unit.

[0084] In the embodiments of the present disclosure, if the first image processing result has been cached in step 408, the second image processing result from the first processing unit can be used to replace the previously cached first image processing result.

[0085] In the embodiments of the present disclosure, the second processing unit can repeatedly perform the steps 402-412, so as to implement the image processing of each image frame from the upper layer application until the upper layer application is ended.

[0086] As mentioned above, the image processing can include one or a combination of 6DoF tracking algorithm processing, gesture recognition algorithm processing, eye tracking algorithm processing, and face tracking algorithm processing. Taking the eye tracking algorithm processing as an example, Figure 5 The specific implementation process of the second processing unit for performing eye tracking image processing on the image frame is shown. As shown in Figure 5 The eye tracking image processing can include the following steps:

[0087] In step 502, the cached third image processing result is obtained.

[0088] In an embodiment of the present disclosure, the third image processing result can include the pupil center of the previous image frame, the eye rotation center of the previous image frame, and the line of sight direction of the previous image frame. The previous image frame generally refers to the last image frame of the image frame to be processed. If the second processing unit does not cache the last image frame of the image frame to be processed, the second image frame, …, or the Nth image frame before the image frame to be processed can be used as the previous image frame.

[0089] In step 504, the pupil region is selected from the image frame to be processed based on the pupil center of the previous image frame in the third image processing result.

[0090] In step 506, the image frame to be processed and the selected pupil region are input into the inference engine to obtain the eye segmentation result of the image frame to be processed and the pupil center.

[0091] In an embodiment of the present disclosure, the inference engine can be implemented by a convolutional network (CNN).

[0092] In step 508, the camera parameters, the third image processing result, the eye segmentation result of the image frame to be processed, and the pupil center of the image frame to be processed are input into the solver for solving to obtain the line of sight landing point of the image frame to be processed.

[0093] In step 510, the eye rotation center and the line of sight direction of the image frame to be processed are determined based on the line of sight landing point.

[0094] Next, the second processing unit can call back the pupil center, eye rotation center, and line of sight direction of the image frame to be processed as the image processing result of the image frame to be processed to the caller that sends the current image frame.

[0095] In the above Figure 4In the illustrated image frame processing method, after receiving the image frame to be processed from the caller, the second processing unit first waits for a preset waiting time. If the image processing result of the first processing unit is received within the waiting time, it indicates that the first processing unit has not been preempted by the task of other callers and has completed image processing in time, and the second processing unit does not need to repeat image processing on the image frame to be processed. If the image processing result of the first processing unit is not received after waiting for the waiting time, it indicates that the first processing unit may have been preempted by the task of other callers and cannot complete image processing in time, and the second processing unit will then process the image frame to be processed and timely call back the image processing result to the corresponding caller. It can be seen that, by using the above image processing method, the situation that the rendering result of the upper application appears to "jump" due to the first processing unit being preempted and unable to timely output the image processing result can be effectively avoided, and therefore the user experience can be greatly improved.

[0096] Based on the above image frame processing method, an embodiment of the present disclosure further provides an image frame processing device, the internal structure of which is shown in Figure 6 as can include:

[0097] The image frame receiving module 602 is configured to receive the image frame to be processed sent by the caller.

[0098] The timing control module 604 is configured to set and start a waiting timer based on a preset waiting time.

[0099] The image processing module 606 is configured to, in response to determining that the waiting timer is timed out, perform image processing on the image frame to be processed to obtain a first image processing result; or, in response to determining that the second image processing result is received from the first processing unit before the waiting timer is timed out, the second processing unit terminates the waiting timer.

[0100] The callback module 608 is configured to, after the image processing is completed, call back the first image processing result to the caller.

[0101] In an embodiment of the present disclosure, the image frame processing device can further include an image processing result caching module configured to cache the first image processing result or the second image processing result.

[0102] In addition, in the case where the image processing is line-of-sight tracking processing, the image processing module 606 can specifically include:

[0103] An image processing result obtaining unit is configured to obtain the cached third image processing result. The third image processing result can include the pupil center of the previous image frame, the eye rotation center of the previous image frame, and the line-of-sight direction of the previous image frame.

[0104] a pre-processing unit configured to select a pupil region from the image frame to be processed based on the pupil center of the previous image frame;

[0105] a reasoning engine configured to obtain an eye segmentation result of the image frame to be processed based on the selected pupil region, and calculate the pupil center in the image frame to be processed;

[0106] a solver configured to solve based on the camera parameter, the third image processing result, the eye segmentation result of the image frame to be processed, and the pupil center in the image frame to be processed, to obtain the gaze landing point of the image frame to be processed; and

[0107] an image processing result calculation unit configured to determine the eye rotation center and the gaze direction of the image frame to be processed based on the gaze landing point.

[0108] In an embodiment of the present disclosure, the reasoning engine can be implemented by a convolution network.

[0109] The specific implementation of each module can refer to the foregoing method and the accompanying drawings, and the implementation method of each module can also refer to the foregoing method and the accompanying drawings, which will not be described here.

[0110] For the convenience of description, the above device is described as various modules respectively described in function. Of course, the functions of each module can be implemented in the same or multiple software and / or hardware when implementing the present disclosure. The device of the above embodiment is used to implement the corresponding image frame processing method in any of the foregoing embodiments, and has the beneficial effects of the corresponding method embodiments, which will not be described here.

[0111] Figure 7 A more specific hardware structure schematic diagram of an electronic device provided by the present embodiment can include a processor 2010, a memory 2020, an input / output interface 2030, a communication interface 2040, and a bus 2050. The processor 2010, the memory 2020, the input / output interface 2030, and the communication interface 2040 are connected to each other through the bus 2050 for communication within the device.

[0112] The processor 2010 can be implemented by a general-purpose CPU (Central Processing Unit, central processor), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits, etc., for executing related programs to implement the technical solutions provided by the embodiments of the present disclosure.

[0113] The memory 2020 can be implemented in the form of a ROM (Read Only Memory), a RAM (Random Access Memory), a static storage device, a dynamic storage device, etc. The memory 2020 can store an operating system and other application programs, and when the technical solutions provided in the embodiments of the present specification are implemented by software or firmware, the related program codes are stored in the memory 2020 and are called and executed by the processor 2010.

[0114] The input / output interface 2030 is configured to connect input / output devices to realize information input and output. The input / output devices can be configured as components in the device or externally connected to the device to provide corresponding functions. The input devices can include a microphone, various sensors, etc., and the output devices can include a display, a speaker, a vibrator, an indicator light, etc.

[0115] The communication interface 2040 is configured to connect a communication module (not shown in the figure) to realize communication interaction between the device and other devices. The communication module can realize communication through a wired manner (for example, a USB, a network cable, etc.) or through a wireless manner (for example, a mobile network, WIFI, Bluetooth, etc.).

[0116] The bus 2050 includes a channel for transmitting information between various components (for example, the processor 2010, the memory 2020, the input / output interface 2030, and the communication interface 2040) of the device.

[0117] It should be noted that although the above device only shows the processor 2010, the memory 2020, the input / output interface 2030, the communication interface 2040, and the bus 2050, in the specific implementation process, the device can also include other components necessary for normal operation. In addition, those skilled in the art can understand that the above device can also only include components necessary for implementing the embodiments of the present specification, and does not have to include all the components shown in the figure.

[0118] The electronic device of the above embodiments is used to implement the corresponding image frame processing method in any of the above embodiments, and has the beneficial effects of the corresponding method embodiments, which are not described here.

[0119] Based on the same inventive concept, the present disclosure also provides a non-transitory computer readable storage medium storing computer instructions for causing the computer to execute the image frame processing method according to any of the above embodiments.

[0120] The computer readable media of the embodiments can include permanent and non-permanent, removable and non-removable media, which can be implemented by any method or technology to store information. The information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassette, magnetic tape, magnetic disk storage or other magnetic storage device, or any other non-transmission medium that can be used to store information accessible by a computing device.

[0121] The storage medium of the above embodiments stores computer instructions for causing the computer to perform the image frame processing method as described in any of the above embodiments, and has the beneficial effects of the corresponding method embodiments, which are not described here.

[0122] Those skilled in the art should understand that the discussion of any of the above embodiments is only exemplary and is not intended to imply that the scope of the present disclosure (including claims) is limited to these examples; under the idea of the present disclosure, the above embodiments or technical features between different embodiments can also be combined, the steps can be implemented in any order, and there are many other changes of different aspects of the embodiments of the present disclosure as described above. In order to be brief, they are not provided in detail.

[0123] In addition, in order to simplify the description and discussion, and so as not to make the embodiments of the present disclosure difficult to understand, the well-known power / ground connections of integrated circuit (IC) chips and other components can or can not be shown in the provided drawings. In addition, the devices can be shown in the form of block diagrams in order to avoid making the embodiments of the present disclosure difficult to understand, and this also takes into account the fact that the details of the implementation of these block diagram devices are highly dependent on the platform to be implemented the embodiments of the present disclosure (i.e. these details should be fully within the understanding of those skilled in the art). Where specific details (e.g. circuits) are set forth in order to describe an exemplary embodiment of the present disclosure, it will be apparent to those skilled in the art that the embodiments of the present disclosure can be implemented without these specific details or with variations on these specific details. Therefore, these descriptions should be considered illustrative rather than limiting.

[0124] While the present disclosure has been described in connection with certain embodiments thereof, many modifications, substitutions, and variations will be apparent to those of ordinary skill in the art from the foregoing description. For instance, other memory architectures (e.g., dynamic RAM (DRAM)) can use the embodiments discussed.

[0125] Embodiments of the disclosure are intended to cover all such alternatives, modifications, and variations as falling within the broad scope of the appended claims. Accordingly, any one or more of the above-described embodiments can be combined with any one or more of the above-described embodiments in any manner within the scope of the disclosure.

Claims

1. An image frame processing method, comprising: receiving an image frame to be processed sent by a caller; starting a waiting timer based on a pre-set waiting time setting; determining that the waiting timer is timed out, performing first image processing on the image frame to be processed to obtain a first image processing result, and calling back the first image processing result to the caller; wherein the first image processing comprises one or a combination of six-degree-of-freedom tracking algorithm processing, gesture recognition algorithm processing, gaze tracking algorithm processing, and face tracking algorithm processing; and terminating the waiting timer in response to determining that a second image processing result is received from a first processing unit before the waiting timer is timed out; wherein the second image processing result is a processing result obtained by performing second image processing on the image frame to be processed by the first processing unit; and the second image processing is the same type of image processing as the first image processing. The image processing on the image frame to be processed comprises:

2. The method of claim 1, wherein, obtaining a cached third image processing result; wherein the third image processing result comprises a pupil center of a previous image frame, an eye rotation center of the previous image frame, and a gaze direction of the previous image frame; selecting a pupil region from the image frame to be processed based on the pupil center of the previous image frame; inputting the image frame to be processed and the selected pupil region into an inference engine to obtain an eye segmentation result of the image frame to be processed and a pupil center of the image frame to be processed; sending camera parameters, the third image processing result, the eye segmentation result of the image frame to be processed, and the pupil center of the image frame to be processed into a solver to obtain a gaze landing point of the image frame to be processed; and determining the eye rotation center and the gaze direction of the image frame to be processed based on the gaze landing point of the image frame to be processed.

3. The method of claim 1, further comprising: in response to determining that no second image processing result is received from the first processing unit, caching the first image processing result; or in response to determining that a second image processing result is received from the first processing unit, caching the second image processing result. The waiting time of the waiting timer is set to the pre-set waiting time.

5. An image frame processing method, comprising:

4. The method of claim 1, wherein, a caller sending an image frame to be processed to a first processing unit and a second processing unit respectively; after receiving the image frame to be processed, the first processing unit performs second image processing on the image frame to be processed to obtain a second image processing result, and calls back the second image processing result to the caller; wherein the second image processing comprises one or a combination of six-degree-of-freedom tracking algorithm processing, gesture recognition algorithm processing, gaze tracking algorithm processing, and face tracking algorithm processing; and ​ ​ The second processing unit sets and starts a waiting timer based on a preset waiting time setting, in response to determining that the waiting timer expires, performs first image processing on the image frame to be processed to obtain a first image processing result, and calls back the first image processing result to the caller, and in response to determining that the second image processing result is received from the first processing unit before the waiting timer expires, terminates the waiting timer; wherein the first image processing and the second image processing are the same type of image processing.

6. An image frame processing apparatus, comprising: an image frame receiving module configured to receive an image frame to be processed sent by a caller; a timing control module configured to set and start a waiting timer based on a preset waiting time setting; an image processing module configured to, in response to determining that the waiting timer expires, perform first image processing on the image frame to be processed to obtain a first image processing result; and in response to determining that a second image processing result is received from the first processing unit before the waiting timer expires, terminate the waiting timer; wherein the first image processing comprises one or a combination of a six-degree-of-freedom tracking algorithm processing, a gesture recognition algorithm processing, a line-of-sight tracking algorithm processing, and a face tracking algorithm processing; the second image processing result is a processing result obtained by the first processing unit performing second image processing on the image frame to be processed; the second image processing and the first image processing are the same type of image processing; and a callback module configured to call back the first image processing result to the caller.

7. The image frame processing apparatus according to claim 6, further comprising: an image processing result caching module configured to cache the first image processing result or the second image processing result.

8. The image frame processing apparatus according to claim 6, wherein The image processing module comprises: an image processing result obtaining unit configured to obtain a third image processing result cached; wherein the third image processing result comprises a pupil center of a previous image frame, an eye rotation center of the previous image frame, and a line-of-sight direction of the previous image frame; a preprocessing unit configured to select a pupil region from the image frame to be processed based on the pupil center of the previous image frame; an inference engine configured to obtain an eye segmentation result of the image frame to be processed and a pupil center based on the image frame to be processed and the selected pupil region; a solver configured to perform calculation based on camera parameters, the third image processing result, the eye segmentation result of the image frame to be processed, and the pupil center to obtain a line-of-sight landing point of the image frame to be processed; and an image processing result calculation unit configured to determine an eye rotation center and a line-of-sight direction of the image frame to be processed based on the line-of-sight landing point.

9. An electronic device comprising: A memory, a processor, and a computer program stored on the memory and executable on the processor, wherein the processor implements the image frame processing method according to any one of claims 1-4 when executing the computer program.

10. A non-transitory computer-readable storage medium storing computer instructions for causing a computer to perform the image frame processing method of any one of claims 1-4.

11. A computer program product comprising computer program instructions which, when executed on a computer, cause the computer to perform the image frame processing method of any one of claims 1-4.

Images