Algorithm scheduling method and device, chip system and storage medium

Abstract

The invention provides an algorithm scheduling method and device, a chip system and a storage medium, and relates to the technical field of video processing. In the scheme of the invention, an algorithm scheduling system based on the speed of the moving object is constructed. After an algorithm scheduling system obtains a video stream, based on the speed of a moving object in an image frame of the video stream, a frame rate corresponding to each algorithm in multiple algorithms needing to be scheduled at the current moment is obtained through calculation, and then the frame rate is set for an interface of the integration side of each algorithm in the multiple algorithms. Therefore, when the speed of the moving object changes, the algorithm scheduling system can dynamically adjust the frame rate corresponding to each algorithm, for example, the frame rate corresponding to the algorithm is reduced when the moving object is in a static state or at a low speed, and the frame rate corresponding to the algorithm is improved when the moving object is in a medium-high speed. Particularly, when system resources are congested and tense, and picture contents are basically unchanged or slightly changed, the performance of the electronic equipment can be optimized by adopting the scheme, and the power consumption can be reduced.

Description

Technical Field

[0001] This application relates to the field of video processing, and in particular to an algorithm scheduling method, device, chip system and storage medium. Background Technology

[0002] With the development of smart terminals, photo and video recording functions have become a common feature of electronic devices.

[0003] Currently, after the camera is activated, the electronic device begins to capture a video stream through the camera and displays a preview based on the video stream. During this process, the electronic device schedules algorithms to process the image frames of the video stream to perform functions such as adjusting portrait exposure parameters, human tracking, and scene detection. Typically, these algorithms are executed in parallel, and the frame rate (i.e., the number of times an algorithm is scheduled per second) for each algorithm is fixed. In some scenarios, executing multiple algorithms simultaneously at a fixed frame rate can increase the load and lead to higher power consumption. Summary of the Invention

[0004] This application provides an algorithm scheduling method, device, chip system, and storage medium, which solves the high power consumption problem in the process of scheduling multiple algorithms to process video streams.

[0005] To achieve the above objectives, this application adopts the following technical solution:

[0006] Firstly, embodiments of this application provide an algorithm scheduling method. The method includes:

[0007] The system acquires a video stream captured by a camera; based on the speed of moving objects in the image frames of the video stream, it determines the frame rate corresponding to each of the various algorithms that need to be scheduled at the current moment; and schedules each algorithm to process the image frames of the video stream according to its corresponding frame rate. The frame rate for each algorithm represents the number of times each algorithm is scheduled within a preset time period.

[0008] In the above scheme, the frame rate corresponding to each algorithm can be dynamically adjusted when the speed of the moving object changes. For example, the frame rate of the algorithm can be reduced when the moving object is stationary or moving at low speed, and increased when the moving object is moving at medium to high speed. In particular, when system resources are congested and the screen content remains largely unchanged or changes only slightly, this scheme can optimize the performance of electronic devices and reduce power consumption.

[0009] In one possible implementation, the velocity of moving objects in the image frames of the aforementioned video stream is obtained through an optical flow module or software module of an electronic device. The optical flow module may include an optical flow sensor, a high-resolution image sensor used to calculate motion information such as the velocity and direction of motion of moving objects based on the video stream captured by the camera. It is understood that an image frame may include one or more moving objects. When an image frame includes multiple moving objects, a primary object can be identified among the multiple moving objects, and then the velocity of that primary object can be calculated, so that the velocity of the primary object is used as the velocity of the moving object (i.e., the primary object) in that image frame. Utilizing the velocity of the moving objects allows for better adjustment of the frame rate corresponding to each algorithm, optimizing performance.

[0010] In one possible implementation, determining the frame rate corresponding to each of the multiple algorithms that need to be scheduled at the current moment, based on the speed of the moving object in the image frame of the video stream, may include: obtaining the speed of the moving object in the image frame of the video stream; obtaining the preset frame rate corresponding to each algorithm at M preset speeds, where M is an integer greater than or equal to 2; and determining the frame rate corresponding to each algorithm based on the speed of the moving object and the preset frame rate corresponding to each algorithm at the M preset speeds. For example, the electronic device pre-stores a scheduling data table, which includes the preset frame rates corresponding to N algorithms at M preset speeds, and the N algorithms include the multiple algorithms that need to be scheduled at the current moment.

[0011] In the above scheme, a scheduling data table is usually stored in advance, so that after the electronic device determines the multiple algorithms that need to be scheduled at the current moment, it can look up the scheduling data table to obtain the preset frame rate corresponding to each of the multiple algorithms at M preset speeds. Based on the speed of the moving object and the preset frame rate corresponding to each of the algorithms at M preset speeds, the frame rate corresponding to each algorithm is determined.

[0012] In one possible implementation, determining the frame rate corresponding to each algorithm based on the speed of the moving object and the preset frame rate corresponding to each algorithm at M preset speeds can include:

[0013] For each algorithm, the following steps are performed: The speed of the moving object is determined, and its relationship to the M preset speeds corresponding to an algorithm is analyzed. If the speed of the moving object is greater than the first preset speed among the M preset speeds corresponding to the algorithm, and the speed of the moving object is less than the second preset speed among the M preset speeds corresponding to the algorithm, then the frame rate corresponding to the algorithm is determined based on the speed of the moving object, the preset frame rate corresponding to the first preset speed, and the preset frame rate corresponding to the second preset speed. If the speed of the moving object is equal to the third preset speed among the M preset speeds corresponding to the algorithm, then the preset frame rate corresponding to the third preset speed under the algorithm is used as the frame rate corresponding to the algorithm. If the speed of the moving object is greater than the maximum preset speed among the M preset speeds corresponding to the algorithm, then the preset frame rate corresponding to the maximum preset speed under the algorithm is used as the frame rate corresponding to the algorithm. If the speed of the moving object is less than the minimum preset speed among the M preset speeds corresponding to the algorithm, then the preset frame rate corresponding to the minimum preset speed under the algorithm is used as the frame rate corresponding to the algorithm. Among them, the M preset speeds corresponding to an algorithm are arranged in ascending order. The first preset speed and the second preset speed are two adjacent preset speeds among the M preset speeds corresponding to the algorithm. That is, there are no other preset speeds in the speed range from the first preset speed to the first speed, and there are no other preset speeds in the speed range from the first speed to the second preset speed.

[0014] For example, when the speed of a moving object is greater than the first preset speed among the M preset speeds corresponding to an algorithm, and the speed of the moving object is less than the second preset speed among the M preset speeds corresponding to the algorithm, the frame rate (also known as the actual frame rate) corresponding to each of the multiple algorithms can be calculated according to the interpolation algorithm. The interpolation algorithm can be a linear interpolation algorithm, a velocity frame interpolation algorithm, or other interpolation algorithms, etc.

[0015] The following example illustrates the process of calculating the actual frame rate for each of the various algorithms that need to be scheduled at the current moment using a linear interpolation algorithm.

[0016] Electronic devices can determine the frame rate corresponding to a given algorithm using either the first or the second relational expression.

[0017] The first relation is:

[0018]

[0019] The second relationship is:

[0020]

[0021] in, The floor symbol is used for rounding down. The rounding sign is S, which represents the speed of the moving object, F, which represents the frame rate corresponding to the algorithm, Sa, which represents the first preset speed, Sb, which represents the second preset speed, Fa, which represents the preset frame rate corresponding to the first preset speed, and Fb, which represents the preset frame rate corresponding to the second preset speed.

[0022] In the above scheme, the speed of the moving object typically changes gradually, and the speed of the moving object in the image frames calculated by the electronic device also gradually increases or decreases. Thus, in multi-round algorithm scheduling, the frame rate corresponding to each algorithm calculated based on the speed of the moving object will also conform to a gradual change pattern. For example, when the moving object changes from slow motion to stillness, the frame rate corresponding to a certain algorithm changes sequentially to 30 FPS, 25 FPS, 20 FPS, 10 FPS, and 5 FPS, achieving a gradual decrease in frame rate. Similarly, when the moving object changes from stillness to slow motion, the frame rate corresponding to a certain algorithm changes sequentially to 5 FPS, 10 FPS, 20 FPS, 25 FPS, and 30 FPS, achieving a gradual increase in frame rate. It can be understood that when the moving object in the image is basically still, the above algorithm scheduling method offers significant performance and power consumption optimization; gradual frame rate adjustment can maximize both performance and power consumption reduction.

[0023] In one possible implementation, before determining the frame rate corresponding to each of the multiple algorithms to be scheduled at the current moment based on the speed of the moving object in the image frame of the video stream, the method may further include: obtaining the speed of the moving object in the image frame of the video stream according to a first scheduling speed. Correspondingly, after determining the frame rate corresponding to each of the multiple algorithms to be scheduled at the current moment based on the speed of the moving object in the image frame of the video stream, the method may further include: redetermining the first scheduling speed based on the maximum frame rate among the frame rates corresponding to each of the multiple algorithms. For example, when the speed of the moving object is S = 2 m / s, the frame rate corresponding to Algorithm 1 is 8 FPS through linear interpolation, the frame rate corresponding to Algorithm 2 is 5 FPS, the frame rate corresponding to Algorithm 3 is 2 FPS, and the frame rate corresponding to Algorithm 4 is 1 FPS. The final redetermined first scheduling speed is 8 FPS. Thus, in the next round of algorithm scheduling, the object speed module can be requested at a frame rate of 8 FPS, i.e., the object speed module is scheduled once every 125 ms.

[0024] In the above scheme, the algorithm scheduling system can request the speed of the moving object from the object speed module according to the first scheduling speed. The first scheduling speed is associated with the speed of the moving object obtained by the object speed module in the previous round of algorithm scheduling. That is to say, in each round of algorithm scheduling, the algorithm scheduling system needs to redetermine the first scheduling speed based on the maximum frame rate among the frame rates corresponding to each of the multiple algorithms that need to be scheduled at the current time, so as to facilitate the next round of algorithm scheduling.

[0025] Secondly, this application provides an apparatus comprising units for performing the method described in the first aspect above. This apparatus is adapted to perform the method described in the first aspect above, and a description of the units within this apparatus is provided in the description of the first aspect above; for brevity, it will not be repeated here.

[0026] Thirdly, this application provides an electronic device comprising: one or more processors, and a memory. The memory is coupled to the one or more processors and is used to store computer program code, the computer program code including computer instructions, wherein the one or more processors invoke the computer instructions to cause the electronic device to perform the methods provided by the first aspect and any possible implementation thereof.

[0027] Fourthly, this application provides a computer-readable storage medium. The computer-readable storage medium includes computer instructions. When executed on an electronic device, the computer instructions cause the electronic device to perform the method provided by the first aspect and any possible implementation thereof.

[0028] Fifthly, this application provides a computer program product. When the computer program product is run on a computer, it causes the computer to perform the method provided by the first aspect and any possible implementation thereof.

[0029] In a sixth aspect, this application provides a chip system applied to an electronic device, the chip system including one or more processors, the one or more processors being configured to invoke computer instructions to cause the electronic device to perform the methods provided in the first aspect and any possible implementation thereof.

[0030] It is understood that the beneficial effects achieved by the apparatus of the second aspect, the electronic device of the third aspect, the computer-readable storage medium of the fourth aspect, the computer program product of the fifth aspect, and the chip system of the sixth aspect provided above can be referred to as the beneficial effects of the first aspect and any possible implementation thereof, which will not be repeated here. Attached Figure Description

[0031] Figure 1 This application provides an operational schematic diagram for a portrait photography scenario.

[0032] Figure 2 A schematic diagram of the dynamic scheduling system provided in the embodiments of this application;

[0033] Figure 3 A schematic diagram of the hardware structure of the electronic device provided in the embodiments of this application;

[0034] Figure 4 A schematic diagram of the architecture of an electronic device provided in an embodiment of this application;

[0035] Figure 5 A flowchart of an algorithm scheduling method provided in an embodiment of this application;

[0036] Figure 6 A flowchart of another algorithm scheduling method provided in the embodiments of this application;

[0037] Figure 7 This diagram illustrates how various algorithms provided in this application process image frames according to their respective frame rates. Detailed Implementation

[0038] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are some embodiments of this application, but not all embodiments.

[0039] In the description of this application, unless otherwise stated, " / " means "or". For example, A / B can mean A or B. In the description of this application, "and / or" is merely a way of describing the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent three cases: A alone, A and B simultaneously, and B alone.

[0040] In the specification and claims of this application, the terms "first" and "second," etc., are used to distinguish different objects or to distinguish different treatments of the same object, rather than to describe a specific order of the objects. For example, "first speed" and "second speed," etc., are used to distinguish different speeds, rather than to describe a specific order of speeds. In the embodiments of this application, "a plurality of" refers to two or more.

[0041] References to "one embodiment" or "some embodiments" as described in this specification mean that one or more embodiments of this application include a specific feature, structure, or characteristic described in connection with that embodiment. Therefore, the phrases "in one embodiment," "in some embodiments," "in other embodiments," "in still other embodiments," etc., appearing in different parts of this specification do not necessarily refer to the same embodiment, but rather mean "one or more, but not all, embodiments," unless otherwise specifically emphasized. The terms "comprising," "including," "having," and variations thereof mean "including but not limited to," unless otherwise specifically emphasized.

[0042] With the development of terminal technology, cameras offer an increasing variety of shooting modes, such as still photography, portrait, video recording, night scene, aperture, panorama, and slow motion. Users can choose a shooting mode from these options based on their needs, such as portrait mode. Typically, after a user selects a shooting mode, the electronic device displays a preview screen based on a video stream (also known as a data stream). If the user is satisfied with the preview, they can tap the shutter button, and the electronic device will respond to the tap to take a photo or video.

[0043] Once the camera application is enabled, in various preview modes, such as photo preview mode or video preview mode, and during video recording, the electronic device will capture a video stream composed of image frames and call some perception-based algorithms to process the image frames in order to complete the corresponding functions.

[0044] In the embodiments of this application, the aforementioned perception-type algorithms, also known as preview algorithms, refer to a series of algorithms used to process images, including but not limited to face detection algorithms, object tracking algorithms, single-target object tracking algorithms, human body tracking algorithms, face tracking algorithms, portrait exposure parameter adjustment algorithms, scene detection algorithms, and key point detection algorithms.

[0045] For example, taking a mobile phone as an electronic device, Figure 1 This diagram illustrates an operation in a portrait photography scenario.

[0046] like Figure 1As shown in (a), the phone displays icons for applications such as the camera on the desktop. When a user wants to take a picture, they can tap the camera icon 01. The phone responds to the user's tap on the camera icon 01 and launches the camera application. After the camera application completes initialization, it notifies the camera to capture an image via the camera frame and camera driver. The camera transmits the data stream acquired by the sensor to the image signal processor. The image signal processor performs a first format conversion on the data stream acquired by the camera to obtain a first data stream (also called a tiny stream, preview stream, or preview video stream). For example, the data stream acquired by the camera is in RAW format, while the tiny stream is in RGB format. The tiny stream has a relatively small resolution, such as several hundred pixels multiplied by several hundred pixels. The image signal processor sends the tiny stream to the screen, so that the tiny stream is presented as a preview image in the preview frame, for example, in a... Figure 1 The preview image displayed in the preview interface of (b) is generated based on the tiny stream. For example... Figure 1 Taking the preview interface shown in (b) as an example, the preview interface may include a preview box 02, aperture mode selection options, night scene mode selection options, portrait mode selection options, photo mode selection options, video recording mode selection options, professional mode selection options, photo preview control 04, photo capture control 05, and other controls. The preview box 02 is used to display the video stream captured by the camera, such as a video stream composed of multiple frames of the user's selfie. When the user selects the photo mode selection option, a triangle control 03 is displayed below the photo mode selection option to indicate to the user that the photo mode has been successfully selected. The photo preview control 04 is used to display the user's most recently taken photo or video in thumbnail format. The photo capture control 05 is used by the user to confirm and trigger the phone to capture a photo or video.

[0047] During the display of the video stream, the mobile phone can call various algorithms such as face tracking algorithm and portrait exposure parameter adjustment algorithm to process the image frames in order to achieve the corresponding functions.

[0048] Among them, face tracking algorithms are mainly based on advanced technologies such as computer vision and artificial intelligence. They capture facial information through a camera, automatically identify the face's location using algorithms, and then extract facial features, eye features, mouth features, etc., to achieve face tracking. For example, in... Figure 1Taking the image displayed in preview box 02 shown in (b) as an example, which is the i-th frame image, the mobile phone can use a face tracking algorithm to determine that the i-th frame image includes two face images. The algorithm then identifies one of the two face images as the main subject, and displays a face frame 06 in the area where this main subject is located. When the relative position of the main subject to the camera changes, the mobile phone can track it based on the features of the main subject and continue to use the face frame 06 to surround the area where the main subject is located in subsequent image frames, thereby achieving face tracking.

[0049] The main functions of the portrait exposure parameter adjustment algorithm include: adjusting the camera's exposure parameters in real time based on the brightness of the face region in the video stream, thereby adjusting the brightness of subsequently acquired video streams. Specifically, the mobile phone can obtain the overall brightness of the face region in the acquired i-th frame image, and then adjust the camera's exposure parameters based on this overall brightness, thereby capturing the i+j-th frame image based on the adjusted exposure parameters, where i and j are positive integers. The mobile phone can crop a face image 1 from image frame 1 acquired at time t1 and determine exposure parameter 1 based on the overall brightness of face image 1; crop a face image 2 from image frame 2 acquired at time t2 and determine exposure parameter 2 based on the overall brightness of face image 2; crop a face image 3 from image frame 3 acquired at time t3 and determine exposure parameter 3 based on the overall brightness of face image 3... Then, at time t(1+j), image frame 1+j is acquired based on exposure parameter 1; at time t(2+j), image frame 2+j is acquired based on exposure parameter 2; at time t(3+j), image frame 3+j is acquired based on exposure parameter 3. This allows for the following... Figure 1 Taking the image displayed in preview box 02 shown in (b) as the i-th frame as an example, if the phone detects that the overall brightness of the face area (such as the rectangular area surrounded by face frame 06) in the i-th frame is too dark, the phone can increase the camera's exposure value and / or gain value, and acquire the i+j-th frame image based on the increased exposure value and / or gain value, and then... Figure 1 Preview box 06 (c) shows the (i+j)th frame image. The (i+j)th frame image can be the next frame image captured after the i-th frame image, or there can be at least one frame interval between the (i+j)th frame image and the i-th frame image. It should be noted that... Figure 1 The brightness of a face image is represented by grayscale. Darker grayscale indicates a lower brightness of the face image, while lighter grayscale indicates a higher brightness of the face image.

[0050] When the user is like Figure 1 When the preview image shown in (c) is satisfactory, such as Figure 1As shown in (d), the user can click the camera control 05. In response to the user's click on the camera control 05, the camera application, through the camera frame and camera driver, notifies the image signal processor to perform a second format conversion on the data stream acquired by the camera, obtaining a second data stream (also called a photo stream). For example, the photo stream is in RGB format. The photo stream has a relatively high resolution, such as several thousand pixels multiplied by several thousand pixels. Because each frame in the photo stream has a higher resolution and better image quality, the camera application usually generates the final photo or video based on the photo stream, for example, in a... Figure 1 Photo 07 in the album interface of (f) is generated based on the photo stream.

[0051] The frames in the tiny stream and the frames in the captured image stream have a one-to-one mapping relationship. Each frame in both the tiny stream and the captured image stream is marked with a timestamp to represent the time the image was acquired. For example, frame 001 in the tiny stream and frame 101 in the captured image stream are both marked with timestamp t0, indicating that frames 001 and 101 are both converted from the data stream captured by the camera at time t0; frame 002 in the tiny stream and frame 102 in the captured image stream are both marked with timestamp t1, indicating that frames 002 and 102 are both converted from the data stream captured by the camera at time t1; frame 003 in the tiny stream and frame 103 in the captured image stream are both marked with timestamp t2, indicating that frames 003 and 103 are both converted from the data stream captured by the camera at time t2; ...; frame 012 in the tiny stream and frame 112 in the captured image stream are both marked with timestamp t11, indicating that frames 012 and 112 are both converted from the data stream captured by the camera at time t11. Furthermore, the formats of the tiny stream and the image stream can be the same or different. For example, both the tiny stream and the image stream can be in RGB format. The difference between the tiny stream and the image stream is that the resolution of the tiny stream is lower than that of the image stream. Taking frames 001 and 101 as examples, since both frames 001 and 101 are converted from the data stream captured by the camera at time t0, theoretically, the image content of frames 001 and 101 should be the same. Frame 001 is obtained by the image signal processor performing a first format conversion on the data stream, resulting in a smaller resolution, such as 360*720 pixels. Frame 101 is obtained by the image signal processor performing a second format conversion on the data stream, resulting in a larger resolution, such as 1024*2048 pixels. It should be noted that the video stream mentioned in the embodiments of this application for processing image frames of video streams by calling perception-type algorithms can be a RAW format data stream captured by a camera, a tiny stream converted from a RAW format data stream, a photo stream converted from a RAW format data stream, or other data streams such as RGB or YUV involved in the process of taking photos or videos. This application does not make any specific limitations.

[0052] Furthermore, such as Figure 1 As shown in (e), after the phone completes taking a photo, it updates the thumbnail in the photo preview control 04 to the thumbnail of the photo taken this time. Because the thumbnail in the photo preview control 04 is small, the user may not be able to clearly see the specific content of the photo. In this case, as... Figure 1 As shown in (f), the user can click the photo preview control 04. In response to the user's click on the photo preview control 04, the phone opens the gallery application and, as shown in [the image / image / etc.], displays the preview image. Figure 1 The interface shown in (f) displays photo 07 taken this time.

[0053] Referring to the description of the above embodiments, during the process of taking photos or videos, the electronic device calls some algorithms to process the image frames of the video stream to complete corresponding functions such as portrait exposure parameter adjustment, human body tracking, and scene detection. It should be noted that the number and types of perception algorithms called by the electronic device can be the same or different in different shooting modes. Furthermore, the frame rates corresponding to the same type of perception algorithms called by the electronic device can be the same or different in different shooting modes. In this embodiment, the unit of the frame rate corresponding to the algorithm is frames per second (FPS), representing the number of times an algorithm is scheduled within a preset duration (e.g., one second).

[0054] Currently, algorithms related to video streaming mainly employ the following two scheduling strategies:

[0055] One approach is a static scheduling strategy. For example, algorithm A might have a frame rate of 30 FPS (i.e., algorithm A is scheduled 30 times per second, or once every 33 ms), while algorithm B might have a frame rate of 60 FPS (i.e., algorithm B is scheduled 60 times per second, or once every 16.7 ms). Regardless of whether the subject is stationary, moving slowly, or moving quickly, the frame rate for each algorithm remains constant. When system resources are congested and the image content remains largely unchanged or changes only slightly, executing multiple algorithms simultaneously at a high fixed frame rate will increase the load and lead to higher power consumption.

[0056] Another strategy is dynamic scheduling. This strategy assesses the current system resource congestion, such as the processing time of previous algorithms and CPU utilization. Then, based on the total frame output time (usually determined by the electronic device system), it prioritizes the algorithms to be executed in the current frame, scheduling them from highest to lowest priority. If the total execution time of an algorithm exceeds the total frame output time, that algorithm is not executed in the current frame and is moved to the next frame. This dynamic strategy can reduce the number of algorithm calls to some extent while maintaining the smoothness of the image output process, but the overall algorithm scheduling strategy remains largely unchanged. In particular, for stationary or low-speed moving scenarios, this strategy does not significantly improve performance or power consumption.

[0057] In view of the above problems, embodiments of this application provide an algorithm scheduling method. This method can be applied to electronic devices. In this electronic device, an algorithm scheduling system based on the velocity of moving objects is constructed. Figure 2 As shown, after acquiring the video stream, the algorithm scheduling system can obtain the speed information of moving objects (such as the main object) in the image frames of the video stream. Then, based on the speed information of the moving objects, the algorithm scheduling system calculates the frame rate corresponding to each of the various algorithms (such as Algorithm 1, Algorithm 2, and Algorithm 3) that need to be scheduled at the current time. For example, the frame rate of Algorithm 1 is frame rate 1, the frame rate of Algorithm 2 is frame rate 2, and the frame rate of Algorithm 3 is frame rate 3. Then, the algorithm scheduling system sets the frame rate for the interface of each algorithm integration side and completes the scheduling.

[0058] The following is combined Figure 3 and Figure 4 This section introduces the hardware and software architecture of electronic devices.

[0059] In some embodiments, the aforementioned electronic device is also referred to as a terminal device or user equipment (UE). For example, the electronic device may be a personal computer (PC), mobile phone, smart screen, smart TV, tablet computer, wearable device, computer with wireless transceiver function, virtual reality (VR) device, augmented reality (AR) device, wireless terminal in industrial control, wireless terminal in self-driving, wireless terminal in remote medical surgery, wireless terminal in smart grid, wireless terminal in transportation safety, wireless terminal in smart city, or wireless terminal in smart home, etc., or it may be other devices or apparatuses with photography or video recording functions.

[0060] For example, Figure 3 This is a schematic diagram of the hardware structure of an electronic device 100 provided in an embodiment of this application.

[0061] like Figure 3 As shown, the electronic device 100 may include a processor 110, a memory 120, buttons 130, a sensor module 140, a display screen 150, an audio module 160, a speaker 160A, a receiver 160B, a microphone 160C, a headphone jack 160D, a camera 170, and an optical flow module 180, etc.

[0062] The processor 110 can be used to execute the algorithm scheduling method in this embodiment based on the video stream captured by the camera 170. The processor 110 may include one or more processing units, such as a central processing unit (CPU), a neural processing unit (NPU), a graphics processing unit (GPU), an application processor (AP), a digital signal processor (DSP), or an image signal processor (ISP). Different processing units can be independent devices; they can also be integrated into one or more processors, for example, the NPU can be located within the DSP.

[0063] The memory 120 can be used to store computer executable program code, which includes instructions. The processor 110 executes various functional applications and data processing of the electronic device 100 by running the instructions stored in the memory 120. The memory 120 may include a program storage area and a data storage area. The program storage area may store the operating system and at least one application (APP) required for a function, such as a camera application, a gallery application, etc. The data storage area may store configuration files for each APP, as well as data created during the use of the electronic device 100, such as YUV images obtained by processing RAW data streams captured by a camera through 3A processing and image signal processing (ISP).

[0064] Button 130 includes physical buttons such as a power button and volume buttons. Button 130 can be a mechanical button or a touch button. Electronic device 100 can receive button input and generate key signal inputs related to user settings and function control of electronic device 100, such as key signal input to trigger camera 170 to capture images.

[0065] Sensor module 140 may include various types of sensors, such as image sensors and touch sensors. An image sensor, also known as a photosensitive element, is a device that uses the photoelectric conversion function of an optoelectronic device to convert a light image on a photosensitive surface into an electrical signal proportional to the light image. For example, an image sensor may be a complementary metal-oxide-semiconductor image sensor (CMOS image sensor, CIS). A touch sensor, also called a "touch panel," can be disposed on the display screen 150. The touch sensor and the display screen 150 together form a touch screen, also called a "touch screen." The touch sensor is used to detect touch operations applied to or near it. The touch sensor can transmit the detected touch operation to the application processor to determine the type of touch event and provide visual output related to the touch operation through the display screen 150. In some embodiments, the touch sensor may also be disposed on the surface of the electronic device 100, in a different location than the display screen 150. It should be noted that the above sensors may be independent functional modules in the electronic device or may be disposed within certain functional devices; for example, an image sensor may be integrated into a camera 170.

[0066] The display screen 150 includes a display panel for displaying the desktop, a photo preview, various images in the gallery, etc.

[0067] Electronic device 100 can implement audio functions, such as music playback and recording, through an audio module 160, a speaker 160A, a receiver 160B, a microphone 160C, a headphone jack 160D, and an application processor. The audio module 160 is used to convert digital audio information into analog audio signals for output, and also to convert analog audio input into digital audio signals. The audio module 160 can also be used for encoding and decoding audio signals. The speaker 160A, also called a "loudspeaker," is used to convert audio electrical signals into sound signals. Electronic device 100 can listen to music or make hands-free calls through the speaker 160A. The receiver 160B, also called a "handpiece," is used to convert audio electrical signals into sound signals. When electronic device 100 answers a phone call or voice message, the receiver 160B can be brought close to the user's ear to hear the voice. The microphone 160C, also called a "microphone" or "voice transducer," is used to convert sound signals into electrical signals. When making a phone call or sending a voice message, the user can speak by bringing their mouth close to the microphone 160C, inputting the sound signal into the microphone 160C. The headphone jack 160D is used to connect wired headphones.

[0068] Camera 170 can be used to capture still images or videos. Light from an object passes through a lens to generate an optical image, which is then projected onto a photosensitive element. The photosensitive element can be a charge-coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor. The photosensitive element converts the light signal into an electrical signal, which is then transmitted to an ISP (Internet Service Provider) for conversion into a digital image signal. The ISP outputs the digital image signal to a DSP (Digital Signal Processor) for processing. The DSP converts the digital image signal into image signals in standard formats such as RGB and YUV. In some embodiments, electronic device 100 may include one or N cameras 170, where N is a positive integer greater than 1.

[0069] The optical flow module 180 can be used to calculate the velocity information of moving objects in image frames of a video stream. In some embodiments, the optical flow module 180 is linked to the camera 170. The optical flow module 180 may include an optical flow sensor, which is a high-resolution image sensor used to output pixel-level motion information such as velocity and direction based on the video stream acquired by the camera.

[0070] It is understood that the structures illustrated in the embodiments of this application do not constitute a specific limitation on the electronic device 100. In other embodiments of this application, the electronic device 100 may include more or fewer components than illustrated, or combine some components, or split some components, or have different component arrangements. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

[0071] The following provides an example illustration of the software system of an electronic device. The software system of an electronic device can adopt a layered architecture, event-driven architecture, microkernel architecture, or microservice architecture. This application embodiment uses the layered architecture Android system as an example to illustrate the software system architecture of an electronic device.

[0072] For example, Figure 4 A schematic diagram of the architecture of an electronic device provided in an embodiment of this application is shown.

[0073] like Figure 4 As shown, electronic devices can adopt a layered architecture, dividing the software into several layers, each with a clear role and function. Layers communicate with each other through software interfaces. In some embodiments, the software layers of the software architecture are divided from top to bottom as follows: application layer, application framework (FW) layer, system library (FWK LIB), hardware abstraction layer (HAL) layer, and kernel layer. This software architecture runs on top of the hardware layers, which may include components such as CPU, NPU, DSP, and camera.

[0074] The application layer, also known as the application layer, can include a series of application packages. For example, the application layer can include a camera application, a gallery application, etc. The camera application is used to call the camera to capture tiny streams and photo streams, and display the captured preview images; the gallery application is used to store and display the final photos obtained through the camera application. When these application packages are run, they can access the various service modules provided by the application framework layer through the application programming interface (API) and execute corresponding intelligent business logic.

[0075] The application framework layer provides APIs and a programming framework for applications in the application layer. The application framework layer includes some predefined functions. It may include an Activity Manager Service (AMS), a Window Manager Service (WMS), and a camera service, among others. AMS manages the lifecycle of each application. WMS manages all windows in the system. The camera service may include at least an algorithm scheduling system. The specific implementation of the algorithm scheduling system, which sets the frame rate for each of the various algorithms needed for scheduling at the current moment based on the speed of the moving object, can be found in the description of the following embodiments, and will not be elaborated upon here.

[0076] The system library can include multiple functional modules, such as a surface manager, media libraries, a 2D graphics engine (e.g., SGL), and a 3D graphics processing library (e.g., OpenGLES). The surface manager manages the display subsystem and provides the fusion of 2D and 3D layers for multiple applications. The media library supports playback and recording of various common audio and video formats, as well as still image files. It supports multiple audio and video encoding formats. The 2D graphics engine is the drawing engine for 2D graphics. The 3D graphics processing library is used to implement 3D graphics drawing, image rendering, compositing, and layer processing.

[0077] Within the system libraries, the Android Runtime comprises the core libraries and the virtual machine. The Android Runtime is responsible for the scheduling and management of the Android system. The core libraries consist of two parts: one part contains the functionalities that Java calls, and the other part comprises the core Android libraries. The application layer and application framework layer run within the virtual machine. The virtual machine executes the Java files of the application layer and application framework layer as binary files. The virtual machine is used for managing object lifecycles, stack management, thread management, security and exception management, and garbage collection, among other functions.

[0078] The hardware abstraction layer has standard interfaces implemented by hardware vendors. For example, the hardware abstraction layer may include camera HAL, abstract interfaces for various algorithms related to video streaming, and algorithm deployment modules. The algorithm deployment module is used to deploy abstract interfaces for multiple perception algorithms scheduled by an algorithm scheduling system to run on corresponding processor platforms, such as CPUs, NPUs in DSPs, and GPUs. In some embodiments, to reduce the overall performance and power consumption of the electronic device, some algorithms can be deployed on the CPU, while model inference and some preprocessing algorithms can be placed in the NPU of the DSP. The NPU's AI model can perform model inference; for example, after the CPU acquires an image of a face region, it can input the face region image into the AI ​​model to obtain a mask image (MSAK) carrying real face data. The CPU can then calculate exposure parameters based on the average brightness of the real face portion corresponding to the MSAK and adjust the camera's exposure parameters accordingly.

[0079] The kernel layer is the layer between hardware and software, and belongs to the lower level of the Android system. The kernel layer can contain various driver interfaces, such as camera drivers, display drivers, audio drivers, etc.

[0080] It should be noted that although the embodiments of this application are illustrated using the Android system as an example, the basic principles are equally applicable to electronic devices based on operating systems such as iOS or Windows.

[0081] Below Figure 3 and Figure 4 Based on the provided functional modules, combined with Figures 5 to 7 The specific implementation of the algorithm scheduling method provided in the embodiments of this application is illustrated with examples.

[0082] For example, Figure 5 A flowchart of an algorithm scheduling method is shown.

[0083] This method can be applied to electronic devices. For example... Figure 5 As shown, the method may include the following steps S101 to S113.

[0084] S101, the camera application receives the user's action of opening the camera application, or receives a request from other applications to call the camera application, or receives the user's action of triggering video recording.

[0085] For example, a camera app can receive user clicks on the camera app icon, or user selection of a shooting mode (such as video recording mode) in the camera app's preview interface, or user actions in other apps that trigger the opening of the camera (such as video call, photo taking, or scanning).

[0086] S102, the camera application sends a shooting request to the camera driver, instructing the capture of a video stream.

[0087] S103, in response to the shooting request, the camera driver sends a shooting command to the camera, instructing it to capture the video stream.

[0088] For example, in response to a user opening the camera application or a request from another application to invoke the camera application, the camera application at the application layer identifies the current shooting scene and determines the camera corresponding to the current shooting scene (e.g., opening the rear camera in a distant shooting scene, or opening the front camera in a selfie scene), the camera's frame rate range (including the maximum and minimum frame rates), and the shooting mode. The camera application sequentially sends the camera ID, frame rate range, shooting mode, and other parameters corresponding to the current shooting scene to the camera driver through the camera service and camera HAL. The camera driver can open the corresponding camera based on the camera ID corresponding to the current shooting scene. The opened camera acquires a video stream based on the frame rate range and shooting mode. Referring to the description of the above embodiments, the video stream acquired by the camera is a RAW format electrical signal, i.e., a RAW image.

[0089] S104, the camera returns the captured video stream, i.e., RAW image, to the camera driver.

[0090] S105, the camera driver transmits the video stream to the object velocity module.

[0091] The camera driver can directly transmit the acquired video stream (i.e., RAW image) to the object velocity module, or certain functional modules can convert the acquired video stream (i.e., RAW image) into other video stream formats (such as YUV or RGB format) and then transmit it to the object velocity module so that the object velocity module can calculate the velocity of the moving object (such as the main object) based on the video stream.

[0092] S106, the algorithm scheduling system requests the velocity of the moving object from the object velocity module.

[0093] In some embodiments, such as Figure 6As shown, S106 can be specifically implemented through S106A: the algorithm scheduling system can request the speed of the moving object from the object speed module according to the first scheduling speed.

[0094] As an optional implementation, the first scheduling speed is a preset value, which is greater than or equal to the maximum frame rate corresponding to each perception algorithm. For example, if the perception algorithms that the electronic device can schedule include Algorithm 1, Algorithm 2, and Algorithm 3, and the maximum frame rate corresponding to Algorithm 1 is 30 FPS, Algorithm 2 is 20 FPS, and Algorithm 3 is 10 FPS, then this preset value is greater than or equal to 30 FPS.

[0095] As an alternative implementation, the first scheduling speed is a dynamically changing value, which is associated with the speed of the moving object obtained through the object speed module. For example, after the electronic device enters a certain shooting mode, in each round of algorithm scheduling, the algorithm scheduling system can schedule the object speed module according to the first scheduling speed to schedule perception-based algorithms. The first scheduling speed used in each round of algorithm scheduling is determined based on the maximum frame rate among the frame rates corresponding to each algorithm calculated based on the speed of the moving object in the previous round of algorithm scheduling. This can be referred to in the description of S114 in the following embodiment, and will not be repeated here.

[0096] S107, the object velocity module calculates the velocity of moving objects (such as the main object) in the image frames of the video stream.

[0097] For example, the object speed module can calculate the speed of moving objects (such as the main object) in the image frames of the video stream according to the first scheduling speed in S106A. Here, the first scheduling speed is less than the preview frame rate corresponding to the video stream, which refers to the number of preview frames displayed per second in the preview interface.

[0098] The object velocity module is used to calculate the velocity of moving objects in image frames of a video stream. An image frame may include one or more moving objects. When an image frame includes multiple moving objects, the object velocity module can first identify a main object among the multiple moving objects, and then calculate the velocity of that main object, so that the velocity of the main object can be used as the velocity of the moving objects (such as the main moving object, i.e., the subject object) in that image frame.

[0099] In some embodiments, the object velocity module is a software module of the electronic device, that is, the electronic device can calculate the velocity of moving objects in the image frames of the video stream through some preset algorithms.

[0100] In other embodiments, the object velocity module is a hardware module of the electronic device, such as... Figure 3 or Figure 4The optical flow module shown is an optical flow module. The optical flow module may include an optical flow sensor, which is a high-resolution image sensor used to calculate motion information such as the speed and direction of motion of a moving object based on the video stream captured by a camera.

[0101] S108, the object velocity module returns the velocity of the moving object (such as the main object) to the algorithm scheduling system.

[0102] S109, The algorithm scheduling system determines the multiple algorithms that need to be scheduled at the current moment.

[0103] The number and types of perception algorithms scheduled by the electronic device may differ depending on the shooting mode. After the electronic device enters a certain shooting mode, the number and types of perception algorithms scheduled may also differ in response to different user operations. Therefore, before calculating the frame rate corresponding to each algorithm, the algorithm scheduling system needs to determine the specific algorithm that needs to be scheduled at the current moment. In some embodiments, the algorithm scheduling system can read the algorithms that need to be scheduled at the current moment from the algorithm decision module of the camera service. This algorithm decision module determines the multiple algorithms that need to be scheduled at the current moment based on factors such as the shooting mode and user operations.

[0104] In this embodiment, the algorithm scheduling system can schedule object velocity modules at a certain scheduling speed to schedule perception-based algorithms. Specifically, in each round of algorithm scheduling, the system can acquire the speed of moving objects in the image frames of the video stream from the object velocity module; then, based on the speed of the moving objects, it calculates multiple algorithms that need to be scheduled at the current moment; and then, it sets the frame rate for the interface of each algorithm integration side to complete the algorithm scheduling. As mentioned above, the algorithms to be scheduled may change in each round of algorithm scheduling; therefore, the number and types of algorithms to be scheduled at the current moment need to be re-determined in each round. That is, "current moment" refers to the period between the end of the previous round of algorithm scheduling and the start of the next round.

[0105] S110, the algorithm scheduling system calculates the frame rate corresponding to each of the various algorithms that need to be scheduled at the current moment based on the speed of the moving object (such as the main object) obtained through S108.

[0106] In some embodiments, the electronic device pre-stores a scheduling data table, which includes the optimal frame rates (also called preset frame rates) for N algorithms at M preset speeds. The "N algorithms" are statistically derived perceptual algorithms used for processing video streams, including but not limited to face detection algorithms, object tracking algorithms, single-target object tracking algorithms, human body tracking algorithms, face tracking algorithms, portrait exposure parameter adjustment algorithms, scene detection algorithms, and keypoint detection algorithms. The "M preset speeds" are multiple representative motion speeds set according to usage requirements, where M is an integer greater than or equal to 2. The "optimal frame rates for the N algorithms at the M preset speeds" are the optimal frame rates (also called offline frame rates) for moving objects measured offline at the M preset speeds, forming prior knowledge. The optimal frame rate for an algorithm at a certain speed can be understood as: at a certain speed, when this algorithm uses the optimal frame rate, the power consumption of the electronic device is less than when using other frame rates. For example, a neural network can be introduced, using the speed of the moving object as input, and the frame rate with the lowest power consumption can be selected as the optimal frame rate through training.

[0107] It is understood that the types and number of algorithms included in the scheduling data table, as well as the number and values ​​of motion speeds corresponding to each algorithm, can be adjusted according to factors such as the model of the electronic product, and this application does not impose specific limitations.

[0108] For example, Table 1 shows a scheduling data table. This scheduling data table includes the optimal frame rates for Algorithm 1, Algorithm 2...Algorithm 8 at low, medium, and high speeds. Low speed is represented by s1, medium speed by s2, and high speed by s3. For example, s1 = 1 m / s, s2 = 5 m / s, and s3 = 10 m / s.

[0109] Table 1

[0110]

[0111] Table 2 illustrates another scheduling data table. This table includes the optimal frame rates for algorithms a, b, ... h at stationary, low, medium, and high speeds. Stationary is represented by s1', low speed by s2', medium speed by s3', and high speed by s4'. For example, s1' = 0 m / s, s2' = 4 m / s, s3' = 8 m / s, and s4' = 13 m / s.

[0112] Table 2

[0113]

[0114] Thus, after obtaining the velocity of the moving object (such as the main object) through S108 and the various algorithms that need to be scheduled at the current moment through S109, the algorithm scheduling system can calculate the frame rate (also known as the actual frame rate) corresponding to each of the various algorithms based on the velocity of the moving object (such as the main object) and the pre-stored scheduling data table, according to the interpolation algorithm. The interpolation algorithm can be a linear interpolation algorithm, a velocity frame interpolation algorithm, or other interpolation algorithms, etc.

[0115] Based on the scheduling data table provided in Table 1 above, the following explanation will be based on the example of calculating the actual frame rate of each of the various algorithms that need to be scheduled at the current time using the linear interpolation algorithm.

[0116] For each of the multiple algorithms that need to be scheduled at the current moment, obtained through S109, the preset speeds corresponding to each algorithm are arranged in ascending or descending order. Based on this, the algorithm scheduling system can perform the following steps: determine the relationship between the speed of the moving object (referred to as the first speed) obtained through S108 and the various preset speeds corresponding to an algorithm; if the first speed is greater than the first preset speed and less than the second preset speed, then based on the linear interpolation algorithm, using the first speed, the first preset speed, and the second preset speed, determine the frame rate corresponding to the first speed. The first preset speed and the second preset speed are all preset speeds corresponding to an algorithm. Two adjacent preset speeds are selected, meaning there are no other preset speeds within the speed range from the first preset speed to the second preset speed. If the first speed equals the third preset speed, the frame rate corresponding to the third preset speed under an algorithm is used as the frame rate corresponding to the first speed. If the first speed is greater than the maximum preset speed, the best frame rate corresponding to the maximum preset speed under an algorithm is used as the frame rate corresponding to the first speed. If the first speed is less than the minimum preset speed, the best frame rate corresponding to the minimum preset speed under an algorithm is used as the frame rate corresponding to the first speed.

[0117] Wherein, if the first speed is greater than the first preset speed and the first speed is less than the second preset speed, then based on the linear interpolation algorithm, the frame rate corresponding to the first speed is determined using the first speed, the first preset speed and the second preset speed, including calculating the frame rate corresponding to the first speed through the following relationship (1) or relationship (2):

[0118]

[0119] in, The floor symbol is used for rounding down. The rounding sign is S, which represents the first speed, F, which represents the frame rate corresponding to the first speed, Sa, which represents the first preset speed, Sb, which represents the second preset speed, Fa, which represents the frame rate corresponding to the first preset speed, and Fb, which represents the frame rate corresponding to the second preset speed.

[0120] As shown in Table 3, taking the velocity of the moving object obtained through S108 as S = 2 m / s, and the various algorithms to be scheduled at the current moment, including Algorithm 1, Algorithm 2, Algorithm 3, and Algorithm 4, obtained through S109 as an example. For Algorithm 1, Sa = 1 m / s, Sb = 5 m / s, Fa = 6 FPS, Fb = 15 FPS. Substituting these parameters into the above relation (1), we can obtain F = 8 FPS. For Algorithm 2, Sa = 1 m / s, Sb = 5 m / s, Fa = 4 FPS, Fb = 10 FPS. Substituting these parameters into the above relation (1), we can obtain F = 5 FPS. For Algorithm 3, Sa = 1 m / s, Sb = 5 m / s, Fa = 2 FPS, Fb = 5 FPS. Substituting these parameters into the above relation (1), we can obtain F = 2 FPS. For Algorithm 4, Sa = 1 m / s, Sb = 5 m / s, Fa = 1 FPS, Fb = 3 FPS, substituting these parameters into the above relation (1) yields F = 1 FPS.

[0121] Table 3

[0122]

[0123] As shown in Table 4, taking the velocity of the moving object obtained through S108 as S = 7 m / s, and the various algorithms to be scheduled at the current moment, including Algorithm 1, Algorithm 2, Algorithm 3, and Algorithm 4, obtained through S109, as an example: For Algorithm 1, Sa = 5 m / s, Sb = 10 m / s, Fa = 15 FPS, Fb = 30 FPS. Substituting these parameters into the above relation (1), we can obtain F = 21 FPS. For Algorithm 2, Sa = 5 m / s, Sb = 10 m / s, Fa = 10 FPS, Fb = 20 FPS. Substituting these parameters into the above relation (1), we can obtain F = 14 FPS. For Algorithm 3, Sa = 5 m / s, Sb = 10 m / s, Fa = 5 FPS, Fb = 10 FPS. Substituting these parameters into the above relation (1), we can obtain F = 7 FPS. For Algorithm 4, Sa = 5 m / s, Sb = 10 m / s, Fa = 3 FPS, Fb = 5 FPS, substituting these parameters into the above relation (1) yields F = 3 FPS.

[0124] Table 4

[0125]

[0126] As shown in Table 5, taking the velocity of the moving object obtained through S108 as S = 12 m / s, and the various algorithms that need to be scheduled at the current moment, including Algorithm 1, Algorithm 2, Algorithm 3, and Algorithm 4, obtained through S109 as an example, since the velocity of the moving object 12 m / s is greater than the maximum preset velocity of Algorithm 1, Algorithm 2, Algorithm 3, and Algorithm 4 10 m / s, for each algorithm that needs to be scheduled, the frame rate corresponding to the maximum preset velocity of each algorithm can be used as the frame rate corresponding to the velocity of the moving object.

[0127] Table 5

[0128]

[0129]

[0130] Referring to the description of S106 in the above embodiment, the algorithm scheduling system can request the speed of the moving object from the object speed module according to the first scheduling speed. The first scheduling speed is associated with the speed of the moving object obtained by the object speed module in the previous round of algorithm scheduling. That is, in each round of algorithm scheduling, the algorithm scheduling system needs to redetermine the first scheduling speed to facilitate the next round of algorithm scheduling. Based on this, as... Figure 6 As shown, after calculating the frame rate corresponding to each of the multiple algorithms that need to be scheduled at the current time through S110, the method may further include: S114, redetermining the first scheduling speed based on the maximum frame rate among the frame rates corresponding to each of the multiple algorithms that need to be scheduled at the current time.

[0131] Taking Table 3 above as an example, when the speed of the moving object is S = 2 m / s, the frame rate obtained through linear interpolation is 8 FPS for Algorithm 1, 5 FPS for Algorithm 2, 2 FPS for Algorithm 3, and 1 FPS for Algorithm 4. The final redefined first scheduling speed is 8 FPS. In the next round of algorithm scheduling, the algorithm scheduling system can request the speed of the moving object from the object speed module at a frame rate of 8 FPS, that is, schedule the object speed module once every 125 ms.

[0132] Taking Table 4 above as an example, when the speed of the moving object is S = 7 m / s, the frame rate obtained through linear interpolation is 21 FPS for Algorithm 1, 14 FPS for Algorithm 2, 7 FPS for Algorithm 3, and 3 FPS for Algorithm 4. The final redefined first scheduling speed is 21 FPS. In the next round of algorithm scheduling, the algorithm scheduling system can request the speed of the moving object from the object speed module at a frame rate of 21 FPS, that is, schedule the object speed module once every 47.6 ms.

[0133] Taking Table 5 above as an example, when the speed of the moving object is S = 12 m / s, the frame rate obtained through linear interpolation is 30 FPS for Algorithm 1, 20 FPS for Algorithm 2, 10 FPS for Algorithm 3, and 5 FPS for Algorithm 4. The final redefined first scheduling speed is 30 FPS. In the next round of algorithm scheduling, the algorithm scheduling system can request the speed of the moving object from the object speed module at a frame rate of 30 FPS, that is, schedule the object speed module once every 33.3 ms.

[0134] It should be noted that the speed of a moving object typically changes gradually, and the speed of the moving object in the image frame calculated by the object speed module also gradually increases or decreases. Thus, in multi-round algorithm scheduling, the frame rate corresponding to each algorithm calculated by the algorithm scheduling system based on the speed of the moving object will also conform to a gradual change pattern. For example, when the moving object changes from slow motion to stillness, the frame rate corresponding to a certain algorithm changes sequentially to 30 FPS, 25 FPS, 20 FPS, 10 FPS, and 5 FPS, achieving a gradual decrease in frame rate. Similarly, when the moving object changes from stillness to slow motion, the frame rate corresponding to a certain algorithm changes sequentially to 5 FPS, 10 FPS, 20 FPS, 25 FPS, and 30 FPS, achieving a gradual increase in frame rate. It can be understood that when the moving object in the image is essentially still, the algorithm scheduling method provided in this application embodiment offers significant performance and power consumption optimization; gradual frame rate adjustment can maximize both performance and power consumption reduction.

[0135] S111, the algorithm scheduling system sends the algorithm scheduling scheme to the algorithm scheduling module.

[0136] The aforementioned algorithm scheduling scheme may include: multiple algorithms that need to be scheduled at the current moment, and the frame rate corresponding to each algorithm that needs to be scheduled at the current moment. Specifically, the multiple algorithms that need to be scheduled at the current moment can be determined through S109, and the frame rate corresponding to each algorithm that needs to be scheduled at the current moment can be determined through S110.

[0137] S112, the algorithm scheduling module calls the corresponding algorithm interface according to the algorithm scheduling scheme, and sends the corresponding frame rate to each algorithm interface.

[0138] In this embodiment, each algorithm corresponds to a different algorithm interface. After determining the multiple algorithms that need to be scheduled at the current moment through S109, the algorithm scheduling system can call the algorithm interfaces corresponding to these multiple algorithms through the algorithm scheduling module, and send the corresponding frame rate to each of the multiple algorithm interfaces. In this way, each algorithm can process the image frames in the video stream according to its corresponding frame rate. It should be noted that the video stream here can be a RAW format video stream captured by a camera, or a video stream in other formats (such as YUV or RGB format) after format conversion.

[0139] For example, Figure 7 This diagram illustrates how various algorithms process image frames in a video stream according to their respective frame rates. For example... Figure 7 As shown, the frame rate of the electronic device system is T. The frame rate corresponding to Algorithm 0 is T / 2, the frame rate corresponding to Algorithm 1 is T / 3, and the frame rate corresponding to Algorithm 2 is T / 4. According to the acquisition order of image frames in the video stream, the electronic device assigns a frame number to each image frame. This frame number is equivalent to a timestamp, used to identify the acquisition order of the image frames. For example, the electronic device acquires image frames M1, M2, M3, M4, M5, M6, M7, M8, and M9 in the order from front to back. Algorithm 0 processes image frames M1, M3, M5, M7, and M9 at a frame rate of T / 2; that is, image frames M1, M3, M5, M7, and M9 are scheduled using Algorithm 0. Algorithm 1 processes image frames M1, M4, and M7 at a frame rate of T / 3; that is, image frames M1, M4, and M7 are scheduled using Algorithm 1. Algorithm 2 processes image frames M1, M5, and M9 according to a frame rate of T / 4, i.e., scheduling algorithm 2 for image frames M1, M5, and M9.

[0140] S113, the algorithm scheduling module deploys each of the multiple algorithms to its corresponding processor.

[0141] It should be noted that S113 is an optional step. If the corresponding algorithm has been deployed to the corresponding processor before this round of algorithm scheduling, then in this round of algorithm scheduling, the algorithm scheduling system only needs to schedule the corresponding algorithm interface to issue the frame rate corresponding to this algorithm through the algorithm scheduling module.

[0142] The processor mentioned above can be as follows: Figure 4The CPU, DSP's NPU, GPU, or other processors are shown. Depending on the specific function each algorithm performs, different algorithms may be deployed on different processor platforms. For example, a portrait exposure parameter adjustment algorithm might require deploying part of the algorithm on the CPU and another part (such as model inference) on the DSP's NPU. The NPU's AI model can perform model inference, thus enabling the adjustment of exposure parameters.

[0143] In the above-described solution provided in this application embodiment, by constructing an algorithm scheduling system based on the speed of moving objects, after acquiring the video stream, the system can calculate the frame rate corresponding to each of the various algorithms that need to be scheduled at the current moment based on the speed of moving objects (such as the main subject) in the image frames of the video stream, and then set the frame rate for the interface on the integration side of each of the various algorithms. In this way, when the speed of the moving object changes, the algorithm scheduling system can dynamically adjust the frame rate corresponding to each algorithm. For example, it can reduce the frame rate corresponding to the algorithm when the moving object is stationary or moving at low speed, and increase the frame rate corresponding to the algorithm when the moving object is moving at medium to high speed. In particular, when system resources are congested and the image content remains largely unchanged or changes only slightly, this solution can optimize the performance of electronic devices and reduce power consumption.

[0144] This application also provides a terminal device, including a processor coupled to a memory, the processor being used to execute computer programs or instructions stored in the memory, so that the terminal device implements the methods in the above embodiments.

[0145] This application also provides a computer-readable storage medium storing computer instructions. When the computer instructions are executed on a computer, the computer performs the method described above. The computer instructions can be stored in the computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium accessible to a computer or can include one or more data storage devices such as servers or data centers that can be integrated with the medium. Available media can be magnetic media (e.g., floppy disks, hard disks, or magnetic tapes), optical media, or semiconductor media (e.g., solid-state disks (SSDs)).

[0146] This application also provides a computer program product, which includes computer program code that, when run on a computer, causes the computer to perform the methods described in the above embodiments.

[0147] This application also provides a chip coupled to a memory. This chip is used to read and execute computer programs or instructions stored in the memory to perform the methods described in the above embodiments. The chip can be a general-purpose processor or a special-purpose processor. It should be noted that the chip can be implemented using one or more field-programmable gate arrays (FPGAs), programmable logic devices (PLDs), controllers, state machines, gate logic, discrete hardware components, any other suitable circuits, or any combination of circuits capable of performing the various functions described throughout this application.

[0148] The terminal device, computer-readable storage medium, computer program product, and chip provided in the embodiments of this application are all used to execute the methods provided above. Therefore, the beneficial effects they can achieve can be referred to the beneficial effects corresponding to the methods provided above, and will not be repeated here.

[0149] Through the above description of the embodiments, those skilled in the art can clearly understand that, for the sake of convenience and brevity, only the division of the above functional modules is used as an example. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above.

[0150] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another apparatus, or some features may be ignored or not executed. Furthermore, the mutual coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0151] The units described as separate components may or may not be physically separate. A component shown as a unit can be one or more physical units; that is, it can be located in one place or distributed in multiple different locations. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0152] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0153] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a readable storage medium. Based on this understanding, the technical solutions of the embodiments of this application, in essence, or the parts that contribute to the prior art, or all or part of the technical solutions, can be embodied in the form of a software product. This software product is stored in a storage medium and includes several instructions to cause a device (which may be a microcontroller, chip, etc.) or processor to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0154] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any changes or substitutions within the technical scope disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. An algorithm scheduling method, characterized by, The method comprises: acquiring a video stream collected by a camera; determining a frame rate corresponding to each of a plurality of algorithms to be scheduled at a current time based on a speed of a moving object in an image frame of the video stream; scheduling each of the algorithms to process the image frame of the video stream according to the frame rate corresponding to the algorithm; wherein the frame rate corresponding to each of the algorithms represents a number of times the algorithm is scheduled within a preset time length.

2. The method of claim 1, wherein, The determination of the frame rate corresponding to each of the algorithms to be scheduled at the current time based on the speed of the moving object in the image frame of the video stream comprises: acquiring the speed of the moving object in the image frame of the video stream; acquiring preset frame rates corresponding to each of the algorithms at M preset speeds, M being an integer greater than or equal to 2; determining the frame rate corresponding to each of the algorithms based on the speed of the moving object and the preset frame rates corresponding to each of the algorithms at the M preset speeds.

3. The method of claim 2, wherein, The determination of the frame rate corresponding to each of the algorithms based on the speed of the moving object and the preset frame rates corresponding to each of the algorithms at the M preset speeds comprises: for each of the algorithms, performing the following steps: judging a size relationship between the speed of the moving object and the M preset speeds corresponding to the algorithm; if the speed of the moving object is greater than a first preset speed among the M preset speeds corresponding to the algorithm and the speed of the moving object is less than a second preset speed among the M preset speeds corresponding to the algorithm, determining the frame rate corresponding to the algorithm according to the speed of the moving object, a preset frame rate corresponding to the first preset speed and a preset frame rate corresponding to the second preset speed; if the speed of the moving object is equal to a third preset speed among the M preset speeds corresponding to the algorithm, taking a preset frame rate corresponding to the third preset speed under the algorithm as the frame rate corresponding to the algorithm; if the speed of the moving object is greater than a maximum preset speed among the M preset speeds corresponding to the algorithm, taking a preset frame rate corresponding to the maximum preset speed under the algorithm as the frame rate corresponding to the algorithm; if the speed of the moving object is less than a minimum preset speed among the M preset speeds corresponding to the algorithm, taking a preset frame rate corresponding to the minimum preset speed under the algorithm as the frame rate corresponding to the algorithm.

4. The method of claim 3, wherein, The determination of the frame rate corresponding to the algorithm according to the speed of the moving object, the preset frame rate corresponding to the first preset speed and the preset frame rate corresponding to the second preset speed comprises: determining the frame rate corresponding to the algorithm through a first relationship or a second relationship; the first relationship is: the second relationship is: wherein, is a floor symbol, is a ceiling symbol, S represents a speed of the moving object, F represents a frame rate corresponding to the one algorithm, Sa represents the first preset speed, Sb represents the second preset speed, Fa represents a preset frame rate corresponding to the first preset speed, and Fb represents a preset frame rate corresponding to the second preset speed.

5. The method according to claim 3 or 4, characterized in that, the M preset speeds corresponding to the algorithm are arranged in ascending order, and the first preset speed and the second preset speed are two adjacent preset speeds among the M preset speeds corresponding to the algorithm.

6. The method according to any one of claims 2 to 5, characterized in that, The preset frame rates corresponding to the M preset speeds of each algorithm are obtained. The preset frame rates corresponding to the M preset speeds of each algorithm are obtained by consulting a scheduling data table. The scheduling data table includes preset frame rates corresponding to the M preset speeds of N algorithms, and the N algorithms include the multiple algorithms.

7. The method according to any one of claims 1 to 6, characterized in that, Before determining the frame rate corresponding to each algorithm in the multiple algorithms to be scheduled at the current time based on the speed of the moving object in the image frame of the video stream, the method further includes: According to the first scheduling speed, the speed of the moving object in the image frame of the video stream is obtained.

8. The method of claim 7, wherein, After determining the frame rate corresponding to each algorithm in the multiple algorithms to be scheduled at the current time based on the speed of the moving object in the image frame of the video stream, the method further includes: Based on the maximum frame rate in the frame rates corresponding to each algorithm in the multiple algorithms, the first scheduling speed is re-determined.

9. The method according to any one of claims 1 to 8, characterized in that, The speed of the moving object in the image frame of the video stream is obtained by an optical flow module or a software module of an electronic device.

10. An electronic device, comprising: The electronic device includes one or more processors and a memory. The memory is coupled to the one or more processors, and the memory is configured to store computer program codes including computer instructions, and the one or more processors are configured to invoke the computer instructions to enable the electronic device to perform the method of any one of claims 1 to 9.

11. A chip system, characterized by The chip system is applied to an electronic device, and the chip system includes one or more processors configured to invoke computer instructions to enable the electronic device to perform the method of any one of claims 1 to 9.

12. A computer-readable storage medium, characterized in that, The computer readable storage medium includes instructions configured to enable an electronic device to perform the method of any one of claims 1 to 9 when the instructions are executed on the electronic device.

Images