Processing circuitry of video stream data and electronic device

Abstract

This application discloses a video stream data processing circuit and electronic device. The circuit includes: multiple data interfaces acquiring multiple video stream data based on a preset processing unit; a video input processing unit performing first data processing on the multiple video stream data to determine the data source of the multiple video stream data and writing the multiple video stream data into different address segments of a first memory; a scheduler writing or reading the multiple video stream data based on a preset processing order; an image signal processor processing the current video stream data through current context data and configuring other context data to a non-working state; after the current video stream data processing is completed, switching to other context data to continue processing other video stream data; and a second memory storing context data in a waiting state at a first address and context data in a working state at a second address. This application has the technical effect of improving the processing efficiency of multiple video streams and reducing hardware resource costs.

Description

[0001] This application is a divisional application of the following parent case: Parent application number: 202511657368.X; The parent application was filed on November 13, 2025. The parent application is titled: "Method, circuit, apparatus, and electronic device for processing video stream data." Technical Field

[0002] This application relates to the field of video data processing technology, and in particular to a video stream data processing circuit and electronic device. Background Technology

[0003] With the rapid development of various electronic products, the integration of multiple cameras into a single electronic device has become widespread. When acquiring image information through a camera, the light signal is often converted into an electrical signal by a sensor chip and then sent to a video processing chip via a standard interface, such as the Mobile Industry Processor Interface (MIPI). The image signal processor (ISP) inside the chip then digitizes the data. Each video stream requires an independent ISP module to ensure continuous processing and low latency. However, image signal processors typically process images frame by frame, requiring the acquisition of the entire frame, resulting in high latency. Furthermore, inputting multiple video streams necessitates increasing the number of ISP modules, thereby increasing hardware resource costs. Therefore, the processing efficiency and hardware resource cost of multiple video streams are issues that warrant attention. Summary of the Invention

[0004] In view of this, embodiments of this disclosure provide a video stream data processing circuit and electronic device to improve the processing efficiency of multiple video streams and reduce hardware resource costs. A first aspect provides a video stream data processing method, comprising: acquiring multiple video stream data based on a preset processing unit, wherein the multiple video stream data originate from different image sensors, and the preset processing unit is determined based on a pixel array in a frame image; performing image processing on the multiple video stream data based on a preset processing order and multiple context data, including: when processing the current video stream data using the current context data, other context data are switched to a non-working state; after the current video stream data processing is completed, switching to other context data to continue processing other video stream data, and the current context data being switched to a non-working state; wherein the context data corresponds to the processed video stream data, and the non-working state includes: a waiting state or a storage state; and outputting data when the video stream data is completed based on the preset processing order.

[0005] In the above video stream data processing method, by switching between multiple context data, the data processing of each video stream can be correctly matched and maintain continuity, thereby enabling alternating processing of multiple video streams on a single image signal processor. Using preset processing units for video stream data acquisition allows for finer-grained scheduling of video stream processing, reducing processing wait time and achieving lower latency. Alternating processing of multiple video streams can be achieved on a single processor, eliminating the need for separate processing modules for each video stream, reducing hardware resource consumption, and lowering chip area and cost. This method can flexibly adjust the processing units and processing order according to different application scenarios and can adapt to dual, triple, or more video streams, exhibiting good scalability.

[0006] Optionally, the multiple video stream data includes first video stream data and second video stream data, and the multiple context data includes: first context data and second context data, wherein the first context data corresponds to the first video stream data and the second context data corresponds to the second video stream data; when processing the first video stream data, the first context data is switched to a working state and the second context data is switched to a waiting state; when processing the second video stream data, the first context data is switched to a waiting state and the second context data is switched to a working state.

[0007] Optionally, the multiple video stream data includes first video stream data, second video stream data, and third video stream data, and the multiple context data includes: first context data, second context data, and third context data, wherein the first context data corresponds to the first video stream data, the second context data corresponds to the second video stream data, and the third context data corresponds to the third video stream data; when processing the first video stream data, the first context data is switched to the working state, the second context data is switched to the waiting state, and the third context data is switched to the storage state; when processing the second video stream data, the first context data is switched to the storage state, the second context data is switched to the working state, and the third context data is switched to the waiting state; when processing the third video stream data, the first context data is switched to the waiting state, the second context data is switched to the storage state, and the third context data is switched to the working state.

[0008] Optionally, the preset processing unit is configured as at least one row of pixels in a frame of an image, or the preset processing unit is configured as at least one column of pixels in a frame of an image.

[0009] In a second aspect, a video stream data processing circuit is provided, comprising: multiple data interfaces configured to acquire multiple video stream data based on a preset processing unit, wherein the multiple data interfaces are respectively coupled to multiple image sensors, and the preset processing unit is determined based on a pixel array in a frame image; a video input processing unit coupled to the multiple data interfaces, configured to perform a first data processing on the multiple video stream data, the first data processing including: determining the data source of the multiple video stream data and writing the multiple video stream data into a first memory; a scheduler coupled to the video input processing unit, configured to write or read the multiple video stream data based on a preset processing order; and an image signal processor coupled to the scheduler, configured to perform image processing on the multiple video stream data based on multiple context data through the scheduler, including: when processing the current video stream data using the current context data, other context data are switched to a non-working state; after the current video stream data processing is completed, switching to other context data to continue processing other video stream data, the current context data being switched to a non-working state; and data output when the video stream data is completed based on the preset processing order, wherein the context data corresponds to the processed video stream data, and the non-working state includes: a waiting state or a storage state.

[0010] Optionally, the context data of the waiting state is configured to be stored at a first address in the second memory, and the context data of the working state is configured to be run at a second address in the second memory, wherein the second memory is the on-chip buffer of the image signal processor, and the first memory is the off-chip memory of the image signal processor.

[0011] Optionally, the context data of the stored state is configured to be stored in a first memory.

[0012] Optionally, multiple context data are configured to be pre-stored in a first memory outside the image signal processor; or, multiple context data are configured to be stored in a third memory within the image signal processor; the image signal processor is also configured to read or write multiple context data from the first memory or the third memory via a scheduler.

[0013] Thirdly, a video stream data processing apparatus is provided, comprising: an acquisition unit for acquiring multiple video stream data based on a preset processing unit, wherein the multiple video stream data originate from different image sensors, and the preset processing unit is determined based on a pixel array in a frame image; an image signal processor for performing image processing on the multiple video stream data based on a preset processing order and multiple context data, including: when processing the current video stream data using the current context data, other context data are switched to a non-working state; after the current video stream data processing is completed, switching to other context data to continue processing other video stream data, and the current context data being switched to a non-working state; wherein the context data corresponds to the processed video stream data, and the non-working state includes: a waiting state or a storage state; and an output unit for outputting data when the video stream data is completed based on the preset processing order.

[0014] Fourthly, an electronic device is provided, including the video stream data processing circuit provided in the second aspect. Attached Figure Description

[0015] The accompanying drawings used in the description of the embodiments of this disclosure are briefly introduced below: Figure 1 A schematic diagram of an image acquisition and processing flow provided in some embodiments of this application is shown; Figure 2 A flowchart illustrating a video stream data processing method provided in some embodiments of this application is shown. Figure 3 The diagram shows a schematic of the structure of a video stream data processing circuit provided in some embodiments of this application; Figure 4 This invention illustrates a schematic diagram of the structure of another video stream data processing circuit provided in some embodiments of this application; Figure 5 This paper shows a schematic diagram of the structure of another video stream data processing circuit provided in some embodiments of this application; Figure 6 A schematic diagram of the structure of a video stream data processing apparatus provided in some embodiments of this application is shown. Detailed Implementation

[0016] To more clearly illustrate the technical solutions in the embodiments of this disclosure, examples of implementation methods of this disclosure will be described below with reference to the accompanying drawings. The accompanying drawings described below are merely some embodiments of this disclosure. For those skilled in the art, other drawings and other implementation methods can be obtained based on these drawings without creative effort. Adjustments and improvements made without departing from the concept of this disclosure are all within the protection scope of this disclosure.

[0017] To keep the drawings simple, each figure only schematically shows the parts relevant to the embodiment, and they do not represent the actual structure of the product. In addition, for the sake of clarity and ease of understanding, some figures only schematically show parts of components with the same structure or function, and there may actually be more or fewer components with the same structure or function.

[0018] In this disclosure, unless otherwise expressly specified and limited, ordinal numbers, such as “first,” “second,” etc., are used only to distinguish and describe related objects and should not be construed as indicating or implying the relative importance or order between related objects; furthermore, they do not represent the quantity of related objects. “Multiple” includes two or more, and other quantifiers are similar. “ / ” is used to describe the relationship between related objects, indicating an “or” relationship between them. “And / or” is used to describe the relationship between related objects, including any combination relationship between them, such as “a and / or b” including: “a alone,” “b alone,” or “a and b.” “One or more” or “at least one” of multiple objects refers to any object or any combination of multiple objects, such as “one or more of a1, a2, a3” or “at least one of a1, a2, a3” including: “a1 alone,” “a2 alone,” “a3 alone,” “a1 and a2,” “a1 and a3,” “a2 and a3,” or “a1, a2 and a3.”

[0019] With the rapid development of various electronic products, the integration of multiple cameras into a single electronic device has become widely used. When acquiring image information through a camera, the light signal is often converted into an electrical signal by a sensor chip and sent to a video processing chip via a standard interface, such as a mobile industrial processor interface. The image signal processor inside the chip then digitizes the data. Each video stream requires an independent ISP module to ensure continuous processing and low latency. Figure 1The diagram illustrates an image acquisition and processing flow provided in some embodiments of this application. In this flow, the lens of the image sensor receives the light signal from the target scene, which is converted into an electrical signal by the photoelectric conversion of the image sensor chip and then sent to the video processing system chip. The chip typically integrates video input (VI) processing to interface with the image sensor, receiving and processing the input video data. An image signal processor (ISP) performs digital processing on the raw image signal (e.g., black level correction, automatic white balance, noise reduction, edge removal, etc.) to obtain a usable, time-correct image. Video output (VO) processing outputs the processed video signal to a display. Alternatively, a video encoder (VENC) may be included to compress and encode the video data for transmission or storage via network structures such as Ethernet, forming a time-sequential video stream through these links. In this process, the image sensor forms a frame of data line by line through raster scanning, with common resolutions such as 1920×1080 or 3840×2160, and typical frame rates of 25, 30, 50, or 60 frames per second. When the data enters the video processing system chip, there are two common processing granularities: one is frame-by-frame processing, where the image signal processor only begins processing after an entire frame of data has been received; the other is line-by-line processing, where processing is triggered once a certain number of line blocks have accumulated, without waiting for the entire frame to arrive. The difference lies in the end-to-end latency. In frame-by-frame mode, the image signal processor must wait at least "one frame's time" before starting processing; for example, at 30fps, this is approximately 33ms. Line-by-line processing, on the other hand, alleviates this longer wait by processing segments within a frame. When multiple video streams (acquired by multiple image sensors) are simultaneously accessed, each video stream is often equipped with its own independent image signal processor to prevent interference between different video streams and to easily guarantee the latency target. However, its hardware resource consumption is extremely high, and the cost increases linearly or superlinearly with the number of channels. When supporting 4 or more channels, the solution of multiple image signal processors is equivalent to multiple hardware sets, which often faces difficulties in terms of miniaturizing the integrated circuit area and saving production costs. Therefore, this application provides a method, circuit, device, and electronic device for processing video stream data. It acquires multiple video stream data through a preset processing unit, which helps to improve the processing efficiency of video stream data in subsequent processing. Based on the preset processing order and the context data corresponding to the video stream data, the video stream data processing circuit can reuse the same image signal processor, thereby reducing hardware resource costs.

[0020] The following description is in conjunction with the accompanying drawings: Figure 2The illustration shows a flowchart of a video stream data processing method provided in some embodiments of this application. The video stream data processing method includes at least the following steps: S210: Acquire multiple video stream data based on a preset processing unit, wherein the multiple video stream data come from different image sensors, and the preset processing unit is determined based on the pixel array in a frame image; S220: Perform image processing on multiple video stream data based on a preset processing order and multiple context data, including: when processing the current video stream data through the current context data, other context data are switched to a non-working state; S230: After the current video stream data is processed, the system switches to other context data to continue processing other video stream data. The current context data is switched to a non-working state. The context data corresponds to the processed video stream data. The non-working states include: waiting state or storage state. S240: Output data when the video stream data is completed based on a preset processing order.

[0021] In the embodiments of the above video stream data processing method, multiple video stream data originate from different image sensors. For example, different image sensors can be sensors mounted at different locations on vehicles, aircraft, etc., such as the front, rear, left, and right image sensors of a vehicle, or multiple cameras on a mobile phone. Each image sensor converts the light signal from the external target into an electrical signal, and then transmits it to the video processing system chip. The preset processing unit can be determined based on the pixel array in a frame of an image. For example, a frame of a video image includes multiple pixels, so a processing unit can be a partial row, a partial column, a row, a column, multiple rows, multiple columns, or a matrix block; no specific limitation is made here.

[0022] In image signal processing, the algorithm requires two types of data: input video data, which is the real-time, continuously updated data from the image sensor, and algorithm configuration data and intermediate result data. This latter part consists of parameters configured during algorithm implementation or intermediate results generated during algorithm calculations—the context data. This context data is continuously used during processing and cannot be lost; its loss will cause errors in processing subsequent input data. If there is only one video stream, the context data will continue to be used for that single input video frame without issue. However, problems may arise when processing multiple video streams simultaneously. Figure 1For example, if there are two video stream inputs, after being received by the video input processing unit (VI), and if there are two independent image signal processors (ISPs) internally, each with its own independent resources and maintaining its own independent context data, then they can be processed simultaneously according to the aforementioned preset processing unit, achieving low-latency processing of the two video streams. However, this technical solution wastes a lot of hardware resources, significantly increasing hardware costs. When we need to support more video streams simultaneously, such as four or more, simply increasing the number of ISPs to achieve low-latency processing for each stream becomes unacceptable in terms of hardware resources. In this application, a preset processing order can be used to select one video stream for processing and call the context data corresponding to that video stream. While the current context data is in an active state, other context data are switched to a non-active state, such as a waiting state or a stored state, to ensure that data processing between different video streams does not interfere with each other. The preset processing order can be sequential, such as processing each video stream individually when there are two or more video streams, or it can be cyclical, where the first video stream is processed after the last one is processed, thus completing several processing loops. Alternatively, the processing order can be adjusted according to the required progress; for example, when there are four video streams, the third, second, first, and fourth video streams can be pre-programmed. No specific restrictions are placed here. In an Image Signal Processor (ISP), the hardware resources involved in the algorithm processing are always active; they are stateless. The hardware resources for context data, however, can be stateful. Context data in an active state can be called "active resources," while other context data in a waiting state can be called "shadow resources." Context data in a stored state can be stored in memory and not accessed temporarily. Only the context data in an active state works in conjunction with the algorithm. After processing the current video stream, the process can switch to another video stream, simultaneously switching the context data processing. During the switching process, it is ensured that there is no residual unprocessed data from the previous video stream in the hardware resources used by the algorithm. Because there are multiple completely independent context data sets, the processing of the two video streams can be independent and unaffected by each other, even if their algorithm parameters are completely different. Therefore, when one video stream is processed, the system automatically switches the current context data to a non-working state and activates another context data set to continue processing the corresponding video stream. This cycle continues until all video streams in the preset order have been processed. Finally, when the preset processing order is complete, the system outputs the processed video data.In this embodiment, by switching between multiple context data, the data processing of each video stream can be correctly matched and maintain continuity, thereby enabling alternating processing of multiple video streams on a single image signal processor. The embodiment of this application uses preset processing units to acquire video stream data, allowing video stream processing to be scheduled at a finer granularity, reducing processing wait time and thus achieving lower latency. Alternating processing of multiple video streams can be achieved on a single processor, eliminating the need for a separate processing module for each video stream, reducing hardware resource consumption, and lowering chip area and cost. This method can flexibly adjust the processing units and processing order according to different application scenarios and can adapt to dual, triple, or more video streams, exhibiting good scalability.

[0023] In some embodiments of this application, the multiple video stream data includes first video stream data and second video stream data, and the multiple context data includes: first context data and second context data, wherein the first context data corresponds to the first video stream data and the second context data corresponds to the second video stream data; when processing the first video stream data, the first context data is switched to a working state and the second context data is switched to a waiting state; when processing the second video stream data, the first context data is switched to a waiting state and the second context data is switched to a working state.

[0024] In the above embodiments, the first video stream data can be acquired by the first image sensor, and the second video stream data can be acquired by the second image sensor. They can represent different shooting scenes or angles, respectively. Corresponding to the aforementioned video stream data, first context data and second context data can be configured accordingly. The first context data corresponds to the first video stream data, and the second context data corresponds to the second video stream data, thereby forming parameter configurations and intermediate processing states related to the corresponding video stream processing. During processing, when image processing is performed on the first video stream data according to a preset order, the system switches the first context data to the working state to process the first video stream data. Simultaneously, the second context data is switched to the waiting state, i.e., its existing data and parameters are kept unmodified or uninterrupted. For example, in an image signal processor, the first context data is switched to the active resource to process the first video stream data, and the second context data is switched to the shadow resource. When the first video stream data processing is completed, a corresponding switch is performed, switching to the second video stream data according to a preset order. At this time, the second context data is switched to the working state to ensure that it can match the second video stream data and complete the correct image processing. Simultaneously, the first context data is switched to a waiting state to ensure its processing result remains valid and awaits the next call, thereby enabling the two video streams to reuse a single image signal processor. Through this configuration, the first video stream data and the first context data, as well as the second video stream data and the second context data, can always maintain consistency, avoiding image processing errors caused by context sharing or overwriting, and ensuring the accuracy and stability of image processing.

[0025] In some embodiments of this application, the multiple video stream data includes first video stream data, second video stream data, and third video stream data, and the multiple context data includes: first context data, second context data, and third context data, wherein the first context data corresponds to the first video stream data, the second context data corresponds to the second video stream data, and the third context data corresponds to the third video stream data; when processing the first video stream data, the first context data is switched to a working state, the second context data is switched to a waiting state, and the third context data is switched to a storage state; when processing the second video stream data, the first context data is switched to a storage state, the second context data is switched to a working state, and the third context data is switched to a waiting state; when processing the third video stream data, the first context data is switched to a waiting state, the second context data is switched to a storage state, and the third context data is switched to a working state.

[0026] The three video streams are acquired by three different image sensors, which may correspond to different monitoring areas, shooting angles, or lighting conditions. Corresponding to these video streams, the system is configured with three sets of context data: first context data, second context data, and third context data. Each video stream is bound to its corresponding context data, used to store the algorithm parameter configuration, intermediate processing results, and statistical information for that video stream. During processing, the system can process the three video streams sequentially according to a preset order. When processing the first video stream, the first context data is switched to active mode and combined with the first video stream data to complete image processing; the second context data is switched to a waiting state, becoming a shadow resource awaiting subsequent use; and the third context data is switched to a storage state, temporarily not requiring retrieval from memory. When processing the second video stream, the second context data is switched to active mode for processing; the first context data is switched to storage mode, writing its corresponding context information to the storage medium for saving; and the third context data is switched to a waiting state, becoming a shadow resource awaiting subsequent processing. When processing the third video stream data, the third context data is switched to the working state, cooperating with the third video stream data to complete the processing; the second context data is switched to the storage state, and its data is securely saved; the first context data is switched to the waiting state, and continues to switch as a shadow resource awaiting subsequent use. Through the above switching method, the three sets of context data cycle between "working state, waiting state, and storage state," ensuring that each video stream can be matched with its corresponding context data during processing, realizing the rational allocation of limited hardware resources, that is, the multiplexing of three video stream data for one image signal processor. In the above processing, this application can alleviate the pressure on on-chip resources by introducing a "storage state." For example, context data in the storage state can be temporarily not read into the on-chip memory of the image signal processor, but stored in other memory outside the image signal processor. Through the switching of context data in the three states, the system can not only support at least three video streams, but also flexibly select which contexts to keep on-chip and which contexts to move to storage media when expanding to a larger scale of multiple video streams, thereby meeting diverse application scenarios. The video stream data in this application is not limited to two or three video streams, but can also be four, five, six, or more video streams. Correspondingly, there can also be four, five, six, or more context data. No specific limitation is made here, so that the structure of this application can support the processing of multiple video streams.

[0027] In some embodiments of this application, the preset processing unit is configured as at least one row of pixels in a frame of an image, or the preset processing unit is configured as at least one column of pixels in a frame of an image.

[0028] For example, when the preset processing unit is configured as at least one row of pixels in a frame of image, taking a 1920×1080 resolution image frame at 30 frames per second as an example, if processing is done in "whole frames," each frame must wait 33 milliseconds before entering the image signal processor for processing, resulting in significant latency. In this embodiment, if one row of pixels is used as the processing unit, the image signal processor immediately begins processing when the sensor outputs the first row of pixel data. For example, multiple rows of pixels can also be selected as the processing unit, or it can be a non-complete row; for example, if a row contains 1920 pixels, processing can begin when 1000 pixels are processed. No specific limitation is made here. Once the sensor outputs one row of pixel data, the system can trigger the image signal processor to process that row of data without waiting for the entire frame of data to be acquired. This method effectively reduces processing latency and is suitable for applications with high low-latency requirements. In another embodiment, the preset processing unit is configured as at least one column of pixels in a frame of image. For example, within the same frame, one or more columns of pixels can be selected as the processing unit. Upon receiving the data in this column, the system can perform corresponding processing, enabling rapid analysis of local areas of the image in the horizontal direction and fast processing of specific vertical regions, thus improving flexibility. Through this method, the system can segment image data at the dimensions of row pixels, column pixels, or pixel blocks, depending on the specific application scenario, thereby implementing different latency optimization and processing strategies.

[0029] In existing technologies, when processing video streams frame by frame using an image signal processor, processing of the next frame cannot begin until one frame has elapsed. With N video streams (30 frames / second), each stream requires a wait of (N-1) frames = 33 * (N-1) ms, resulting in significant latency. The solution proposed in this application achieves low latency for each video stream. If processing is done in 1 / 32 lines of a frame (e.g., for a 3840×2160 resolution image, 2160×1 / 32 ≈ 68 lines), the waiting time for each stream is (N-1) / 32 frames = 33 / 32 * (N-1) ms, greatly reducing processing latency. Furthermore, when switching between context data and video stream data, this application can prepare for the next frame while processing the current frame, without negatively impacting the currently processed data. This background preparation can be executed in parallel with the current processing without additional time consumption.

[0030] Figure 3This illustration shows a schematic diagram of a video stream data processing circuit 300 provided in some embodiments of this application. The video stream data processing circuit 300 includes: multiple data interfaces configured to acquire multiple video stream data based on a preset processing unit, wherein the multiple data interfaces are respectively coupled to multiple image sensors, and the preset processing unit is determined based on a pixel array in a frame image; a video input processing unit coupled to the multiple data interfaces, configured to perform a first data processing on the multiple video stream data, the first data processing including: determining the data source of the multiple video stream data and writing the multiple video stream data into a first memory; and a scheduler coupled to the video input processing unit, configured to... To write or read multiple video stream data based on a preset processing order; an image signal processor, coupled to a scheduler, is configured to perform image processing on multiple video stream data based on multiple context data through the scheduler, including: when processing the current video stream data using the current context data, configuring other context data to a non-working state; after the current video stream data processing is completed, switching to other context data to continue processing other video stream data, configuring the current context data to a non-working state; and outputting data when the video stream data is completed based on the preset processing order, wherein the context data corresponds to the processed video stream data, and the non-working states include: a waiting state or a storage state.

[0031] In the above video stream data processing circuit, two image sensors (i.e. Figure 3 Image sensor A and image sensor B respectively send the two video stream data into the video stream data processing circuit through their image sensor chips, via multiple data interfaces (e.g., ... Figure 3The video input processing unit 320 distinguishes video stream data by using virtual channels in data interfaces 310A and 310B. When the video input processing unit 320 receives video stream data, it can distinguish which virtual channel the data comes from, determine which video stream the data belongs to, extract and integrate the respective data, and write it into different address segments of the first memory. The first memory can be a memory integrated into the video stream data processing circuit 300, and the storage type can be SRAM. When the data of a certain video stream accumulates to a certain number of rows, the scheduler 330 can read it out and send it to the image signal processor 340 for processing of the frame data of these pixel rows, columns, or blocks. The scheduler 330 can be preset with a processing order, regularly sending the two video streams to the image signal processor 340 alternately. Furthermore, in the image signal processor 340, taking two video streams as an example, there is one set of hardware resources for algorithm processing and two sets of hardware resources for context data. The hardware resources for algorithm processing are always active and are stateless, while the hardware resources for the other two sets of context data are stateful. The context data in the active state is the active resource, and the context data in the waiting state is the shadow resource. The two sets of context data can be temporarily stored in the on-chip memory of the image signal processor 340. When the scheduler finishes processing a portion of the frame data of one video stream and switches to processing the other video stream, the image signal processor 340 will also correspondingly switch the hardware resources of the two sets of context data. Of course, during the switch, it is necessary to ensure that there is no residual data from the previous stream in the hardware resources for algorithm processing. This switching time is very short, for example, it can be implemented using a MUX selection switch circuit structure, completing the switch within one clock cycle, achieving extremely short processing time. Because there are two completely independent hardware resources for context data, the processing of the two video streams can be independent of each other and do not affect each other, even if the algorithm parameters they process are completely different, there will be no negative effects.

[0032] In some embodiments of this application, the context data of the waiting state is configured to be stored at a first address of the second memory, and the context data of the working state is configured to be run at a second address of the second memory, wherein the second memory is the on-chip memory of the image signal processor, and the first memory is the off-chip memory of the image signal processor.

[0033] Figure 4This diagram illustrates a structural schematic of another video stream data processing circuit provided in some embodiments of this application. When more video stream data is input, compared to a two-stream solution, the video input processing unit 320 can store the data received from each video stream in a first memory. The first memory can be a DDR memory, rather than the second memory of the image signal processor 340, such as SRAM. Simultaneously, the image signal processor 340 also stores the context data corresponding to each video stream in the first memory, while its internal resources still only include two context data streams, reserving storage space for active and shadow resources. Storing in the external first memory saves resources within the image signal processor 340. If multiple video streams all store video stream data and context data internally within the image signal processor 340, hardware resources will be idle when some video stream data and context data are not used. In the above three-stream data processing process, before processing the first video stream data, the image signal processor 340 first reads the first context data from the first memory and writes it to the address of the active resource in the second memory, and then reads the frame data of the first video stream data for processing. The scheduler 330 can then prepare for processing the next second video stream data by reading the second context data of the second video stream data from the first memory and writing it to the storage address of the shadow resource. After the processing time for the first video stream data is over, the scheduler reads the frame data of the second video stream data from the first memory and simultaneously notifies the image signal processor 340 to complete the switching between the active resource and the shadow resource. At this time, the first context data that has been switched back to the shadow resource needs to be written back to the first memory for retention. The scheduler 330 can then prepare for processing the next third video stream data by reading the third context data of the third video stream data from the first memory and writing it to the storage address of the shadow resource in the second memory. After the processing time for the second video stream data is over, the scheduler 330 reads the frame data of the third video stream data from the first memory and simultaneously notifies the image signal processor 340 to switch between the active resource and the shadow resource. Note that at this time, the second context data that has been switched back to the shadow resource needs to be written back to the first memory for retention. Then the scheduler 330 can continue to prepare for processing the next first video stream data, which will not be elaborated here. This continues until all video streams have completed the processing of one frame.

[0034] Figure 5 A schematic diagram of another video stream data processing circuit provided in some embodiments of this application is shown. The third memory is a memory located inside the image signal processor 340, so that when the amount of context data is small, it does not need to be placed in the external memory of the image signal processor 340, and the on-chip resources can be used.

[0035] Figure 6 A schematic diagram of a video stream data processing apparatus provided in some embodiments of this application is shown. The processing apparatus 600 includes: an acquisition unit 610, configured to acquire multiple video stream data based on a preset processing unit, wherein the multiple video stream data originate from different image sensors, and the preset processing unit is determined based on a pixel array in a frame image; an image signal processor 620, configured to perform image processing on the multiple video stream data based on a preset processing order and multiple context data, including: when processing the current video stream data using the current context data, other context data are configured to a non-working state; after the current video stream data processing is completed, switching to other context data to continue processing other video stream data, the current context data being configured to a non-working state; wherein the context data corresponds to the processed video stream data, and the non-working state includes: a waiting state or a storage state; and an output unit 630, configured to output data when the video stream data is completed based on the preset processing order.

[0036] Based on the same technical concept, this application also provides an electronic device, including the video stream data processing circuit provided in the above embodiments.

[0037] In the above embodiments, the descriptions of each embodiment have their own emphasis. Parts not described in detail or in a particular embodiment can be referred to in the relevant descriptions of other embodiments. Furthermore, the above embodiments can be freely combined as needed.

Claims

1. A processing circuit of video stream data, characterized by, include: Multiple data interfaces are configured to acquire multiple video stream data based on a preset processing unit, wherein the multiple data interfaces are respectively coupled to multiple image sensors, and the preset processing unit is determined based on the pixel array in a frame image, wherein the preset processing unit is at least one row of pixels in the frame image, or at least one column of pixels in the frame image, or a pixel block in the frame image. A video input processing unit, coupled to multiple data interfaces, is configured to perform a first data processing on multiple video stream data, the first data processing including: determining the data source of the multiple video stream data, and writing the multiple video stream data into different address segments of a first memory; A scheduler, coupled to the video input processing unit, is configured to write or read multiple video stream data based on a preset processing order; An image signal processor, coupled to the scheduler, is configured to have a set of hardware resources for algorithm processing and multiple context data. It is further configured to perform image processing on multiple video streams based on the hardware resources for algorithm processing and the multiple context data via the scheduler. This includes: when processing the current video stream data using the current context data, configuring other context data to a non-working state; after processing the current video stream data, switching to other context data to continue processing other video stream data, and switching the current context data to a non-working state; wherein, when switching video stream data, the hardware resources for algorithm processing continue to work; and outputting data when the video stream data is completed according to the preset processing order. The context data includes: image processing algorithm configuration data and intermediate image processing result data, and the context data corresponds to the processed video stream data; the non-working state includes: a waiting state or a storage state. It also includes: a second memory configured to store context data of the waiting state at a first address and context data of the working state at a second address, wherein the second memory is an on-chip buffer of the image signal processor and the first memory is an off-chip memory of the image signal processor.

2. The processing circuit of video stream data according to claim 1, wherein, The context data of the storage state is configured to be stored in the first memory.

3. The processing circuit of video stream data according to claim 1, wherein, The image signal processor is also configured to clear the previous video stream data when switching the context data.

4. The video stream data processing circuit according to any one of claims 1 to 3, characterized in that, The plurality of video stream data includes first video stream data and second video stream data, and the plurality of context data includes: first context data and second context data, wherein the first context data corresponds to the first video stream data, and the second context data corresponds to the second video stream data; The image signal processor is further configured to, when processing the first video stream data, control the first context data to the working state and control the second context data to the waiting state; and is configured to, when processing the second video stream data, control the first context data to the waiting state and control the second context data to the working state.

5. The processing circuitry of video stream data according to any one of claims 1 to 3, characterized in that, The plurality of video stream data includes first video stream data, second video stream data, and third video stream data, and the plurality of context data includes: first context data, second context data, and third context data, wherein the first context data corresponds to the first video stream data, the second context data corresponds to the second video stream data, and the third context data corresponds to the third video stream data; The image signal processor is further configured to, when processing the first video stream data, control the first context data to the working state, control the second context data to the waiting state, and control the third context data to the storage state; and is configured to, when processing the second video stream data, control the first context data to the storage state, control the second context data to the working state, and control the third context data to the waiting state; and is further configured to, when processing the third video stream data, control the first context data to the waiting state, control the second context data to the storage state, and control the third context data to the working state.

6. The processing circuitry of video stream data according to claim 5, wherein, The first memory is also configured to store the first context data, the second context data, and the third context data before the image signal processor begins processing; When the image signal processor processes the first video stream data, it is also configured to read the first context data from the first memory to the address of the active resource of the second memory, and the scheduler is also configured to read the second context data from the first memory to the address of the shadow resource of the second memory. After the first video stream data is processed, the scheduler is also configured to read the second video stream data from the first memory; The image signal processor writes the first context data to the first memory and switches the second context data to the address of the active resource of the second memory; The scheduler is also configured to read the third context data to the address of the shadow resource of the second memory; After the second video stream data is processed, the scheduler is also configured to read the third video stream data from the first memory; The image signal processor writes the second context data to the first memory and switches the third context data to the address of the active resource in the second memory; The scheduler is also configured to read the first context data to the address of the shadow resource of the second memory.

7. The processing circuitry of video stream data according to any one of claims 1 to 3, characterized in that, Multiple sets of context data are configured to be pre-stored in a first memory outside the image signal processor; or, Multiple sets of context data are configured to be stored in a third memory within the image signal processor chip; The image signal processor is also configured to read or write a plurality of the context data from the first memory or the third memory via the scheduler.

8. The processing circuitry of video stream data according to claim 1, wherein, The multiple data interfaces are configured to acquire portions of data from a frame of image sent by the image sensor as video stream data.

9. The processing circuitry of video stream data according to claim 1, wherein, The preset processing order includes: sequential processing, cyclic processing, or custom processing order.

10. An electronic device, comprising: The circuit includes the video stream data processing circuit according to any one of claims 1 to 9.

Images