Image processing system and method, and readable storage medium

By introducing shared memory and ping-pong buffering mechanisms in the image processing system, using the hardware acceleration characteristics of digital signal processors for image post-processing, the problem of low efficiency of image post-processing tasks in ARM architecture chips is solved, and efficient image processing and Low-energy image output.

CN119071498APending Publication Date: 2024-12-03SUZHOU KEDA TECH
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202411359024.6
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-09-27
Publication Date
2024-12-03

AI Technical Summary

Technical Problem

In ARM architecture chips, the communication efficiency between the central processor and the digital signal processor is low, resulting in low efficiency of image post-processing tasks. Especially in a multi-threaded environment, the computing resources of the central processor are not fully utilized, and the increase The complexity of multi-threaded scheduling affects the efficiency of the overall image processing system.

Method used

By introducing shared memory and ping-pong buffering mechanisms in the image processing system, direct memory access between the digital signal processor and the central processor is carried out, and image post-processing, including color, is performed using the hardware acceleration characteristics and parallel processing capabilities of the digital signal processor. For format conversion and image size scaling, the central processor and the digital signal processor work together through the instruction interaction port.

Benefits of technology

The multi-tasking capability of the image processing system is optimized, the image processing time is shortened, the data transmission delay is reduced, the overall energy consumption of the system is reduced, the high-quality image output is achieved, and the overall performance of the image processing system is improved.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119071498A_ABST
    Figure CN119071498A_ABST
Patent Text Reader

Abstract

The invention provides an image processing system and method and a readable storage medium, and the system comprises a central processing unit which comprises a coding module and a decoding module, and the decoding module is used for decoding video data to obtain original image data; the digital signal processor is used for selecting to-be-synthesized original image data according to the image synthesis parameters, and performing image post-processing on the to-be-synthesized original image data by using a built-in hardware algorithm and / or software algorithm to obtain a synthesized target image frame; and the coding module is used for coding the target image frame and outputting a corresponding picture synthesis image. According to the invention, a software and hardware algorithm cooperative work mode is adopted, and the hardware acceleration characteristic and the parallel processing capability of the digital signal processor are utilized, so that the image post-processing flow is optimized, the image processing time is shortened, the data transmission delay is reduced, and the overall energy consumption of the system is reduced under the condition that the CPU load is kept unchanged; the complex image processing requirement can be met, and high-quality image output is provided.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present disclosure relates to the field of image processing technology, and in particular, to an image processing system, method and readable storage medium. Background Art

[0002] A digital signal processor (DSP) is a microprocessor specifically designed for processing digital signals. It is well-known for its dedicated hardware acceleration function and is widely used in image signal processing chips, which can significantly improve or even exceed the processing power of the central processing unit. At the same time, digital signal processor suppliers also provide a comprehensive basic operator library for the functional implementation of digital signal processors in high-performance application designs such as image and video processing, which contains reference implementations of specific parameter combinations of many standard operators, which can provide great convenience for developers.

[0003] In the prior art, the digital signal processor in the ARM architecture chip is usually configured to perform a part of specific image post-processing tasks, such as image filtering, format parameter conversion, image compression, etc. The central processing unit needs to make multiple operator calls to the digital signal processor and wait for its execution to be completed. Especially in a multi-threaded environment, the computing resources of the central processing unit are not fully utilized, and the complexity of multi-threaded scheduling is increased, which is not conducive to improving the overall efficiency of the image processing system. Summary of the invention

[0004] In response to the problems raised in the background technology, the present disclosure provides an image processing system, method and readable storage medium, which optimize the multi-tasking processing capability of ARM architecture chips, thereby improving the performance of the image processing system as a whole.

[0005] Specifically, the first aspect of the present disclosure provides an image processing system, which may specifically include:

[0006] The central processing unit includes an encoding module and a decoding module, wherein the decoding module is used to decode the received video data to obtain original image data;

[0007] A digital signal processor, configured to select at least one raw image data to be synthesized according to the required image synthesis parameters, and perform image post-processing on the raw image data to be synthesized using a hardware algorithm and / or a software algorithm built into the digital signal processor to obtain a synthesized target image frame;

[0008] The encoding module is used to encode the target image frame and then output the corresponding screen composite image.

[0009] In a possible implementation of the first aspect, there is a shared memory between the digital signal processor and the central processing unit, and the digital signal processor transfers the original image data from the shared memory to an internal cache of the digital signal processor through a direct memory access transmission module;

[0010] The internal cache of the digital signal processor includes at least one original image frame structure and at least one target image frame structure, each original image frame structure is used to store an original image data, and each target image frame structure is used to store at least one target image data, wherein the target image data is generated by image post-processing of the target original image data to be synthesized; the target original image data to be synthesized is selected from at least one original image frame structure based on the image synthesis parameters corresponding to the target frame structure; and the image synthesis parameters corresponding to each target image frame structure are different.

[0011] In a possible implementation of the first aspect, the decoding module includes a plurality of concurrently set decoding threads and a first buffer pool, the decoding threads concurrently decode the video data, and instantly transmit the acquired original image data to the first buffer pool of the shared memory;

[0012] The encoding module includes multiple concurrently set encoding threads and a second buffer pool. The digital signal processor transfers the acquired target image frame to the second buffer pool of the shared memory in real time, and the encoding threads concurrently encode the target image frame in the second buffer pool according to a preset period.

[0013] In a possible implementation of the first aspect above, the first buffer pool and the second buffer pool are located in a shared memory between the digital signal processor and the central processing unit;

[0014] The digital signal processor adopts a ping-pong buffer mechanism to transfer the raw image data to be synthesized from the first buffer pool to the internal cache of the digital signal processor through at least two direct memory access transmission modules; and / or

[0015] The digital signal processor adopts a ping-pong buffer mechanism, and moves the target image frame from the internal cache of the digital signal processor to the second buffer pool through at least two direct memory access transmission modules.

[0016] In a possible implementation of the first aspect, performing image post-processing on the raw image data to be synthesized by using a hardware algorithm and / or a software algorithm built into a digital signal processor includes:

[0017] Acquire a first image parameter corresponding to the original image data;

[0018] According to the image synthesis parameters, the target image data corresponding to the original image data is obtained, and the second image parameters corresponding to the target image data are obtained;

[0019] comparing the first image parameter with the second image parameter, and if there is a difference between the first image parameter and the second image parameter, performing a parameter adjustment operation on the original image data based on a hardware algorithm built into the digital signal processor;

[0020] The parameter adjustment operation includes performing a color format conversion operation and an image size scaling operation on the original image data.

[0021] In a possible implementation of the first aspect above, there is at least one instruction interaction port between the central processing unit and the digital signal processor;

[0022] The central processing unit generates corresponding image processing instructions according to the received image post-processing requirement instructions and provides them to the digital signal processor through the instruction interaction port;

[0023] The digital signal processor calls a corresponding software algorithm from a plurality of built-in software algorithms according to the image processing instruction to perform a corresponding image processing operation.

[0024] In a possible implementation of the first aspect, the target image data is placed into a corresponding position of the target image frame using a memory copy function of a digital signal processor to obtain the target image frame.

[0025] A second aspect of the present disclosure provides an image processing method, which is applied to the image processing system provided in the first aspect. The image processing method comprises the following steps:

[0026] Decoding the received video data to obtain original image data;

[0027] Selecting at least one original image data to be synthesized according to the required image synthesis parameters;

[0028] Performing image post-processing on the raw image data to be synthesized by using a hardware algorithm and / or a software algorithm built into a digital signal processor to obtain a synthesized target image frame;

[0029] After encoding the target image frame, the corresponding composite image is output.

[0030] In a possible implementation of the second aspect, performing image post-processing on the raw image data to be synthesized by using a hardware algorithm and / or a software algorithm built into a digital signal processor includes the following steps:

[0031] Acquire a first image parameter corresponding to the original image data;

[0032] According to the image synthesis parameter, obtaining a second image parameter of the target image data corresponding to the original image data;

[0033] comparing the first image parameter and the second image parameter, and if there is a difference between the first image parameter and the second image parameter, performing a parameter adjustment operation on the original image data based on a hardware algorithm built into the digital signal processor; wherein the parameter adjustment operation includes performing a color format conversion operation and an image size scaling operation on the original image data; and / or

[0034] According to the image processing instruction, a corresponding software algorithm is called from a plurality of built-in software algorithms to perform a corresponding image processing operation on the original image data.

[0035] Digital Signal Processor A third aspect of the present disclosure provides a computer-readable storage medium for storing a program, which, when executed by a processor, implements the steps of the image processing method provided in the second aspect.

[0036] Compared with the background technology, the present invention has the following beneficial effects:

[0037] The technical solution provided by the present disclosure adopts a software-hardware collaborative working method and utilizes the hardware acceleration characteristics and parallel processing capabilities of the digital signal processor. It can optimize the entire image post-processing process while maintaining the central processing unit load unchanged, shorten the image processing time, reduce data transmission delays, and reduce the overall energy consumption of the system. At the same time, it can meet complex image processing requirements and provide high-quality image output. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] Other features, objects and advantages of the present disclosure will become more apparent from a reading of the detailed description of non-limiting embodiments made with reference to the following accompanying drawings.

[0039] Figure 1 According to an embodiment of the present disclosure, a structural schematic diagram of an image processing system is shown.

[0040] Figure 2 According to an embodiment of the present disclosure, a schematic diagram of data interaction in an image processing system is shown.

[0041] Figure 3 According to an embodiment of the present disclosure, a data flow diagram of a central processing unit and a digital signal processor performing image post-processing through instruction interaction is shown.

[0042] Figure 4 According to an embodiment of the present disclosure, a schematic diagram of image data distribution of an original image frame structure and a target image frame structure in an image synthesis process is shown.

[0043] Figure 5 According to an embodiment of the present disclosure, a flowchart of an image processing method is shown.

[0044] Figure 6 According to an embodiment of the present disclosure, a specific flow chart of performing image post-processing on raw image data to be synthesized using a hardware algorithm and / or software algorithm built into a digital signal processor is shown. DETAILED DESCRIPTION

[0045] The following describes the embodiments of the present disclosure through specific examples, and those skilled in the art can easily understand other advantages and effects of the present disclosure from the contents disclosed in the present disclosure. The present disclosure can also be implemented or applied through other different specific embodiments, and the details in the present disclosure can also be modified or changed in various ways according to different viewpoints and application systems without departing from the spirit of the present disclosure. It should be noted that the embodiments in the present disclosure and the features in the embodiments can be combined with each other without conflict.

[0046] The following is a detailed description of the embodiments of the present disclosure with reference to the accompanying drawings so that those skilled in the art can easily implement the present disclosure. The present disclosure can be embodied in many different forms and is not limited to the embodiments described herein.

[0047] In the representations of the present disclosure, the reference terms "one embodiment", "some embodiments", "examples", "specific examples", or "some examples" and the like mean that the specific features, structures, materials, or characteristics represented in conjunction with the embodiment or example are included in at least one embodiment or example of the present disclosure. Moreover, the specific features, structures, materials, or characteristics represented may be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine different embodiments or examples represented in the present disclosure and the features of different embodiments or examples, unless they are mutually contradictory.

[0048] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Therefore, a feature defined as "first" or "second" may explicitly or implicitly include at least one of the features. In the representation of the present disclosure, the meaning of "plurality" is two or more, unless otherwise clearly and specifically defined.

[0049] In order to clearly describe the present disclosure, components not related to the description are omitted, and the same reference numerals are given to the same or similar components throughout the specification.

[0050] Throughout the specification, when a device is said to be "connected" to another device, this includes not only the case of "direct connection" but also the case of "indirect connection" by placing other elements therebetween. In addition, when a device is said to "include" a certain component, unless otherwise stated, it does not exclude other components, but means that other components may be included.

[0051] When a device is said to be "on" another device, it may be directly on the other device, but there may also be other devices between it. In contrast, when a device is said to be "directly" on another device, there are no other devices between it.

[0052] Although the terms first, second, etc. are used to represent various elements in some examples, these elements should not be limited by these terms. These terms are only used to distinguish one element from another element. For example, the first interface and the second interface, etc. are represented. Moreover, as used in this article, the singular forms "one", "one" and "the" are intended to also include plural forms, unless there is an opposite indication in the context. It should be further understood that the terms "comprising" and "including" indicate the existence of features, steps, operations, elements, components, projects, kinds, and / or groups, but do not exclude the existence, occurrence or addition of one or more other features, steps, operations, elements, components, projects, kinds, and / or groups. The terms "or" and "and / or" used herein are interpreted as inclusive, or mean any one or any combination. Therefore, "A, B or C" or "A, B and / or C" means "any of the following: A; B; C; A and B; A and C; B and C; A, B and C". Only when the combination of elements, functions, steps or operations is inherently mutually exclusive in some way, will there be an exception to this definition.

[0053] The technical terms used herein are only used to refer to specific embodiments and are not intended to limit the present disclosure. The singular form used herein also includes the plural form unless the sentence clearly indicates the contrary meaning. The meaning of "including" used in the specification is to specify specific characteristics, regions, integers, steps, operations, elements and / or components, and does not exclude the existence or addition of other characteristics, regions, integers, steps, operations, elements and / or components.

[0054] Although not defined differently, all terms, including technical and scientific terms used herein, have the same meaning as those generally understood by those skilled in the art to which the present disclosure belongs. Terms defined in commonly used dictionaries are additionally interpreted as having meanings consistent with the content of relevant technical literature and current disclosures, and unless defined, shall not be overly interpreted as ideal or very formal meanings.

[0055] In order to overcome the problems raised in the background technology, the present disclosure provides an image processing system, method and readable storage medium, which can improve the image post-processing speed and processing quality while maintaining the central processing unit resource consumption. Specifically, in some embodiments of the present disclosure, Figure 1 A schematic diagram of the structure of an image processing system is shown. Figure 2 FIG. 1 shows a schematic diagram of data interaction in an image processing system. Figure 1 and Figure 2 As shown, the image processing system provided by the specific embodiment of the present disclosure may specifically include:

[0056] The central processor 100 specifically includes an encoding module 102 and a decoding module 101, wherein the decoding module 101 is used to decode the received video data to obtain a plurality of original image data. Specifically, in the above-mentioned embodiment of the present disclosure, Figure 2 As shown, the shared memory between the central processing unit and the digital signal processor further includes a first buffer pool 103a and a second buffer pool 103b; the encoding module 101 includes a plurality of concurrently set decoding threads 101a, and the decoding threads 101a concurrently decode the received video data and transmit the acquired original image data to the first buffer pool 103a of the shared memory in real time.

[0057] The digital signal processor 200 is used to select at least one raw image data to be synthesized according to the required image synthesis parameters, and perform image post-processing on the raw image data to be synthesized using the hardware algorithm and / or software algorithm built into the digital signal processor to obtain a synthesized target image frame. Figure 2 As shown, the digital signal processor 200 can be configured to carry the original image data from the first buffer pool 103a after the video data is decoded at a preset time interval, and according to the required image synthesis parameters, the digital signal processor determines and selects a certain amount of original image data for subsequent synthesis processing to generate a corresponding target image frame. Among them, the image synthesis parameters determine the number and type of images required to achieve a specific visual effect, and the object to be synthesized can be all images obtained based on the video data, or part of the images therein, such as multiple or one image, etc., which is not limited here. That is to say, in the embodiment provided by the present disclosure, the image processing system needs to perform certain post-processing steps for the original image data to be synthesized, and does not involve end-to-end direct transmission and presentation of the screen image.

[0058] It can be understood that in the above embodiments of the present disclosure, the digital signal processor (DSP) 200 is a microprocessor specifically used for processing digital signals, and it and the aforementioned central processing unit 100 are both hardware modules or chips; it has significant performance advantages in performing complex mathematical operations, especially operations related to signal processing: the digital signal processor 200 usually has a specially designed instruction set to accelerate signal processing operations; and supports a single instruction multiple data (SIMD) architecture, which can process multiple data points at the same time, such as vector processing instructions that can perform the same operation on a set of data at the same time, which improves the efficiency of concurrent execution in tasks such as image processing. In the above embodiments provided by the present disclosure, the digital signal processor 200 is built with a dedicated hardware accelerator related to image post-processing requirements, such as a vector processing unit, a floating-point operation unit, etc., to implement the required complex image algorithms, while being able to further improve the processing speed of specific tasks and reduce the burden on the central processing unit 100. That is to say, the embodiment provided by the present disclosure fully utilizes the hardware acceleration characteristics, parallel processing capabilities and dedicated mathematical operation units of the digital signal processor 200 during the entire operation process of post-processing the image, and a software algorithm is built into the digital signal processor 200 to implement the required image post-processing, thereby expanding the image post-processing function of the digital signal processor 200, transferring the computationally intensive tasks in the image post-processing process to the digital signal processor 200 for execution, and releasing the relevant resources of the central processing unit 100, thereby improving the overall performance of the image processing system, and achieving faster data processing speed and higher energy efficiency. The specific processing of the original image data by the digital signal processor 200 will be described in detail later.

[0059] In the above embodiment, the encoding module 102 is used to encode the acquired target image frame after the digital signal processor 200 completes the image post-processing task, so as to output the corresponding picture composite image. Figure 2 As shown, the encoding module 102 includes multiple concurrently set encoding threads 102a, the digital signal processor 200 instantly transfers the acquired target image frame to the second buffer pool 103b of the shared memory, and the encoding thread 102a concurrently encodes the target image data in the second buffer pool 103b according to a preset period.

[0060] It is understandable that in the prior art, the Remote Procedure Call (RPC) framework is a standard practice for implementing communication between the digital signal processor and the central processing unit. The digital signal processor and the central processing unit share the memory address space through the inter-core communication module, but the latency of the digital signal processor 200 directly accessing the DDR memory is usually higher than accessing its internal data memory RAM. In order to further improve the image processing efficiency, in the above-mentioned embodiments provided in the present disclosure, the digital signal processor 200 can move the required raw image data to its own internal cache during the image post-processing process. Specifically, in the process of moving the raw image data, the digital signal processor 200, such as Figure 2 As shown, the digital signal processor 200 can use the direct memory access (DMA) technology and a ping-pong buffer mechanism to move the original image data to be synthesized from the first buffer pool to the internal cache of the digital signal processor through at least two direct memory access transmission modules; and in the process of moving the target image frame, the ping-pong buffer mechanism can also be used to move the target image frame from the internal cache of the digital signal processor to the second buffer pool 103b through at least two direct memory access transmission modules.

[0061] In the above embodiment, the ping-pong buffer mechanism can improve the data transmission speed between the digital signal processor 200 and the first buffer pool 103a and the second buffer pool 103b. For example, a frame of image can be divided into multiple blocks, and each time a block is transferred, at least two direct memory access transmission modules are used for alternating transfer. Due to the slow reading and writing, when the previous block data is written into the digital signal processor 200, the next block starts to be read from the memory of the central processing unit 100 synchronously, and the alternation is repeated until the last block data is read and waits to be written into the digital signal processor 200, so as to realize the process of the digital signal processor 200 transferring the original image data from the first buffer pool 103a, thereby realizing the transfer of all the original image data output by the decoding module to the internal cache for processing; the process of transferring the target image frame from the digital signal processor 200 to the second buffer pool 103b is the opposite, which will not be described in detail here. The application of the above ping-pong buffer mechanism further reduces the data interaction delay between the digital signal processor 200 and the central processing unit 100, thereby improving the overall efficiency of image processing.

[0062] In the above embodiment, specifically, at least one original image frame structure 201 and at least one target image frame structure 202 are configured in the internal cache of the digital signal processor 200. When the digital signal processor moves the original image data through the DMA technology, the original image data is stored in the original image frame structure 201, wherein one original image frame structure 201 is used to store one original image data 2011; further, the original image frame structure 201 may specifically include frame header information and original image data, and the frame header information includes the width, height, image format and original image storage information of the corresponding original image data; the target image frame structure 208 is used to store at least one target image data 2081, and a plurality of synthesizers (corresponding to different image synthesis parameters, the image synthesis parameters include image synthesis related information, such as the synthesized image width, height, image format, original image storage information and each target image) may be pre-created in the digital signal processor. The digital signal processor can select the original image frame structure 201 to be synthesized from multiple original image frame structures 201 based on the corresponding image synthesis parameters and the relevant information of the original image frame structure, perform image post-processing on the original image data 2011 in the selected original image frame structure 201, generate the corresponding target image data 2081, and put the target image data 2081 into the position corresponding to the target image frame structure 208 corresponding to the image synthesis parameters to obtain the target image frame, that is, based on the image synthesis parameters, it can be confirmed which original image data are selected for synthesis, as well as the size, format, resolution and position of the original image data after processing in the synthesized screen, and the original image is processed in combination with other post-processing requirements of the user such as image enhancement and color enhancement, and the post-processed target image data is put into the corresponding position of the target image frame structure.

[0063] In the specific implementation process of the digital signal processor 200 performing image post-processing, the first image parameter corresponding to the original image data can be obtained, wherein the first image parameter can be obtained through the frame header information in the aforementioned original image frame structure, which is not limited here; and according to the image synthesis parameter, the second image parameter of the target image data corresponding to the original image data is obtained, wherein the second image parameter can be obtained based on the synthesized image parameter information in the image synthesis parameter, which is not limited here. It can be understood that some preset first image parameters will be carried during the transmission of the original image data, and these first image parameters usually include the width and height, color space format, resolution and other information corresponding to the original image data. At the same time, the corresponding instructions of the image synthesis parameter and / or the image post-processing requirements will also include the second image parameter corresponding to the target image data to be obtained after the original image data is processed by the image post-processing, and the parameter type included in the second image parameter can be consistent with the parameter type included in the first image parameter. By comparing the first image parameter and the second image parameter, when there is a difference between the first image parameter and the second image parameter, the parameter adjustment operation is performed on the original image data based on the hardware algorithm built into the digital signal processor. It is understandable that the above parameter adjustment operation can be implemented through a dedicated hardware accelerator built into a digital signal processor, such as a vector processing unit, a floating point operation unit, etc., which can effectively improve the efficiency of image post-processing.

[0064] In the above embodiment, the parameter adjustment operation includes performing a color format conversion operation and an image size scaling operation on the original image data. Specifically, when there is a difference between the color space format required by the original image data and the target image data, a color conversion operation needs to be performed to obtain intermediate image data consistent with the color space format required by the target image data; and when there is a difference between the image size of the original image data and the target image data, a size scaling operation needs to be performed to obtain intermediate image data consistent with the image size required by the target image data. If no other image post-processing steps need to be performed, the above intermediate image data can be directly used as the target image data corresponding to the original image data; if other image post-processing steps still need to be performed, other image post-processing steps can be further performed on the basis of the above intermediate image data to obtain the required target image data.

[0065] Generally speaking, for the original image data to be synthesized, the unification of the color space format and the adjustment of the image size are necessary for the original image data, that is, the provided digital signal processor 200 needs to be able to at least implement color conversion operations and size scaling operations. The implementation of this part of the function can be directly completed using the image processing function built into the digital signal processor 200, that is, the color conversion operation and size scaling operation are both implemented through the hardware algorithm built into the digital signal processor. This enables the digital signal processor 200 to adaptively improve the processing efficiency in the color conversion operation and size scaling operation process by relying on its own hardware acceleration characteristics during the execution of the above-mentioned image post-processing.

[0066] In some embodiments of the present disclosure, further, at least one instruction interaction port may be configured between the central processor 100 and the digital signal processor 200, for sending other image processing instructions from the central processor 100 to the digital signal processor 200. It is understandable that for the digital signal processor 200, the color conversion operation and size scaling operation provided in the aforementioned embodiment may be directly performed using the hardware image processing function built into the digital signal processor 200, but this does not limit the digital signal processor 200 to only being able to implement these image post-processing functions.

[0067] In the above embodiment, based on different image post-processing requirements, other software programs related to image processing may be configured on the digital signal processor 200. Different from the functions implemented by the hardware algorithm of the digital signal processor 200, such as color conversion operation and size scaling operation, other image processing-related functional modules may be implemented by software program coding according to actual needs, and these software algorithm programs may be determined whether to be called through the control instructions of the central processing unit 100. Specifically, the central processing unit 100 may generate corresponding image processing instructions according to the received image post-processing requirement instructions and provide them to the digital signal processor 200 through the instruction interaction port, and the digital signal processor 200 may perform corresponding image processing operations according to these image processing instructions.

[0068] In the above embodiment, in an alternative specific implementation, the image processing operations that can be implemented by the digital signal processor 200 include but are not limited to a combination of one or more of edge enhancement processing, color enhancement processing, image border configuration processing, and image source identification processing.

[0069] Specifically, the edge enhancement processing implemented by the digital signal processor 200 can detect and enhance the edges of specific objects or image subjects in the decoded original image data, so that the edges of the above objects are more clearly visible, thereby significantly improving the overall clarity and visual expression of the image. It can be understood by those skilled in the art that edge enhancement processing can not only improve the overall visual effect of the processed image, but also provide a positive impact in subsequent target detection tasks such as portrait detection or text recognition, which are not limited here.

[0070] Specifically, the color enhancement processing implemented by the digital signal processor 200 can perform detailed color processing on the decoded original image data to optimize the overall hue, saturation and contrast of the image, enhance the overall visual expression of the image, and make the image content more vivid and realistic.

[0071] Specifically, the image border configuration processing and image source identification processing implemented by the digital signal processor 200 are often used in the image post-processing task of designing image synthesis, wherein the image border configuration processing can enhance the visibility and importance of specific objects: in the process of picture synthesis, the image border is configured to black by default to provide a simple visual effect; and in some special scenes, such as in a conference scene, when special objects such as speakers or hosts need to be highlighted, the border color can be modified by the user and configured separately. The image source identification facilitates the identification of the image source in the picture synthesis image, for example, the terminal number identification corresponding to the image can be displayed in the corner or other corresponding area of ​​each processed image, which is not limited here.

[0072] It is understandable that, for different image post-processing requirements, the various image processing operations provided by the digital signal processor 200 in the above embodiment can be selected from one or more combinations thereof. Figure 3 FIG. 1 shows a data flow diagram of a central processing unit 100 and a digital signal processor 200 performing image post-processing through instruction interaction. Figure 3As shown, the corresponding original image data in the selected original image frame structure 201 can be sequentially passed through the color conversion module 202, the size scaling module 203, the edge enhancement module 204, the color enhancement module 205, the border configuration module 206 and the source identification module 207, and finally the generated target image data is placed in the target image frame structure 208. The edge enhancement module 204, the color enhancement module 205, the border configuration module 206 and the source identification module 207 respectively perform optional image processing tasks according to the instructions of the central processing unit 100 through the edge enhancement instruction port 103, the color enhancement instruction port 104, the border configuration instruction port 105 and the source identification instruction port 106. It can be understood by those skilled in the art that in the process of image post-processing, it is not necessary to perform color enhancement, edge enhancement and other processing steps for each original image data, and the image processing steps can be performed on demand according to the instructions provided by the central processing unit 100. At the same time, Figure 2 The data flow diagram shown only exemplarily presents the execution order of an image processing step. Those skilled in the art can adjust the order of the various image processing modules in the data flow according to actual needs, and this is not limited here.

[0073] Furthermore, in the process of performing the above-mentioned image processing operation configuration, it is also necessary to balance the processing quality and processing efficiency corresponding to the digital signal processor 200, that is, the processing time for a frame of image data cannot be too long to cause image post-processing delay, and at the same time, the quality of the processed image must still meet the preset requirements to avoid display problems in the target image data obtained after post-processing. Those skilled in the art can selectively perform the required image processing steps for the original image data to be processed according to the above-mentioned balance requirements, and this is also not limited here.

[0074] In some embodiments of the present disclosure, after all image post-processing is completed, the target image data can be placed in the corresponding position of the target image frame using the memory copy function of the digital signal processor 200 to obtain the target image frame. Figure 4 A schematic diagram of image data distribution of an original image frame structure and a target image frame structure in an image synthesis process is shown. Figure 4As shown, in the original image sequence, the original image frame structures 201a, 201b, 201c, etc. are arranged in the form of a queue and wait for processing by various functional modules in the digital signal processor 200 to obtain corresponding target image data according to image synthesis parameters and image post-processing requirement instructions. In the face of different image synthesis parameters, the original image data in the original image frame structure can obtain different target image data through different image post-processing. For example, the original image data 201c1 in the original image frame structure 201c can obtain corresponding target image data 208a3, 208b2 and 208c1 respectively in the processing of different image synthesis requirements, and different synthesis methods are used to generate the required target frame structures 208a, 208b, 208c. It can be understood that based on the picture synthesis requirements, the original image data in the same original frame structure can be used to generate corresponding target image data in different target image frame structures, and the same target image frame structure can include target image data generated by image post-processing of the original image data of one or more original frame structures. Figure 4 A simplified diagram of data flow of a digital signal processor is merely provided as an example. The arrangement of target image data in the target image frame structure is set according to actual picture synthesis requirements and is not limited here.

[0075] In some embodiments of the present disclosure, Figure 5 A schematic flow chart of an image processing method is provided, and the image processing method is applied to the image processing system provided in the above embodiment. Figure 5 As shown, the image processing method includes the following steps:

[0076] Step 501: Decode the received video data to obtain original image data.

[0077] Step 502: Select at least one original image data to be synthesized according to the required image synthesis parameters.

[0078] Step 503: perform image post-processing on the raw image data to be synthesized using the hardware algorithm and / or software algorithm built into the digital signal processor to obtain a synthesized target image frame.

[0079] Step 504: Encode the target image frame and output a corresponding composite image.

[0080] Furthermore, in the specific implementation of the aforementioned step 504, Figure 6 The following is a schematic diagram showing a specific process of performing image post-processing on the original image data to be synthesized using the hardware algorithm and / or software algorithm built into the digital signal processor. Figure 6 As shown, the following steps may be specifically included:

[0081] Step 5021: Obtain first image parameters corresponding to the original image data, and obtain second image parameters of the target image data corresponding to the original image data according to the image synthesis parameters. The first image parameters can be directly obtained through the frame header information in the original image frame structure, which is not limited here.

[0082] Step 5022: Compare the first image parameter with the second image parameter. If there is a difference between the first image parameter and the second image parameter, perform a parameter adjustment operation on the original image data based on a hardware algorithm built into the digital signal processor.

[0083] The parameter adjustment operation may include performing a color format conversion operation and an image size scaling operation on the original image data, which will not be described in detail here.

[0084] Step 5023: Call a corresponding software algorithm from a plurality of built-in software algorithms according to the image processing instruction to perform a corresponding image processing operation on the original image data.

[0085] It is understandable that the above steps 5022 and 5023 can be performed sequentially or simultaneously, or one of the steps can be performed according to the situation, which is not limited here. The relevant contents of the above steps 501 to 504 have been explained in the above related embodiments, and will not be repeated here.

[0086] In some embodiments of the present disclosure, a computer-readable storage medium is further provided for storing a program, which, when executed by a processor, implements the steps of the image processing method provided in the aforementioned embodiments.

[0087] To sum up, through the technical solution provided by the present invention, by adopting the collaborative working mode of software and hardware and utilizing the hardware acceleration characteristics and parallel processing capabilities of the digital signal processor, it is possible to optimize the entire image post-processing process while maintaining the central processing unit load unchanged, shorten the image processing time, reduce data transmission delays, and reduce the overall energy consumption of the system. At the same time, it can meet complex image processing requirements and provide high-quality image output.

[0088] The above content is a further detailed description of the present disclosure in combination with specific preferred implementation modes, and it cannot be determined that the specific implementation of the present disclosure is limited to these descriptions. For ordinary technicians in the technical field to which the present disclosure belongs, several simple deductions or substitutions can be made without departing from the concept of the present disclosure, which should be regarded as falling within the scope of protection of the present disclosure.

Claims

1. An image processing system, It is characterized in that include: A central processing unit, comprising an encoding module and a decoding module, wherein the decoding module is used to decode the received video data to obtain original image data; A digital signal processor, configured to select at least one of the original image data to be synthesized according to the required image synthesis parameters, and perform image post-processing on the original image data to be synthesized using a hardware algorithm and / or software algorithm built into the digital signal processor to obtain a synthesized target image frame; The encoding module is used to output a corresponding screen composite image after encoding the target image frame.

2. The image processing system according to claim 1, It is characterized in that There is a shared memory between the digital signal processor and the central processing unit, and the digital signal processor transfers the original image data from the shared memory to the internal cache of the digital signal processor through a direct memory access transmission module; The internal cache of the digital signal processor includes at least one original image frame structure and at least one target image frame structure, each of the original image frame structures is used to store one original image data, and each of the target image frame structures is used to store at least one target image data, wherein the target image data is generated by the target original image data to be synthesized through the image post-processing; The target original image data to be synthesized is selected from the at least one original image frame structure based on the image synthesis parameters corresponding to the target frame structure; and the image synthesis parameters corresponding to each target image frame structure are different.

3. The image processing system according to claim 2, It is characterized in that The decoding module includes a plurality of concurrently arranged decoding threads, the decoding threads concurrently decode the video data, and instantly transmit the acquired original image data to the first buffer pool of the shared memory; The encoding module includes a plurality of concurrently set encoding threads. The digital signal processor instantly transmits the acquired target image frame to the second buffer pool of the shared memory. The encoding threads concurrently encode the target image frame in the second buffer pool according to a preset period.

4. The image processing system according to claim 3, It is characterized in that The digital signal processor adopts a ping-pong buffer mechanism to transfer the raw image data to be synthesized from the first buffer pool to the internal cache of the digital signal processor through at least two of the direct memory access transmission modules; and / or The digital signal processor adopts a ping-pong buffer mechanism to transfer the target image frame from the internal cache of the digital signal processor to the second buffer pool through at least two direct memory access transmission modules.

5. The image processing system according to claim 1, It is characterized in that The performing image post-processing on the raw image data to be synthesized by using the hardware algorithm and / or software algorithm built into the digital signal processor comprises: Acquire a first image parameter corresponding to the original image data; According to the image synthesis parameter, obtaining a second image parameter of the target image data corresponding to the original image data; comparing the first image parameter and the second image parameter, and if there is a difference between the first image parameter and the second image parameter, performing a parameter adjustment operation on the original image data based on a hardware algorithm built into the digital signal processor; The parameter adjustment operation includes performing a color format conversion operation and an image size scaling operation on the original image data.

6. The image processing system according to any one of claims 1 to 5, It is characterized in that There is at least one instruction interaction port between the central processing unit and the digital signal processor; The central processor generates an image processing instruction according to the received image post-processing requirement instruction, and provides the instruction to the digital signal processor through the instruction interaction port; The digital signal processor calls a corresponding software algorithm from a plurality of built-in software algorithms according to the image processing instruction to perform a corresponding image processing operation.

7. The image processing system according to claim 2, It is characterized in that The target image data is placed into a corresponding position of the target image frame structure using a memory copy function of the digital signal processor to obtain the target image frame.

8. An image processing method, It is characterized in that Applied in the image processing system according to any one of claims 1 to 7, the image processing method comprises the following steps: Decoding the received video data to obtain original image data; Selecting at least one of the original image data to be synthesized according to the required image synthesis parameters; Performing image post-processing on the raw image data to be synthesized by using the hardware algorithm and / or software algorithm built into the digital signal processor to obtain a synthesized target image frame; The target image frame is encoded and processed to output a corresponding composite image.

9. The image processing method according to claim 8, It is characterized in that Performing image post-processing on the raw image data to be synthesized by using the hardware algorithm and / or software algorithm built into the digital signal processor comprises the following steps: Acquire a first image parameter corresponding to the original image data; According to the image synthesis parameter, obtain a second image parameter of the target image data corresponding to the original image data; compare the first image parameter with the second image parameter, and when there is a difference between the first image parameter and the second image parameter, perform a parameter adjustment operation on the original image data based on the hardware algorithm built into the digital signal processor; wherein the parameter adjustment operation includes performing a color format conversion operation and an image size scaling operation on the original image data; and / or According to the image processing instruction, a corresponding software algorithm is called from a plurality of built-in software algorithms to perform a corresponding image processing operation on the original image data.

10. A computer-readable storage medium for storing a program, It is characterized in that When the program is executed by a processor, the steps of the image processing method according to claim 8 or 9 are implemented.