Image processing method, device, equipment, and medium

Abstract

The present disclosure provides an image processing method, apparatus, device, and medium, and relates to the computer technology field, particularly to chip technology, artificial intelligence technology, and image processing technology field. The implementation includes reading a plurality of images arranged in a predetermined order from a first memory and writing them into a second memory readable and writable by a processing unit, reading the plurality of images from the second memory using the processing unit, determining a plurality of first feature data corresponding to the plurality of images, determining filler data for each of the first feature data based on the data amount of the first feature data and the predetermined data amount, splicing the first feature data and the filler data to obtain second feature data having a predetermined data amount, sequentially writing the plurality of second feature data into the second memory in the predetermined order, and transferring the second feature data stored in the second memory to the first memory in response to determining that the data amount of the second feature data stored in the second memory has reached a predetermined threshold.

Description

[Technical Field]

[0001] The present disclosure relates to the fields of computer technology, in particular chip technology, artificial intelligence technology and image processing technology, and specifically to an image processing method, apparatus, electronic device, computer-readable storage medium and computer program product. [Background technology]

[0002] Artificial intelligence is a field that studies how computers can mimic some of the human thought processes and intellectual behaviors (e.g., learning, reasoning, thinking, planning, etc.), and includes both hardware and software technologies. AI hardware technologies generally include sensors, AI-specific chips, cloud computing, distributed storage, and big data processing, while AI software technologies mainly include computer vision technology, speech recognition technology, natural language processing technology, machine learning / deep learning, big data processing technology, and knowledge graph technology.

[0003] With the development of artificial intelligence technology, more and more applications are relying on artificial intelligence technology to achieve results far exceeding those of traditional algorithms. Deep learning is a data-intensive and computation-intensive algorithm, so to improve the efficiency of image processing based on artificial intelligence technology, it is necessary to rationally utilize the hardware resources of image processing devices and reduce processing delays.

[0004] The approaches described in this section are not necessarily approaches that have been previously conceived or adopted. Unless otherwise noted, any approach described in this section should not be considered prior art merely because it is included in this section. Likewise, unless otherwise noted, the subject matter addressed in this section should not be considered an admission of any prior art. Summary of the Invention

[0005] [CROSS-REFERENCE TO RELATED APPLICATIONS] This application claims priority to Chinese Patent Application No. 202311170382.8, filed on September 11, 2023, the entire contents of which are incorporated herein by reference.

[0006] The present disclosure provides image processing methods and apparatus, chips, electronic devices, computer-readable storage media and computer program products.

[0007] According to one aspect of the present disclosure, there is provided an image processing method executed by an image processing device, the image processing device including a processing unit, the method including: reading a plurality of images arranged in a predetermined order from a first memory, and writing the images to a second memory readable and writable by the processing unit; reading the plurality of images from the second memory using the processing unit; determining a plurality of first feature data corresponding to the plurality of images, respectively; and performing a first process on each of the plurality of first feature data to obtain second feature data having a predetermined data amount, the first process including: The method includes obtaining filler data consisting of a first number of filler elements by determining a first number based on the data amount of the first feature data and the predetermined data amount, and splicing the first feature data and the filler data to obtain the second feature data; sequentially writing a plurality of second feature data corresponding to the plurality of first feature data respectively to the second memory in the predetermined order; and transferring the second feature data stored in the second memory to the first memory in response to determining that the data amount of the second feature data stored in the second memory has reached a predetermined threshold.

[0008] According to one aspect of the present disclosure, there is provided an image processing device, including: a processing unit; a writing unit configured to read a plurality of images arranged in a predetermined order from a first memory and write the images to a second memory readable and writable by the processing unit; and a transport unit configured to transport the second feature data stored in the second memory to the first memory in response to determining that a data amount of the second feature data stored in the second memory has reached a predetermined threshold, wherein the processing unit includes a reading subunit configured to read the plurality of images from the second memory; and a first determining subunit configured to determine a plurality of first feature data respectively corresponding to the plurality of images. a processing subunit configured to perform a first process on each of the plurality of first feature data to obtain second feature data having a predetermined data amount, the first process including: determining a first number based on the data amount of the first feature data and the predetermined data amount to obtain filler data consisting of a first number of filler elements; and splicing the first feature data and the filler data to obtain the second feature data; and a first writing subunit configured to sequentially write a plurality of second feature data respectively corresponding to the plurality of first feature data into the second memory in the predetermined order.

[0009] According to one aspect of the present disclosure, there is provided a chip including the image processing device described above.

[0010] According to one aspect of the present disclosure, there is provided an electronic device including at least one processor and a memory communicatively connected to the at least one processor, wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform the image processing method described above.

[0011] According to one aspect of the present disclosure, there is provided a non-transitory computer-readable storage medium having stored thereon computer instructions for causing a computer to perform the image processing method described above.

[0012] According to one aspect of the present disclosure, there is provided a computer program product including a computer program, wherein the computer program, when executed by a processor, is capable of implementing the image processing method described above.

[0013] According to one or more embodiments of the present disclosure, image processing efficiency can be improved. It should be understood that the material described in this section is not intended to identify key or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will be readily apparent from the following specification.

[0014] The drawings illustrate exemplary embodiments, constitute a part of the specification, and together with the written description serve to explain exemplary embodiments of the embodiments. The illustrated embodiments are for illustrative purposes only and do not limit the scope of the claims. In all drawings, the same reference numerals refer to similar, but not necessarily identical, elements. [Brief explanation of the drawings]

[0015] [Figure 1] 1 is a schematic diagram illustrating an example system capable of implementing various methods described herein, according to an example embodiment of the present disclosure. [Figure 2] 1 is a flowchart illustrating an image processing method according to an exemplary embodiment of the present disclosure. [Figure 3A] FIG. 2 is a schematic diagram illustrating an image processing process according to an exemplary embodiment of the present disclosure. [Figure 3B] FIG. 2 is a schematic diagram illustrating an image processing process according to an exemplary embodiment of the present disclosure. [Figure 4] FIG. 1 is a block diagram illustrating an image processing device according to an exemplary embodiment of the present disclosure. [Figure 5] FIG. 1 is a block diagram illustrating an exemplary electronic device that can be used to implement embodiments of the present disclosure. DETAILED DESCRIPTION OF THE INVENTION

[0016]

[0023] The following describes exemplary embodiments of the present disclosure in conjunction with the drawings. For ease of understanding, various details of the embodiments of the present disclosure are included therein, but they should be considered merely exemplary. Therefore, it should be recognized by those skilled in the art that various changes and modifications can be made to the embodiments described herein without departing from the scope of the present disclosure. Similarly, for clarity and conciseness, the following description omits descriptions of known functions and structures.

[0017] In this disclosure, unless otherwise specified, the use of terms such as "first," "second," etc. to describe various elements is not intended to limit the location, timing, or importance of these elements. Such terms are used only to distinguish one element from another. In some instances, a first element and a second element may refer to the same instance of the element, or in some cases, may refer to different instances based on the context.

[0018] The terms used in the description of various examples of the present disclosure are intended only to describe particular examples and are not intended to be limiting. Unless the context clearly indicates otherwise, and unless a specific number of elements is not limited, the element may be one or more. Furthermore, as used in this disclosure, the term "and / or" covers any one of the listed items and all possible combinations.

[0019] In the related technology, when it is necessary to use a processing unit to extract feature data of multiple images, the multiple images must first be transferred from a first memory for storing the original images to a second memory that can directly interact with the processing unit, and each time the calculation of the feature data of an image is completed, it must be directly written back to the first memory, which leads to fragmentation of request writes and interruptions and delays in data read and write.

[0020] Based on this, the present disclosure provides an image processing method, which, when using a processing unit to extract feature data of multiple images, converts the first feature data of each image into second feature data having a predetermined data amount, thereby realizing length alignment of the feature data; and further, temporarily stores the multiple feature data in a second memory, and when the temporarily stored data reaches a certain length, writes it back to the first memory in a unified manner, effectively reducing data write operations and improving image processing efficiency; and accurately dividing the temporarily stored feature data based on the predetermined data amount, thereby avoiding the temporary storage operations from affecting the accuracy of image processing.

[0021] Hereinafter, embodiments of the present disclosure will be described in detail with reference to the drawings.

[0022] 1 illustrates a schematic diagram of an exemplary system 100 in which various methods and apparatus described herein may be implemented, according to embodiments of the present disclosure. Referring to FIG. 1, the system 100 includes one or more client devices 101, 102, 103, 104, 105, and 106, a server 120, and one or more communication networks 110 coupling the one or more client devices to the server 120. The client devices 101, 102, 103, 104, 105, and 106 may be configured to run one or more applications.

[0023] In an embodiment of the present disclosure, the server 120 may execute one or more services or software applications that enable image processing methods to be implemented.

[0024] In some embodiments, server 120 may also provide other services or software applications, which may include non-virtualized and virtualized environments. In some embodiments, these services may be provided as web-based or cloud services, for example, provided to users of client devices 101, 102, 103, 104, 105, and / or 106 in a Software as a Service (SaaS) model.

[0025] In the configuration shown in FIG. 1 , server 120 may include one or more assemblies that implement the functionality performed by server 120. These assemblies may include software assemblies, hardware assemblies, or a combination thereof, executable on one or more processors. Users operating client devices 101, 102, 103, 104, 105, and / or 106 may, in turn, utilize one or more client applications to interact with server 120 to utilize services provided by these assemblies. It should be understood that a variety of different system configurations are possible and may differ from system 100. Thus, FIG. 1 is intended to be illustrative of an example system for implementing various methods described herein and is not intended to be limiting.

[0026] A user can submit images to be processed using client devices 101, 102, 103, 104, 105, and / or 106. The client devices can provide an interface through which a user of the client device interacts with the client device. The client devices can also output information to the user through the interface. Although only six client devices are shown in FIG. 1, one skilled in the art will understand that the present disclosure can support any number of client devices.

[0027] Client devices 101, 102, 103, 104, 105, and / or 106 may include various types of computing devices, such as portable handheld devices, general-purpose computers (e.g., personal computers or laptops), workstation computers, wearable devices, smart screen devices, self-service terminal devices, service robots, gaming systems, thin clients, various messaging devices, sensors, or other sensing devices. These computing devices may run various types and versions of software applications and operating systems, such as Microsoft Windows, Apple iOS, UNIX-like operating systems, Linux or Linux-like operating systems (e.g., Google Chrome OS), or include various mobile operating systems, such as Microsoft Windows Mobile OS, iOS, Windows Phone, and Android. Portable handheld devices may include mobile phones, intelligent phones, tablets, personal digital assistants (PDAs), and the like. Wearable devices may include head-mounted displays (e.g., smart glasses) and other devices. Gaming systems may include various handheld gaming devices, Internet-enabled gaming devices, and the like. The client device may run a variety of applications, such as Internet-related applications, communication applications (eg, email applications), and short message service (SMS) applications, and may use a variety of communication protocols.

[0028] Network 110 may be any type of network known to those skilled in the art, which may use any one of several available protocols (including, but not limited to, TCP / IP, SNA, IPX, etc.) to support data communications. By way of example, one or more networks 110 may be a local area network (LAN), an Ethernet-based network, a token ring, a wide area network (WAN), the Internet, a virtual network, a virtual private network (VPN), an intranet, an extranet, a blockchain network, a public switched telephone network (PSTN), an infrared network, a wireless network (e.g., Bluetooth, WIFI), and / or any combination of these and / or other networks.

[0029] Server 120 may include one or more general-purpose computers, dedicated server computers (e.g., PC (personal computer) servers, UNIX servers, mid-range servers), blade servers, mainframes, server clusters, or any other suitable arrangement and / or combination. Server 120 may also include one or more virtual machines running virtual operating systems or other computing architectures involving virtualization (e.g., one or more flexible pools of virtualized logical storage devices to maintain the server's virtual storage devices). In various embodiments, server 120 may run one or more services or software applications that provide the functionality described below.

[0030] The computing units in server 120 may run one or more operating systems, including any of the operating systems listed above and any commercial server operating system. Server 120 may also run any one of a variety of additional server and / or middle-tier applications, such as an HTTP server, an FTP server, a CGI server, a JAVA server, a database server, etc.

[0031] In some embodiments, server 120 may include one or more applications for analyzing and consolidating data feeds and / or event updates received from users of client devices 101, 102, 103, 104, 105, and 106. Server 120 may include one or more applications for displaying data feeds and / or real-time events via one or more display devices of client devices 101, 102, 103, 104, 105, and 106.

[0032] In some embodiments, server 120 may be a server in a distributed system or a server incorporating blockchain. Server 120 may be a cloud server, or an intelligent cloud computing server or intelligent cloud host equipped with artificial intelligence technology. A cloud server is a host product in a cloud computing service system that solves the drawbacks of traditional physical hosts and virtual private server (VPS) services, such as high management difficulty and poor business scalability.

[0033] System 100 may also include one or more databases 130. In some embodiments, these databases may be used to store data or other information. For example, one or more of databases 130 may be used to store information such as audio files or video files. Databases 130 may be located in a variety of locations. For example, a database used by server 120 may be local to server 120 or may be remote from server 120 and in communication with server 120 over a network or dedicated connection. Databases 130 may be of different types. In some embodiments, a database used by server 120 may be a relational database. One or more of these databases may store, update, and retrieve data from the databases in response to instructions.

[0034] In some embodiments, one or more of databases 130 may be used by an application to store data for the application. The databases used by the application may be different types of databases, such as, for example, a key-value repository, an object repository, a general-purpose repository supported by a file system, etc.

[0035] The system 100 of FIG. 1 may be configured and operated in a variety of ways to accommodate the various methods and apparatus described in this disclosure.

[0036] 2 is a flowchart illustrating an image processing method 200 performed by an image processing device including a processing unit, according to an exemplary embodiment of the present disclosure. As shown in FIG. 2, the method 200 includes: Step S210: reading a plurality of images arranged in a predetermined order from a first memory and writing them into a second memory readable and writable by the processing unit; a step S220 of reading the plurality of images from the second memory using the processing unit; a step S230 of determining a plurality of first feature data respectively corresponding to the plurality of images; a step S240 of performing a first process on each of the plurality of first feature data to acquire second feature data having a predetermined data amount, Here, the first process in step S240 is a step S241 of determining a first number based on the data amount of the first feature data and the predetermined data amount, thereby obtaining filling data consisting of the first number of filling elements; Step S240 includes step S242 of splicing the first feature data and the filler data to obtain the second feature data; a step S250 of sequentially writing a plurality of second feature data corresponding to the plurality of first feature data into the second memory in the predetermined order; and a step S260 of transferring the second feature data stored in the second memory to the first memory in response to determining that the amount of second feature data stored in the second memory has reached a predetermined threshold.

[0037] By applying the above method 200, when using a processing unit to extract feature data of multiple images, a feature data conversion operation is performed to achieve length alignment of the second feature data, and the second feature data having a uniform, predetermined data amount is temporarily stored in a second memory. When the temporarily stored data reaches a certain length, it is comprehensively written back to the first memory, effectively reducing data write operations, reducing delays in image processing, and improving efficiency.

[0038] In some examples, the first memory may be a large-capacity main memory, while the second memory may be a cache that can directly interact with the processing unit. By combining a first memory with a large capacity and slow read / write speed with a second memory with a small capacity and fast read / write speed, hardware resources can be used more efficiently and image processing efficiency can be improved.

[0039] In some examples, the processing unit in the image processing device may be a graphics processing unit (GPU), a central processing unit (CPU), or may be various types of logic devices such as computing chips, computing arrays, etc., although this disclosure is not limited thereto.

[0040] In some examples, the preset threshold may be manually set in advance according to needs, for example, according to the maximum storage capacity of the second memory or the transmission bandwidth of the data transmission link between the first memory and the second memory.

[0041] In some examples, the predetermined threshold for the amount of data of the second feature data may be in units of images, for example, may refer to the feature data of N images, i.e., corresponds to N times the predetermined amount of data.

[0042] In some examples, the second feature data may be composed of unit feature data having a preset size, in which case the preset threshold may be set to the unit feature data, i.e., the feature data of a single image may be divided into different batches for transport. By further subdividing the granularity of data sorting and transport, processing efficiency can be improved.

[0043] According to some embodiments, the method 200 further includes, in response to determining that a second processing should be performed on target feature data among the plurality of second feature data, determining the target feature data from the plurality of second feature data based on the predetermined data amount, and performing the second processing based on the target feature data. Therefore, when the plurality of second feature data can be centrally stored and transported, the spliced ​​and transported second feature data can be split based on the predetermined data amount, thereby obtaining accurate target feature data and ensuring the accuracy of image processing while improving efficiency.

[0044] According to some embodiments, each element of the first feature data is in a first data format, and the filler elements are in a second data format, the second data format having a bit width smaller than that of the first data format, and the first processing for each first feature data further includes, for each element of the first feature data, converting the element into a converted element having the second data format to obtain third feature data, and the splicing of the first feature data and the filler data to obtain the second feature data includes splicing the third feature data and the filler data to obtain the second feature data. This allows the first feature data to be further compressed and converted into data with a smaller bit width, thereby improving image processing efficiency. As can be seen, while maintaining the storage capacity of the second memory, a larger set of feature data can be temporarily stored in the second memory by simply performing alignment, splicing, and temporary storage on the compressed feature data, thereby further reducing the number of write operations and improving image processing efficiency.

[0045] In some examples, the third feature data may be bit data, that is, the feature map is converted into a bitmap, thereby achieving data compression.

[0046] According to some embodiments, the first memory includes a first sub-unit in which the plurality of images are stored and a second sub-unit, and the step of reading the plurality of images arranged in a predetermined order from the first memory and writing them to the second memory includes determining a plurality of sub-image data including first sub-image data and second sub-image data based on the plurality of images; writing the first sub-image data to the second sub-unit, so that the first sub-image data can be read from the second sub-unit; and writing the second sub-image data in response to determining that the first sub-image data has already been read. and writing data to the second sub-unit, wherein transferring the second feature data stored in the second memory to the first memory in response to determining that the data amount of the second feature data stored in the second memory has reached a predetermined threshold comprises transferring the second feature data stored in the second memory to the second sub-unit in response to determining that the data amount of the second feature data stored in the second memory has reached a predetermined threshold and in response to determining that the second process should be performed based on the second feature data stored in the second memory.

[0047] Thus, the first memory can be configured as a multi-level storage system, for example, including a first sub-unit and a second sub-unit, in which case the image data to be processed needs to be transported stepwise via the first sub-unit → the second sub-unit → the second memory, and when it is determined that the second feature data needs to be processed again, it can be temporarily stored in the second sub-unit that is closer to the second memory, thereby reducing the delay in data transport and further improving the efficiency of image processing.

[0048] According to some embodiments, transferring the second feature data stored in the second memory to the second sub-unit includes transferring the second feature data stored in the second memory to the target storage location in response to determining that a target storage location exists in the second sub-unit, where the target storage location does not contain sub-image data that has not yet been read, thereby enabling the second feature data to be temporarily stored in this location when it is determined that the data in the target storage location has already been read, in order to avoid data loss and ensure accuracy of image processing.

[0049] According to some embodiments, each image of the plurality of images includes a first image channel and a second image channel, and each feature data of the plurality of feature data includes first sub-feature data corresponding to the first image channel and second sub-feature data corresponding to the second image channel, wherein determining the plurality of sub-image data based on the plurality of images includes determining the first sub-image data based on the first image channel of each image of the plurality of images, and determining the second sub-image data based on the second image channel of each image of the plurality of images.

[0050] Corresponding to the embodiments in the disclosure, when extracting the first feature data in units of image channels, the image data is divided according to the dimension of the channel, which is equivalent to performing the channel extraction operation between the first sub-unit and the second sub-unit, and the processing unit does not need to jump to read the channel data, thereby improving processing efficiency. Figures 3A-3B are schematic diagrams showing an image processing process according to an exemplary embodiment of the present disclosure.

[0051] In this example, the image processing method 200 can be applied to the inference or training process of an image processing model.

[0052] In some examples, the inference or training process of an image processing model can be viewed as a directed acyclic computational graph composed of multiple types of operators, including convolution operators, batch normalization operators, vector addition operators, activation operators, maximum value search operators, pooling operators, etc.

[0053] In some examples, a sequence consisting of a batch normalization operator, a vector addition operator, an activation operator, and a maximum value search operator connected in series appears frequently in a computation graph. Building a fusion operator based on a sequence consisting of the above four operators connected in series can reduce data read and write operations between operators and improve the execution efficiency of the computation graph.

[0054] In some examples, in the inference or training process of an image processing model, a calculation may be performed by invoking a fusion operator based on input task information. For example, when the input task information indicates that a calculation needs to be performed based on a sequence consisting of a batch normalization operator, a vector addition operator, an activation operator, and a maximum value search operator connected in series, the above four operators included in the fusion operator are invoked in sequence. When the input task information indicates that a calculation needs to be performed based only on a batch normalization operator and an activation operator, only the batch normalization operator and the activation operator of the fusion operator are invoked in sequence.

[0055] In some examples, the processing unit for performing the computational processes of image processing may include a cache corresponding to the aforementioned second memory that directly interacts with the processing unit. Furthermore, the first memory may be a multi-level storage system including a first sub-unit and a second sub-unit, i.e., corresponding to different levels of caches.

[0056] As shown in FIG. 3A, the input image data may be divided according to the dimension of the channel, and the sub-image data of multiple image channels may be copied from the first sub-unit to the second sub-unit each time according to the size of the second sub-unit in the first memory, so that the processing unit can read data from the second sub-unit to perform initial calculations. When it is determined that the intermediate results obtained by the initial calculations need to be further used in the remaining calculations, the intermediate results may be written back to the second sub-unit, and then the intermediate results may be directly read from the second sub-unit to perform the remaining calculations. This eliminates the need to re-transfer data from the first sub-unit to the second sub-unit, thereby improving image processing efficiency.

[0057] As shown in FIG. 3B, after using a processing unit to extract feature data of multiple images, the first feature data of each image can be spliced ​​with its corresponding filler data to achieve length alignment of the feature data, i.e., obtain second feature data with a uniform, preset data amount.

[0058] In some examples, the first feature data may be a feature map of the activation function output by the activation operator. Additionally, in one example, the first feature data may be further compressed and the entire activation function graph may be converted into a bitmap, thereby reducing the amount of data and improving image processing efficiency.

[0059] According to an aspect of the present disclosure, there is further provided an image processing device. Fig. 4 is a block diagram illustrating an image processing device 400 according to an exemplary embodiment of the present disclosure. As shown in Fig. 4, the device 400 includes: a processing unit 410; and a writing unit 420 configured to read a plurality of images arranged in a predetermined order from a first memory and write the images to a second memory readable and writable by the processing unit; and a transport unit 430 configured to transport the second feature data stored in the second memory to the first memory in response to determining that the amount of second feature data stored in the second memory has reached a predetermined threshold; Here, the processing unit 410: a reading subunit 411 configured to read the plurality of images from the second memory; a first determining subunit 412 configured to determine a plurality of first feature data corresponding to the plurality of images respectively; a processing subunit 413 configured to perform a first process on each of the plurality of first feature data to obtain second feature data having a predetermined data amount, the first process including: determining a first number based on the data amount of the first feature data and the predetermined data amount to obtain filler data consisting of a first number of filler elements; and splicing the first feature data and the filler data to obtain the second feature data; and a first writing subunit 414 configured to sequentially write, into the second memory, a plurality of second feature data corresponding to the plurality of first feature data, respectively, in the predetermined order.

[0060] According to some embodiments, each element in the first feature data is in a first data format, the filler element is in a second data format, and the bit width of the second data format is smaller than the bit width of the first data format, wherein the first processing for each first feature data further includes, for each element of the first feature data, converting the element into a converted element having the second data format, thereby obtaining third feature data, wherein splicing the first feature data and the filler data to obtain the second feature data includes splicing the third feature data and the filler data to obtain the second feature data.

[0061] According to some embodiments, the first memory includes a first sub-unit in which the plurality of images are stored and a second sub-unit, and the writing unit 420 includes a second determining sub-unit configured to determine a plurality of sub-image data including first sub-image data and second sub-image data based on the plurality of images, and a second writing sub-unit configured to write the first sub-image data to the second sub-unit so that the first sub-image data can be read from the second sub-unit, and further configured to write the second sub-image data to the second sub-unit in response to determining that the first sub-image data has already been read, wherein the transporting unit 430 is configured to transport the second feature data stored in the second memory to the second sub-unit in response to determining that the amount of data of the second feature data stored in the second memory has reached a predetermined threshold and in response to determining that the second processing should be performed based on the second feature data stored in the second memory.

[0062] According to some embodiments, the transport unit 430 is configured to transport second feature data stored in the second memory to the target storage location in response to determining that a target storage location exists in the second sub-unit, wherein the target storage location does not include unreadable sub-image data.

[0063] According to some embodiments, each image of the plurality of images includes a first image channel and a second image channel, and each feature data of the plurality of feature data includes first sub-feature data corresponding to the first image channel and second sub-feature data corresponding to the second image channel, wherein the second determination subunit is configured to determine the first sub-image data based on the first image channel of each image of the plurality of images, and to determine the second sub-image data based on the second image channel of each image of the plurality of images.

[0064] According to some embodiments, the apparatus 400 further includes a determination unit configured to determine target feature data from the plurality of second feature data based on the predetermined data amount in response to determining that a second processing should be performed on target feature data among the plurality of second feature data, wherein the processing unit 410 is further configured to perform the second processing based on the target feature data.

[0065] It should be understood that the operation of each unit of the image processing apparatus 400 shown in Figure 4 may correspond to each step in the image processing method 200 described in Figure 2. Therefore, the operations, features, and advantages described above for the method 200 equally apply to the apparatus 400 and each unit included therein. For the sake of brevity, some operations, features, and advantages will not be described here.

[0066] According to one embodiment of the present disclosure, the image processing device 400 further includes a chip having the image processing device 400 described above.

[0067] According to one aspect of the present disclosure, there is further provided an electronic device including at least one processor and a memory communicatively connected to the at least one processor, wherein the memory stores instructions executable by the at least one processor, and the instructions, when executed by the at least one processor, enable the at least one processor to perform the image processing method described above.

[0068] According to one aspect of the present disclosure, there is further provided a non-transitory computer-readable storage medium having stored thereon computer instructions for causing a computer to perform the image processing method described above.

[0069] According to one aspect of the present disclosure, there is also provided a computer program product including a computer program which, when executed by a processor, implements the image processing method described above.

[0070] As shown in FIG. 5 , a block diagram of an electronic device 500 that can be used as a server or client of the present disclosure is described below as an example of a hardware device applicable to various aspects of the present disclosure. The electronic device represents various types of digital electronic computing devices, such as laptop computers, desktop computers, stages, personal digital assistants, servers, blade servers, mainframes, and other suitable computers. The electronic device may also represent various types of mobile devices, such as personal digital processing devices, cellular phones, smartphones, wearable devices, and other similar computing devices. The components, their connections, and their functions shown herein are merely exemplary and are not intended to limit the practice of the present disclosure as described and / or claimed herein.

[0071] 5, the device 500 includes a computing unit 501, which can perform various appropriate operations and processes according to a computer program stored in a read-only memory (ROM) 502 or loaded from a storage unit 508 into a random access memory (RAM) 503. The RAM 503 may store various programs and data required to operate the device 500. The computing unit 501, the ROM 502, and the RAM 503 are connected to each other via a bus 504. An input / output (I / O) interface 505 is also connected to the bus 504.

[0072] The components of device 500 are connected to I / O interface 505, which includes input unit 506, output unit 507, storage unit 508, and communication unit 509. Input unit 506 may be any type of device capable of inputting information into device 500. Input unit 506 can receive input numeric or character information and generate key signal input for user settings and / or function control of the electronic device, and may include, but is not limited to, a mouse, keyboard, touchscreen, trackboard, trackball, joystick, microphone, and / or remote control. Output unit 507 may be any type of device capable of presenting information, and may include, but is not limited to, a display, speaker, video / audio output terminal, vibrator, and / or printer. Storage unit 508 may include, but is not limited to, a magnetic disk, an optical disk, etc. The communication unit 509 enables the device 500 to exchange information / data with other devices via a computer network, e.g., the Internet, and / or various telecommunications networks, and may include, but is not limited to, a modem, a network card, an infrared communication device, a wireless communication transceiver, and / or a chipset, e.g., a Bluetooth™ device, an 802.11 device, a WiFi device, a WiMax device, a cellular communication device, and / or the like.

[0073] The computing unit 501 may be various general-purpose and / or special-purpose processing components having processing and computing capabilities. Some examples of the computing unit 501 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units that execute machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 501 performs each of the methods and processes described above, such as image processing methods. For example, in some embodiments, the image processing methods may be embodied as computer software programs tangibly contained in a machine-readable medium, such as the storage unit 508. In some embodiments, some or all of the computer programs may be loaded and / or installed into the device 500 via the ROM 502 and / or the communication unit 509. When the computer programs are loaded into the RAM 503 and executed by the computing unit 501, they may perform one or more steps of the image processing methods described above. Alternatively, in other embodiments, the computing unit 501 may be configured to perform the image processing method in any other suitable manner (eg, by firmware).

[0074] Various embodiments of the systems and techniques described herein may be implemented in digital electronic circuitry systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on a chip (SOCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include: embodied in one or more computer programs that may be executed and / or interpreted by a programmable system including at least one programmable processor, which may be a special purpose or general purpose programmable processor, and that may receive data and instructions from, and transmit data and instructions to, a storage system, at least one input device, and at least one output device.

[0075] These program codes may be provided to a processor or controller of a general purpose computer, special purpose computer or other programmable data processing apparatus, so that when the program code is executed by the processor or controller, the functions / operations specified in the flowcharts and / or block diagrams are performed. The program code may be entirely executed on a machine, partially executed on a machine, partially executed on a machine and partially executed on a remote machine as a separate software package, or entirely executed on a remote machine or server.

[0076] In the context of this disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in connection with an instruction execution system, device, or apparatus. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or apparatus, or any suitable combination of the above. More specific examples of machine-readable storage media include one or more wire-based electrical connections, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0077] To provide for interaction with a user, the systems and techniques described herein may be implemented in a computer having a display device (e.g., a CRT (Cathode Ray Tube) or LCD (Liquid Crystal Display) monitor) for displaying information to a user, and a keyboard and pointing device (e.g., a mouse or trackball) by which a user may provide input to the computer. Other types of devices may also be used to provide for interaction with a user, for example, the feedback provided to the user may be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback), and the input from the user may be received in any form (including sound, speech, or tactile input).

[0078] The systems and techniques described herein may be implemented in a computing system including backstage components (e.g., as a data server), middleware components (e.g., as an application server), front-end components (e.g., a user computer having a graphical user interface or a web browser through which a user can interact with the system or technique implementation), or any combination of backstage components, middleware components, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communications network). Examples of communications networks include a local area network (LAN), a wide area network (WAN), the Internet, and a blockchain network.

[0079] The computer system may include a client and a server. The client and server are generally remote from each other and typically interact via a communication network. The client-server relationship is created by running computer programs on the corresponding computers that have the client-server relationship with each other. The server may be a cloud server, a server in a distributed system, or a server incorporating a blockchain.

[0080] It should be understood that steps may be reordered, added, or deleted using the various types of flows shown above. For example, the steps described in this disclosure may be performed in parallel, sequentially, or in a different order, and this specification is not limited thereto, as long as the technical solutions disclosed in this disclosure can achieve the desired results.

[0081] Although embodiments or examples of the present disclosure have been described with reference to the drawings, it should be understood that the above-described methods, systems, and devices are merely exemplary embodiments or examples, and that the scope of the present invention is not limited by these embodiments or examples. Various elements in the embodiments or examples may be omitted or replaced by equivalent elements. Furthermore, each step may be performed in a different order from the order described in this disclosure. Furthermore, various elements in the embodiments or examples may be combined in various ways. In essence, as technology evolves, many elements described herein may be replaced by equivalent elements that appear later in this disclosure.

Claims

1. 1. An image processing method performed by an image processing device, the image processing device including a processing unit, the method comprising: reading a plurality of images arranged in a predetermined order from a first memory and writing the images into a second memory readable and writable by the processing unit; reading the plurality of images from the second memory using the processing unit; determining a plurality of first feature data corresponding to the plurality of images; a first process is performed on each of the plurality of first feature data to obtain second feature data having a predetermined data amount, the first process including: determining a first number based on the data amount of the first feature data and the predetermined data amount, thereby obtaining fill data consisting of the first number of fill elements; splicing the first feature data and the filler data to obtain the second feature data; sequentially writing a plurality of second feature data corresponding to the plurality of first feature data, respectively, into the second memory in the predetermined order; and transferring the second feature data stored in the second memory to the first memory in response to determining that the amount of second feature data stored in the second memory has reached a predetermined threshold.

2. Each element in the first feature data is in a first data format, and the filler element is in a second data format, and a bit width of the second data format is smaller than a bit width of the first data format, and the first processing for each first feature data includes: further comprising, for each element of the first feature data, converting the element into a conversion element having the second data format to obtain third feature data; Here, splicing the first feature data and the filling data to obtain the second feature data includes: The method of claim 1 , comprising splicing the third feature data and the filler data to obtain the second feature data.

3. The first memory includes a first sub-unit and a second sub-unit in which the plurality of images are stored, and reading the plurality of images arranged in a predetermined order from the first memory and writing them to the second memory includes: determining a plurality of sub-image data including first sub-image data and second sub-image data based on the plurality of images; The first sub-image data is written to the second sub-unit, and the first sub-image data can be read from the second sub-unit; and writing the second sub-image data to the second sub-unit in response to determining that the first sub-image data has already been read; Here, transferring the second feature data stored in the second memory to the first memory in response to determining that the amount of the second feature data stored in the second memory has reached a preset threshold value includes:

3. The method of claim 1, further comprising: transferring the second feature data stored in the second memory to the second sub-unit in response to determining that the amount of data of the second feature data stored in the second memory has reached a predetermined threshold and in response to determining that the second processing should be performed based on the second feature data stored in the second memory.

4. transferring the second feature data stored in the second memory to the second sub-unit; 4. The method of claim 3, further comprising, in response to determining that a target storage location exists in the second sub-unit, transferring second feature data stored in the second memory to the target storage location, wherein the target storage location does not contain sub-image data that has not yet been read.

5. Each image of the plurality of images includes a first image channel and a second image channel, and each feature data of the plurality of feature data includes first sub-feature data corresponding to the first image channel and second sub-feature data corresponding to the second image channel; Here, determining the plurality of sub-image data based on the plurality of images includes: determining the first sub-image data based on a first image channel of each image of the plurality of images; and determining the second sub-image data based on a second image channel of each image of the plurality of images.

6. determining the target feature data from the plurality of second feature data based on the predetermined data amount in response to determining that a second process should be performed on target feature data among the plurality of second feature data; The method of any one of claims 1 to 5, further comprising: performing the second processing based on the target feature data.

7. An image processing device, a processing unit; a writing unit configured to read a plurality of images arranged in a predetermined order from a first memory and write the images to a second memory readable and writable by the processing unit; and a transport unit configured to transport the second feature data stored in the second memory to the first memory in response to determining that the amount of second feature data stored in the second memory has reached a predetermined threshold, wherein the processing unit: a reading subunit configured to read the plurality of images from the second memory; a first determination subunit configured to determine a plurality of first feature data corresponding to the plurality of images, respectively; a processing subunit configured to perform a first process on each of the plurality of first feature data to obtain second feature data having a predetermined data amount, The first process includes: determining a first number based on the data amount of the first feature data and the predetermined data amount, thereby obtaining fill data consisting of the first number of fill elements; a processing subunit including splicing the first feature data and the filler data to obtain the second feature data; a first writing subunit configured to sequentially write a plurality of second feature data corresponding to the plurality of first feature data, respectively, into the second memory in the predetermined order.

8. Each element in the first feature data is in a first data format, and the filler element is in a second data format, and a bit width of the second data format is smaller than a bit width of the first data format, and the first processing for each first feature data includes: further comprising, for each element of the first feature data, converting the element into a conversion element having the second data format to obtain third feature data; Here, splicing the first feature data and the filling data to obtain the second feature data includes: The apparatus of claim 7 , further comprising splicing the third feature data and the filler data to obtain the second feature data.

9. The first memory includes a first sub-unit and a second sub-unit in which the plurality of images are stored, and the writing unit a second determination subunit for determining a plurality of sub-image data, including first sub-image data and second sub-image data, based on the plurality of images; configured to write the first sub-image data to the second sub-unit so that the first sub-image data can be read from the second sub-unit; and a second writing sub-unit configured to write the second sub-image data to the second sub-unit in response to determining that the first sub-image data has already been read; 9. The device according to claim 7 or 8, configured to convey the second feature data stored in the second memory to the second sub-unit in response to determining that the amount of second feature data stored in the second memory has reached a predetermined threshold and in response to determining that the second processing should be performed based on the second feature data stored in the second memory.

10. The carrier unit is 10. The apparatus of claim 9, wherein the second sub-unit is configured to, in response to determining that a target storage location exists, convey second feature data stored in the second memory to the target storage location, wherein the target storage location does not contain sub-image data that has not yet been read.

11. Each image of the plurality of images includes a first image channel and a second image channel, and each feature data of the plurality of feature data includes first sub-feature data corresponding to the first image channel and second sub-feature data corresponding to the second image channel; wherein the second determination subunit: determining the first sub-image data based on a first image channel of each image of the plurality of images; 11. The apparatus of claim 9 or 10, configured to determine the second sub-image data based on a second image channel of each image of the plurality of images.

12. a determination unit configured to determine the target feature data from the plurality of second feature data based on the preset data amount in response to determining that a second process should be performed on target feature data among the plurality of second feature data; The apparatus according to any one of claims 7 to 11, wherein the processing unit is further configured to perform the second processing based on the target feature data.

13. A chip comprising the image processing device according to any one of claims 7 to 12.

14. An electronic device, at least one processor; and a memory communicatively coupled to the at least one processor, wherein: An electronic device, wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to cause the at least one processor to perform the method of any one of claims 1 to 6.

15. A non-transitory computer readable storage medium having stored thereon computer instructions, the computer instructions being used to cause a computer to perform the method of any one of claims 1 to 6.

16. A computer program product comprising a computer program which, when executed by a processor, implements the method of any one of claims 1 to 6.

Images