Image caching method, device, electronic device and storage medium

Abstract

The present disclosure provides an image caching method, device, electronic device and storage medium, the method comprising: using each training process in a plurality of training processes to read a group of images to obtain multiple groups of images; wherein the multiple training processes correspond to the multiple groups of images one by one; using a first training process in the multiple training processes to apply for a shared memory corresponding to the multiple groups of images, and sharing the applied shared memory to each training process in the multiple training processes that is different from the first training process; using each training process in the multiple training processes to cache a group of images read from the multiple groups of images to the shared memory, so that each training process in the multiple training processes can read the multiple groups of images from the shared memory during the execution of the neural network training step.

Description

Technical Field

[0001] The present disclosure relates to the field of computer vision technology, and in particular to an image caching method, device, electronic device and storage medium. Background Art

[0002] The training process of a neural network is divided into two parts: data processing and training. The data processing stage mainly includes two steps: reading images from the hard disk and preprocessing the images. Usually, in order to increase the training speed, multiple processes are started on a physical machine to use multiple graphics cards for training at the same time.

[0003] At present, when the data of a single image is relatively large, the time to read the image will be very long during the data processing stage. In this case, the training process may have used the previous image to train the neural network while the next image is still being read. It is impossible to run data processing and training in parallel. Image reading takes too long, resulting in low training efficiency of the neural network. Summary of the invention

[0004] The embodiments of the present disclosure are intended to provide an image caching method, device, electronic device, and storage medium.

[0005] The technical solution of the embodiment of the present disclosure is implemented as follows:

[0006] The present disclosure provides an image caching method, including:

[0007] Using each of the multiple training processes to read a group of images, to obtain multiple groups of images; wherein the multiple training processes correspond to the multiple groups of images one by one;

[0008] Using a first training process among the multiple training processes, applying for shared memory corresponding to the multiple groups of images, and sharing the applied shared memory to each training process among the multiple training processes that is different from the first training process;

[0009] Using each of the multiple training processes, a group of images read from the multiple groups of images is cached in the shared memory, so that each of the multiple training processes can read the multiple groups of images from the shared memory during the execution of the neural network training step.

[0010] In the above method, using each training process in a plurality of training processes to read a group of images to obtain a plurality of groups of images includes:

[0011] Using a second training process among the multiple training processes, reading an image path list recording a storage path of each image in the image data set, and broadcasting the image path list to each training process among the multiple training processes that is different from the second training process;

[0012] Using each of the multiple training processes, based on the image path list and in accordance with a corresponding preset image reading strategy, a group of images is read from the image data set to obtain the multiple groups of images.

[0013] In the above method, before using the first training process among the multiple training processes to apply for the shared memory corresponding to the multiple groups of images, the method further includes:

[0014] Using each of the multiple training processes, calculate a memory size required to support caching of a group of images read from the multiple groups of images, and obtain multiple memory sizes corresponding to the multiple groups of images one by one;

[0015] Using the first training process, summarizing the multiple memory sizes to obtain an overall memory size that supports storing the multiple groups of images;

[0016] The using the first training process among the multiple training processes to apply for the shared memory corresponding to the multiple groups of images includes:

[0017] The shared memory is applied for according to the overall memory size by utilizing the first training process.

[0018] In the above method, after using each of the multiple training processes to calculate the memory size required to cache a corresponding group of images in the multiple groups of images, and obtaining multiple memory sizes corresponding one-to-one to the multiple groups of images, the method further includes:

[0019] Each of the multiple training processes is used separately to aggregate the multiple memory sizes to obtain the overall memory size.

[0020] In the above method, the use of each of the multiple training processes to calculate the memory size required to support caching of a group of images read from the multiple groups of images to obtain multiple memory sizes corresponding to the multiple groups of images includes:

[0021] Using each of the plurality of training processes, obtaining shape information of a group of images read from the plurality of groups of images;

[0022] Using each of the multiple training processes, according to shape information of a group of images read from the multiple groups of images, a memory size required to support cache reading of a group of images is calculated to obtain the multiple memory sizes.

[0023] In the above method, after obtaining shape information of a group of images read from the multiple groups of images by using each of the multiple training processes, the method further includes:

[0024] The shape information of the multiple groups of images is summarized by using each of the multiple training processes to obtain an information summary result.

[0025] The present disclosure provides an image caching device, including:

[0026] A reading module, used for reading a set of images by using each of the multiple training processes to obtain multiple sets of images; wherein the multiple training processes correspond to the multiple sets of images one by one;

[0027] a processing module, configured to apply for a shared memory corresponding to the plurality of groups of images by using a first training process among the plurality of training processes, and share the applied shared memory with each training process among the plurality of training processes that is different from the first training process;

[0028] The cache module is used to cache a group of images read from the multiple groups of images to the shared memory using each of the multiple training processes, so that each of the multiple training processes can read the multiple groups of images from the shared memory during the execution of the neural network training step.

[0029] In the above-mentioned device, the reading module is specifically used to use the second training process among the multiple training processes to read the image path list that records the storage path of each image in the image data set, and broadcast the image path list to each training process among the multiple training processes that is different from the second training process; use each training process among the multiple training processes, based on the image path list, according to the corresponding preset image reading strategy, to read a group of images from the image data set to obtain the multiple groups of images.

[0030] In the above device, the processing module is further used to calculate the memory size required to support caching of a group of images read from the multiple groups of images by using each of the multiple training processes, and obtain multiple memory sizes corresponding to the multiple groups of images one by one; and to summarize the multiple memory sizes by using the first training process to obtain the overall memory size supporting storage of the multiple groups of images;

[0031] The processing module is specifically configured to utilize the first training process to apply for the shared memory according to the overall memory size.

[0032] In the above device, the processing module is further used to respectively use each training process in the multiple training processes to aggregate the multiple memory sizes to obtain the overall memory size.

[0033] In the above-mentioned device, the processing module is specifically used to use each of the multiple training processes to obtain shape information of a group of images read from the multiple groups of images; and use each of the multiple training processes to calculate the memory size required to support cache reading of a group of images based on the shape information of a group of images read from the multiple groups of images, so as to obtain the multiple memory sizes.

[0034] In the above device, the processing module is further used to respectively use each of the multiple training processes to summarize the shape information of the multiple groups of images to obtain an information summary result.

[0035] The present disclosure provides an electronic device, the electronic device comprising: a processor, a memory and a communication bus; wherein:

[0036] The communication bus is used to realize the connection and communication between the processor and the memory;

[0037] The processor is used to execute one or more programs stored in the memory to implement the above-mentioned image caching method.

[0038] An embodiment of the present disclosure provides a computer-readable storage medium, wherein the computer-readable storage medium stores one or more programs, and the one or more programs can be executed by one or more processors to implement the above-mentioned image caching method.

[0039] The disclosed embodiments provide an image caching method, device, electronic device and storage medium, the method comprising: using each of a plurality of training processes to read a group of images to obtain a plurality of groups of images; wherein the plurality of training processes correspond one to one to the plurality of groups of images; using the first training process of the plurality of training processes to apply for a shared memory corresponding to the plurality of groups of images, and sharing the applied shared memory to each of the plurality of training processes that is different from the first training process; using each of the plurality of training processes to cache a group of images read from the plurality of groups of images to the shared memory, so that each of the plurality of training processes can read the plurality of groups of images from the shared memory during the execution of the neural network training step. The technical solution provided by the disclosed embodiments utilizes the training process to pre-cache the images required to be applied in the neural network training stage into the shared memory, thereby improving the image reading speed and improving the neural network training efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] Figure 1A flowchart of an image caching method provided by an embodiment of the present disclosure;

[0041] Figure 2 A schematic diagram of an exemplary training process cache image provided by an embodiment of the present disclosure;

[0042] Figure 3 A schematic diagram of the structure of an image caching device provided by an embodiment of the present disclosure;

[0043] Figure 4 A schematic diagram of the structure of an electronic device provided in an embodiment of the present disclosure. DETAILED DESCRIPTION

[0044] In order to make the purpose, technical solutions and advantages of the embodiments of the present disclosure clearer, the technical solutions in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the drawings in the embodiments of the present disclosure. Obviously, the described embodiments are part of the embodiments of the present disclosure, rather than all of the embodiments. The following embodiments are used to illustrate the present disclosure, but are not used to limit the scope of the present disclosure. Based on the embodiments in the present disclosure, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present disclosure.

[0045] In the following description, reference is made to “some embodiments”, which describe a subset of all possible embodiments, but it will be understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0046] It should be pointed out that the terms "first\second\third" involved in the embodiments of the present disclosure are merely used to distinguish similar objects and do not represent a specific ordering of the objects. It can be understood that the preset order or sequence of "first\second\third" can be interchanged where permitted, so that the embodiments of the present disclosure described here can be implemented in an order other than that illustrated or described here.

[0047] The embodiment of the present disclosure provides an image caching method, the execution subject of which may be an image caching device, for example, the image caching method may be executed by a terminal device or a server or other electronic device, wherein the terminal device may be a user equipment (UE), a mobile device, a user terminal, a cellular phone, a cordless phone, a personal digital assistant (PDA), a handheld device, a computing device, a vehicle-mounted device, a wearable device, etc. In some possible implementations, the image caching method may be implemented by a processor calling a computer-readable instruction stored in a memory.

[0048] An embodiment of the present disclosure provides an image caching method. Figure 1 The following is a flow chart of an image caching method provided by an embodiment of the present disclosure. Figure 1 As shown, in the embodiment of the present disclosure, the image caching method mainly includes the following steps:

[0049] S101. Using each of a plurality of training processes to read a group of images, a plurality of groups of images are obtained; wherein the plurality of training processes correspond to the plurality of groups of images one by one.

[0050] In an embodiment of the present disclosure, the image cache device uses multiple training processes to read multiple groups of images.

[0051] It should be noted that in the embodiment of the present disclosure, multiple groups of images are actually stored on a hard disk, and the image cache device uses multiple training processes to read a group of images for each training process, that is, multiple training processes correspond to multiple groups of images one by one. The specific number of training processes and the images read by each training process can be set according to actual needs and application scenarios, and the embodiment of the present disclosure does not limit them.

[0052] It should be noted that, in the embodiments of the present disclosure, for each of the multiple training processes, a neural network training step can be executed to train the neural network.

[0053] It should be noted that in the embodiments of the present disclosure, a group of images read by each training process may include one or more frames of images. The specific number of images contained in each group of images can be set according to actual needs, and the embodiments of the present disclosure are not limited.

[0054] Specifically, in an embodiment of the present disclosure, an image processing device uses each training process in a plurality of training processes to read a group of images to obtain multiple groups of images, including: using a second training process in the plurality of training processes to read an image path list that records the storage path of each image in an image data set, and broadcasting the image path list to each training process in the plurality of training processes that is different from the second training process; using each training process in the plurality of training processes to read a group of images from the image data set based on the image path list and in accordance with a corresponding preset image reading strategy to obtain multiple groups of images.

[0055] It should be noted that in the embodiment of the present disclosure, the multiple training processes include a first training process, wherein the first training process is used to implement the application and sharing of shared memory, which will be described in detail in the subsequent steps, and the second training process can be the same training process as the first training process, and of course, it can also be any training process among the multiple training processes that is different from the first training process. The specific second training process can be set according to actual needs and application scenarios, and the embodiment of the present disclosure does not limit it.

[0056] It can be understood that in the embodiments of the present disclosure, the image path list records the storage path of each image in the image data set, and the image caching device uses the second training process to read the image path list and broadcasts it to each training process in the multiple training processes that is different from the second training process, thereby enabling each training process in the multiple training processes to know the storage path of each image in the image data set. In this way, the image caching device can use each training process in the multiple training processes to read a group of images based on the image path list.

[0057] It should be noted that, in the embodiment of the present disclosure, each of the multiple training processes is set with a corresponding preset image reading strategy, indicating a group of images that the training thread needs to read from the image data set, that is, each training thread actually reads part of the images in the image data set. For example, the multiple training processes are three training processes, specifically including: training process 1, training process 2 and training process 3, wherein the preset image reading strategy corresponding to training process 1 is to read the first third of the images recorded in the image path list, that is, the group of images corresponding to training process 1 is the first third of the images recorded in the image path list of the image data set, the preset image reading strategy corresponding to training process 2 is to read the middle third of the images recorded in the image path list, and the preset image reading strategy corresponding to training process 3 is to read the last third of the images recorded in the image path list, that is, using three training processes, the reading of all images in the image data set can be realized. The specific preset image reading strategy corresponding to each training process can be set according to actual needs and application scenarios, and the embodiment of the present disclosure is not limited.

[0058] It should be noted that, in the embodiments of the present disclosure, the multiple groups of images may cover the entire image data set, and of course, may also cover only some specific images in the image data set, which is not limited in the embodiments of the present disclosure.

[0059] S102: using a first training process among the multiple training processes to apply for shared memory corresponding to the multiple groups of images, and sharing the applied shared memory with each training process among the multiple training processes that is different from the first training process.

[0060] In an embodiment of the present disclosure, when the image caching device reads multiple groups of images using multiple training processes, it uses the first training process among the multiple training processes to apply for shared memory corresponding to the multiple groups of images, and shares the applied shared memory to each training process among the multiple training processes that is different from the first training process.

[0061] It should be noted that in the embodiments of the present disclosure, the first training process can be any one of the multiple training processes, or it can be a specific one of the pre-set multiple training processes. It can be the same training process as the above-mentioned second training process, or it can be a different training process from the above-mentioned second training process, and the embodiments of the present disclosure are not limited to this.

[0062] It should be noted that in the embodiment of the present disclosure, after the image cache device applies for the shared memory using the first training process, it can share the shared memory with other training processes. Specifically, the image cache device can use the first training process to send a descriptor pointing to the shared memory to other training processes, and other training processes can obtain the specific shared memory based on the descriptor.

[0063] Specifically, in an embodiment of the present disclosure, before the image caching device uses the first training process among multiple training processes to apply for shared memory corresponding to multiple groups of images, the following steps can be performed: using each training process among the multiple training processes to calculate the memory size required to support caching a group of images read from the multiple groups of images, and obtain multiple memory sizes corresponding one-to-one to the multiple groups of images; using the first training process to aggregate the multiple memory sizes to obtain an overall memory size that supports storing the multiple groups of images; accordingly, the image caching device uses the first training process among the multiple training processes to apply for shared memory corresponding to the multiple groups of images, including: using the first training process to apply for shared memory according to the overall memory size.

[0064] It can be understood that in the embodiments of the present disclosure, when each training process reads a group of images, it can calculate the memory size required to cache the group of images, which is actually calculating the image data size of the group of images.

[0065] It should be noted that in the embodiments of the present disclosure, multiple training processes can communicate and interact with each other. Each training process different from the first training process can notify the first training process of the calculated memory size of a group of images supporting cache reading. The first training process can summarize the multiple memory sizes corresponding to the multiple groups of images, thereby obtaining the overall memory size supporting the storage of the multiple groups of images, and then apply for a shared memory matching the overall memory size, that is, the shared memory corresponding to the multiple groups of images. The overall memory size is actually the size of the image data volume of the multiple groups of images.

[0066] Specifically, in an embodiment of the present disclosure, an image caching device uses each of a plurality of training processes to calculate a memory size required to support caching a group of images read from a plurality of groups of images, and obtains a plurality of memory sizes corresponding one-to-one to the plurality of groups of images, including: using each of a plurality of training processes to obtain shape information of a group of images read from a plurality of groups of images; using each of a plurality of training processes to calculate a memory size required to support caching a group of images read based on the shape information of a group of images read from a plurality of groups of images, and obtains a plurality of memory sizes.

[0067] It should be noted that in the embodiments of the present disclosure, the image caching device can utilize each of the multiple training processes to obtain the shape information of a set of images read, wherein for a frame of image, its shape information may include information such as the data type of the image, and each training process can calculate the memory size required to support cache reading of a set of images based on the shape information of the corresponding set of images.

[0068] Specifically, in an embodiment of the present disclosure, the image caching device uses each of the multiple training processes to calculate the memory size required to support caching a group of images read from multiple groups of images, and after obtaining multiple memory sizes corresponding one-to-one to the multiple groups of images, the following steps can also be performed: use each of the multiple training processes separately, summarize the multiple memory sizes, and obtain the overall memory size.

[0069] It should be noted that in the embodiments of the present disclosure, while the image cache device uses each training process to calculate the memory size required to support cache reading for a group of images, it can also use each training process to summarize multiple memory sizes separately, so that each training process can know the size of multiple groups of images.

[0070] Specifically, in an embodiment of the present disclosure, after the image caching device uses each of multiple training processes to obtain shape information of a group of images read from multiple groups of images, it can also perform the following steps: use each of the multiple training processes to summarize the shape information of multiple groups of images and obtain information summary results.

[0071] It should be noted that in the embodiments of the present disclosure, when the image caching device uses each training process to obtain the shape information of a group of images read, it can also use each training process to summarize the shape information of multiple groups of images respectively. In this way, each training process can obtain the shape information of multiple groups of images. In this way, when each training process subsequently reads images from the shared memory, it can select specific images based on the obtained shape information and then read them.

[0072] S103, using each of the multiple training processes to cache a group of images read from the multiple groups of images into a shared memory, so that each of the multiple training processes can read the multiple groups of images from the shared memory during the execution of the neural network training step.

[0073] In an embodiment of the present disclosure, after the image caching device uses the first training process to share the shared memory with other training processes in multiple training processes, that is, the multiple training processes all know the shared memory, so that each training process in the multiple training processes can be used to cache a group of images read from multiple groups of images into the shared memory.

[0074] It can be understood that in the embodiment of the present disclosure, the image caching device uses each of the multiple training processes to cache a group of images read from the multiple groups of images to the shared memory, which actually achieves caching all the multiple groups of images into the shared memory. For each training process, it knows the shared memory that caches the multiple groups of images. Therefore, during the subsequent execution of the neural network training step, the image can be read from the shared memory, and specifically, any image in the multiple groups of images cached in the shared memory can be read for neural network training, thereby improving the image reading speed, and correspondingly, the neural network training speed is also improved.

[0075] It should be noted that in the embodiments of the present disclosure, the above-mentioned image caching method is preferably applicable to neural network training, where the amount of single-frame image data used for training is large, but the total image data of the image is small, that is, the application scenario with a small number of images.

[0076] Figure 2 A schematic diagram of an exemplary training process cache image provided by an embodiment of the present disclosure. Figure 2 As shown, training process 1 reads the image path list and then broadcasts this path list to training process 2 and training process 3. Each training process determines the set of images it needs to read based on the image path list and reads the corresponding set of images from the hard disk. Each training process calculates the memory size required to cache these images of this process based on the images read, and then summarizes them, so that all training processes know the size of the entire image data set. Training process 1 applies for the corresponding shared memory based on the summarized overall memory size and shares this shared memory with other processes. Finally, each process caches the images it has read before to the corresponding position of this shared memory. Each subsequent process can read the images of this shared memory during the training process.

[0077] The disclosed embodiment provides an image caching method, including: using each of a plurality of training processes to read a group of images to obtain a plurality of groups of images; wherein the plurality of training processes correspond one to one to the plurality of groups of images; using the first training process of the plurality of training processes to apply for a shared memory corresponding to the plurality of groups of images, and sharing the applied shared memory to each of the plurality of training processes that is different from the first training process; using each of the plurality of training processes to cache a group of images read from the plurality of groups of images to the shared memory, so that each of the plurality of training processes can read the plurality of groups of images from the shared memory during the execution of the neural network training step. The image caching method provided by the disclosed embodiment uses the training process to pre-cache the images required to be applied in the neural network training stage into the shared memory, thereby improving the image reading speed and improving the neural network training efficiency.

[0078] An embodiment of the present disclosure provides an image caching device. Figure 3 FIG. 1 is a schematic diagram of the structure of an image caching device provided by an embodiment of the present disclosure. Figure 3 As shown, the image caching device comprises:

[0079] A reading module 301 is used to read a group of images using each of the multiple training processes to obtain multiple groups of images; wherein the multiple training processes correspond to the multiple groups of images one by one;

[0080] The processing module 302 is used to apply for the shared memory corresponding to the plurality of groups of images by using the first training process among the plurality of training processes, and share the applied shared memory with each training process among the plurality of training processes that is different from the first training process;

[0081] The cache module 303 is used to utilize each of the multiple training processes to cache a group of images read from the multiple groups of images to the shared memory, so that each of the multiple training processes can read the multiple groups of images from the shared memory during the execution of the neural network training step.

[0082] In one embodiment of the present disclosure, the reading module 301 is specifically used to use the second training process among the multiple training processes to read the image path list that records the storage path of each image in the image data set, and broadcast the image path list to each training process among the multiple training processes that is different from the second training process; use each training process among the multiple training processes, based on the image path list, according to the corresponding preset image reading strategy, to read a group of images from the image data set to obtain the multiple groups of images.

[0083] In an embodiment of the present disclosure, the processing module 302 is further used to calculate, using each of the multiple training processes, a memory size required to support caching of a group of images read from the multiple groups of images, to obtain multiple memory sizes corresponding to the multiple groups of images one by one; and to summarize the multiple memory sizes using the first training process to obtain an overall memory size supporting storage of the multiple groups of images;

[0084] The processing module 302 is specifically configured to utilize the first training process to apply for the shared memory according to the overall memory size.

[0085] In an embodiment of the present disclosure, the processing module 302 is further configured to respectively utilize each of the multiple training processes to aggregate the multiple memory sizes to obtain the overall memory size.

[0086] In one embodiment of the present disclosure, the processing module 302 is specifically used to use each of the multiple training processes to obtain shape information of a group of images read from the multiple groups of images; use each of the multiple training processes to calculate the memory size required to support cache reading of a group of images based on the shape information of a group of images read from the multiple groups of images, and obtain the multiple memory sizes.

[0087] In an embodiment of the present disclosure, the processing module 302 is further configured to respectively utilize each of the multiple training processes to summarize the shape information of the multiple groups of images to obtain an information summary result.

[0088] An embodiment of the present disclosure provides an electronic device. Figure 4 The structure diagram of an electronic device provided by the embodiment of the present disclosure is shown in FIG. Figure 4 As shown, in an embodiment of the present disclosure, the electronic device includes: a processor 401, a memory 402 and a communication bus 403; wherein,

[0089] The communication bus 403 is used to realize the connection and communication between the processor 401 and the memory 402;

[0090] The processor 401 is used to execute one or more programs stored in the memory 402 to implement the above-mentioned image caching method.

[0091] The embodiment of the present disclosure provides a computer-readable storage medium, wherein the computer-readable storage medium stores one or more programs, and the one or more programs can be executed by one or more processors to implement the above-mentioned image caching method. The computer-readable storage medium can be a volatile memory (volatile memory), such as a random access memory (Random-Access Memory, RAM); or a non-volatile memory (non-volatile memory), such as a read-only memory (Read-Only Memory, ROM), a flash memory (flash memory), a hard disk (Hard Disk Drive, HDD) or a solid-state drive (Solid-State Drive, SSD); or it can be a respective device including one or any combination of the above-mentioned memories, such as a mobile phone, a computer, a tablet device, a personal digital assistant, etc.

[0092] Those skilled in the art will appreciate that the embodiments of the present disclosure may be provided as methods, systems, or computer program products. Therefore, the present disclosure may take the form of hardware embodiments, software embodiments, or embodiments combining software and hardware. Furthermore, the present disclosure may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage and optical storage, etc.) containing computer-usable program code.

[0093] The present disclosure is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present disclosure. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable signal processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable signal processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0094] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable signal processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0095] These computer program instructions may also be loaded onto a computer or other programmable signal processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for executing on the computer or other programmable device to implement the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0096] The above description is only a preferred embodiment of the present disclosure and is not intended to limit the protection scope of the present disclosure.

Claims

1. An image caching method, characterized in that: include: Using each of the multiple training processes to read a group of images, to obtain multiple groups of images; wherein the multiple training processes correspond to the multiple groups of images one by one; Using a first training process among the multiple training processes, applying for shared memory corresponding to the multiple groups of images, and sharing the applied shared memory to each training process among the multiple training processes that is different from the first training process; Using each of the plurality of training processes, cache a group of images read from the plurality of groups of images to the shared memory, so that each of the plurality of training processes can read the plurality of groups of images from the shared memory during the execution of the neural network training step; The method of using each training process in the plurality of training processes to read a group of images to obtain a plurality of groups of images includes: Using a second training process among the multiple training processes, reading an image path list recording a storage path of each image in the image data set, and broadcasting the image path list to each training process among the multiple training processes that is different from the second training process; Using each of the multiple training processes, based on the image path list and in accordance with a corresponding preset image reading strategy, a group of images is read from the image data set to obtain the multiple groups of images.

2. The method according to claim 1, characterized in that Before using the first training process among the multiple training processes to apply for the shared memory corresponding to the multiple groups of images, the method further includes: Using each of the multiple training processes, calculate a memory size required to support caching of a group of images read from the multiple groups of images, and obtain multiple memory sizes corresponding to the multiple groups of images one by one; Using the first training process, summarizing the multiple memory sizes to obtain an overall memory size that supports storing the multiple groups of images; The using the first training process among the multiple training processes to apply for the shared memory corresponding to the multiple groups of images includes: The shared memory is applied for according to the overall memory size by utilizing the first training process.

3. The method according to claim 2, characterized in that After calculating the memory size required to cache a corresponding group of images among the multiple groups of images by using each of the multiple training processes, and obtaining multiple memory sizes corresponding one-to-one to the multiple groups of images, the method further includes: Each of the multiple training processes is used separately to aggregate the multiple memory sizes to obtain the overall memory size.

4. The method according to claim 2, characterized in that: The method of calculating the memory size required to cache a group of images read from the multiple groups of images by using each of the multiple training processes to obtain multiple memory sizes corresponding to the multiple groups of images comprises: Using each of the plurality of training processes, obtaining shape information of a group of images read from the plurality of groups of images; Using each of the multiple training processes, according to shape information of a group of images read from the multiple groups of images, a memory size required to support cache reading of a group of images is calculated to obtain the multiple memory sizes.

5. The method according to claim 4, characterized in that After obtaining shape information of a group of images read from the multiple groups of images by using each of the multiple training processes, the method further includes: The shape information of the multiple groups of images is summarized by using each of the multiple training processes to obtain an information summary result.

6. An image caching device, characterized in that: include: A reading module, used for reading a group of images by using each of the multiple training processes to obtain multiple groups of images; wherein the multiple training processes correspond to the multiple groups of images one by one; a processing module, configured to apply for a shared memory corresponding to the plurality of groups of images by using a first training process among the plurality of training processes, and share the applied shared memory with each training process among the plurality of training processes that is different from the first training process; A cache module, used to cache a group of images read from the multiple groups of images to the shared memory by using each of the multiple training processes, so that each of the multiple training processes can read the multiple groups of images from the shared memory during the execution of the neural network training step; The reading module is specifically used to use the second training process among the multiple training processes to read the image path list that records the storage path of each image in the image data set, and broadcast the image path list to each training process among the multiple training processes that is different from the second training process; use each training process among the multiple training processes, based on the image path list, according to the corresponding preset image reading strategy, to read a group of images from the image data set to obtain the multiple groups of images.

7. The device according to claim 6, characterized in that The processing module is further configured to calculate, by using each of the multiple training processes, a memory size required to support caching of a group of images read from the multiple groups of images, and obtain multiple memory sizes corresponding one to one to the multiple groups of images; and by using the first training process, summarize the multiple memory sizes to obtain an overall memory size supporting storage of the multiple groups of images; The processing module is specifically configured to utilize the first training process to apply for the shared memory according to the overall memory size.

8. The device according to claim 7, characterized in that The processing module is further configured to respectively utilize each of the multiple training processes to aggregate the multiple memory sizes and obtain the overall memory size.

9. The device according to claim 7, characterized in that The processing module is specifically used to use each of the multiple training processes to obtain shape information of a group of images read from the multiple groups of images; use each of the multiple training processes to calculate the memory size required to support cache reading of a group of images based on the shape information of a group of images read from the multiple groups of images, and obtain the multiple memory sizes.

10. The device according to claim 9, characterized in that The processing module is further used to respectively use each of the multiple training processes to summarize the shape information of the multiple groups of images to obtain an information summary result.

11. An electronic device, characterized in that: The electronic device comprises: a processor, a memory and a communication bus; wherein, The communication bus is used to realize the connection and communication between the processor and the memory; The processor is used to execute one or more programs stored in the memory to implement the image caching method according to any one of claims 1 to 5.

12. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores one or more programs, and the one or more programs can be executed by one or more processors to implement the image caching method according to any one of claims 1 to 5.

Images