Visual language model deployment optimization method and device

Abstract

The embodiment of the invention provides a visual language model deployment optimization method and device, and relates to the technical field of image processing, and the method comprises the steps: obtaining an image through a first thread running in a CPU, and caching the image to a first queue; reading a predetermined number of frame images from the first queue through a second thread running in the GPU, inputting the read predetermined number of frame images into a specified visual language model in parallel, and caching an object detection result for each frame image output by the specified visual language model in a second queue; wherein the specified visual language model is a visual language model obtained by performing predetermined acceleration processing on the target visual language model, and the parallel input number of the specified visual language model for the image is set as a predetermined number; and reading the object detection result of each frame of image from the second queue through a third thread to obtain the object detection result of each frame of image. Through the scheme, the requirement of efficient object detection can be met.

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and in particular to a method and device for optimizing the deployment of a visual language model. Background Art

[0002] Object detection can identify specific objects in images or videos. With the continuous development of deep learning technology, visual language models are usually used to perform object detection on images.

[0003] In the related art, a visual language model is usually used to serially perform object detection on images, that is, object detection is first performed on an image, and after the object detection on the image is completed, object detection is performed on the next image.

[0004] In the scenario of object detection on a large number of images, the existing visual language model's method of performing object detection on images serially cannot meet the needs of efficient object detection.

[0005] Therefore, there is an urgent need for a visual language model deployment optimization method to meet the needs of efficient object detection. Summary of the invention

[0006] The purpose of the embodiment of the present invention is to provide a method and device for optimizing the deployment of a visual language model to meet the needs of efficient object detection. The specific technical solution is as follows:

[0007] In a first aspect, an embodiment of the present invention provides a method for optimizing deployment of a visual language model, the method comprising:

[0008] By running a first thread on the CPU, image acquisition is performed and each acquired frame of the image is cached in a first queue;

[0009] A predetermined number of frame images are read from the first queue by a second thread running on the GPU, and the read predetermined number of frame images are input in parallel to a specified visual language model running on the GPU, and the object detection results for each frame image output by the specified visual language model are cached in the second queue; wherein the specified visual language model is a visual language model obtained after a predetermined acceleration process is performed on the target visual language model, and the number of parallel inputs of the specified visual language model for images is set to the predetermined number;

[0010] The object detection result of each frame image is read from the second queue through the third thread to obtain the object detection result of each frame image.

[0011] Optionally, the step of inputting the read predetermined number of frame images in parallel to a specified visual language model running on a GPU includes:

[0012] Invoking the GPU to perform image preprocessing on the predetermined number of image frames;

[0013] The predetermined number of frames of images after image preprocessing are input into the designated visual language model in parallel.

[0014] Optionally, the target visual language model includes an image processing network and a text processing network, and the image processing network includes a feature extraction module, a feature fusion module and an object detection module;

[0015] The text processing network is used to extract features from the target text to obtain text features, and input the text features into the feature fusion module, wherein the target text is a text that represents the object detection requirements;

[0016] The feature extraction module is used to extract features from each frame of the input predetermined number of frames of images, obtain image features of the frame of image, and input the features to the feature fusion module;

[0017] The feature fusion module is used to fuse the image features of each frame of image received with the text features to obtain the fusion features corresponding to the frame of image and input the fusion features into the object detection module;

[0018] The object detection module is used to perform object detection according to the fusion features corresponding to each received frame image, and obtain the object detection result corresponding to the frame image.

[0019] Optionally, the specified input parameter of the image processing network of the specified visual language model is a dynamic parameter; wherein the specified input parameter is a parameter used to indicate the number of parallel image inputs;

[0020] The setting method of setting the number of parallel inputs of the specified visual language model to the predetermined number includes:

[0021] setting the number of specified input parameters of the image processing network to the predetermined number;

[0022] The characteristic dimension parameters of the text processing network are dynamic parameters; the method further includes:

[0023] The feature dimension parameter of the text processing network is set to a number matching the predetermined number; wherein the feature dimension parameter is a parameter used to indicate the number of dimensions of the extracted text features.

[0024] Optionally, the reading the object detection result of each frame image from the second queue to obtain the object detection result of each frame image includes:

[0025] Reading the object detection result of each frame image from the second queue;

[0026] The object detection result of each frame image is processed by the audio and video processing module to obtain a streaming result including the object detection result of each frame image.

[0027] In a second aspect, an embodiment of the present invention provides a visual language model deployment optimization device, the device comprising:

[0028] An acquisition module, used for acquiring images by running a first thread on the CPU and caching each acquired frame of the image into a first queue;

[0029] An input module, configured to read a predetermined number of frame images from the first queue through a second thread running on the GPU, input the read predetermined number of frame images in parallel to a specified visual language model running on the GPU, and cache the object detection results for each frame image output by the specified visual language model in the second queue; wherein the specified visual language model is a visual language model obtained after a predetermined acceleration process is performed on the target visual language model, and the number of parallel inputs of the specified visual language model for images is set to the predetermined number;

[0030] The reading module is used to read the object detection result of each frame image from the second queue through a third thread to obtain the object detection result of each frame image.

[0031] Optionally, the input module is specifically used to:

[0032] Invoking the GPU to perform image preprocessing on the predetermined number of image frames;

[0033] The predetermined number of frames of images after image preprocessing are input into the designated visual language model in parallel.

[0034] Optionally, the target visual language model includes an image processing network and a text processing network, and the image processing network includes a feature extraction module, a feature fusion module and an object detection module;

[0035] The text processing network is used to extract features from the target text to obtain text features, and input the text features into the feature fusion module, wherein the target text is a text that represents the object detection requirements;

[0036] The feature extraction module is used to extract features from each frame of the input predetermined number of frames of images, obtain image features of the frame of image, and input the features to the feature fusion module;

[0037] The feature fusion module is used to fuse the image features of each frame of image received with the text features to obtain the fusion features corresponding to the frame of image and input the fusion features into the object detection module;

[0038] The object detection module is used to perform object detection according to the fusion features corresponding to each received frame image, and obtain the object detection result corresponding to the frame image.

[0039] Optionally, the specified input parameter of the image processing network of the specified visual language model is a dynamic parameter; wherein the specified input parameter is a parameter used to indicate the number of parallel image inputs;

[0040] The setting method of setting the number of parallel inputs of the specified visual language model to the predetermined number includes:

[0041] setting the number of specified input parameters of the image processing network to the predetermined number;

[0042] The characteristic dimension parameters of the text processing network are dynamic parameters; the device also includes:

[0043] A setting module is used to set the feature dimension parameter of the text processing network to a number matching the predetermined number; wherein the feature dimension parameter is a parameter used to indicate the number of dimensions of the extracted text features.

[0044] Optionally, the reading module is specifically used to:

[0045] Reading the object detection result of each frame image from the second queue;

[0046] The object detection result of each frame image is processed by the audio and video processing module to obtain a streaming result including the object detection result of each frame image.

[0047] Beneficial effects of the embodiments of the present invention:

[0048] The visual language model deployment optimization method provided by the embodiment of the present invention acquires images through a first thread running on the CPU, and caches each frame of the acquired image in a first queue; through a second thread running on the GPU, a predetermined number of frame images are read from the first queue, and input in parallel to a specified visual language model running on the GPU, and the object detection results for each frame of the image output by the specified visual language model are cached in the second queue; and then through a third thread, the object detection results of each frame of the image are read from the second queue to obtain the object detection results of each frame of the image. Among them, the specified visual language model in the embodiment of the present invention is a visual language model obtained after a predetermined acceleration process is performed on the target visual language model, that is, the specified object detection model is a deployment optimized visual language model, and the number of parallel inputs of the specified visual language model for images is set to a predetermined number, so that the specified visual language model can perform efficient object detection on a predetermined number of frame images in parallel. The first thread of the present invention runs on the CPU, and the second thread runs on the GPU, so that the resources of the CPU and GPU are maximized. Through multiple threads, multiple queues, and a specified visual language model with accelerated processing, object detection can be performed on a predetermined number of frame images in parallel, so as to achieve the purpose of improving the efficiency of object detection and meet the needs of efficient object detection.

[0049] Of course, it is not necessary to achieve all of the advantages described above at the same time to implement any product or method of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings in the following description are only some embodiments of the present invention. For ordinary technicians in this field, other embodiments can also be obtained based on these drawings.

[0051] Figure 1 A flowchart of a visual language model deployment optimization method provided by an embodiment of the present invention;

[0052] Figure 2 Another schematic diagram of a flow chart of a visual language model deployment optimization method provided by an embodiment of the present invention;

[0053] Figure 3 A schematic diagram of a target visual language model without scheduled accelerated processing provided by an embodiment of the present invention;

[0054] Figure 4 A schematic diagram of a designated visual language model for predetermined accelerated processing provided by an embodiment of the present invention;

[0055] Figure 5A schematic diagram of the structure of a visual language model deployment optimization device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0056] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field based on the present invention belong to the scope of protection of the present invention.

[0057] With the continuous development of deep learning technology, visual language models have performed well in tasks such as image recognition, object detection, and text generation. As the model size grows, the demand for inference speed and computing resources of visual language models also increases accordingly, especially in scenarios where a large number of video streams need to be processed or processed in real time (that is, scenarios where object detection is required for a large number of images), where the demand for inference speed is higher.

[0058] When existing visual language models are deployed in practical applications, they usually perform object detection on images in series, and often encounter bottlenecks such as computational delays and insufficient throughput. Traditional CPU (Central Processing Unit) or GPU (Graphics Processing Unit) acceleration methods are difficult to fully utilize hardware performance when facing complex multimodal tasks, that is, they cannot fully utilize the CPU or GPU and cannot meet the needs of efficient object detection. Therefore, there is an urgent need for a deployment method that can effectively integrate multi-threading and multi-device parallel object detection on multiple frames of images, and utilize model optimization and acceleration methods to meet the needs of efficient object detection.

[0059] Based on this, an embodiment of the present invention provides a method and device for optimizing the deployment of a visual language model to meet the needs of efficient object detection.

[0060] The following first provides an exemplary introduction to a visual language model deployment optimization method provided by the present invention.

[0061] Among them, a visual language model deployment optimization method provided by an embodiment of the present invention can be applied to electronic devices. In specific applications, the electronic device can be a mobile phone, a computer, etc., and the present application does not limit the specific form of the electronic device. A visual language model deployment optimization method provided by an embodiment of the present invention can be applied to any scenario with object detection requirements, for example: scenarios for efficient object detection of a large number of images, etc., and the present invention does not limit this. That is, the visual language model in this application can be a model with object detection function.

[0062] Among them, an embodiment of the present invention provides a visual language model deployment optimization method, the method comprising:

[0063] By running a first thread on the CPU, image acquisition is performed and each acquired frame of the image is cached in a first queue;

[0064] A predetermined number of frame images are read from the first queue by a second thread running on the GPU, and the read predetermined number of frame images are input in parallel to a specified visual language model running on the GPU, and the object detection results for each frame image output by the specified visual language model are cached in the second queue; wherein the specified visual language model is a visual language model obtained after a predetermined acceleration process is performed on the target visual language model, and the number of parallel inputs of the specified visual language model for images is set to the predetermined number;

[0065] The object detection result of each frame image is read from the second queue through the third thread to obtain the object detection result of each frame image.

[0066] The visual language model deployment optimization method provided by the embodiment of the present invention acquires images through a first thread running on the CPU, and caches each frame of the acquired image in a first queue; through a second thread running on the GPU, a predetermined number of frame images are read from the first queue, and input in parallel to a specified visual language model running on the GPU, and the object detection results for each frame of the image output by the specified visual language model are cached in the second queue; and then through a third thread, the object detection results of each frame of the image are read from the second queue to obtain the object detection results of each frame of the image. Among them, the specified visual language model in the embodiment of the present invention is a visual language model obtained after a predetermined acceleration process is performed on the target visual language model, that is, the specified object detection model is a deployment optimized visual language model, and the number of parallel inputs of the specified visual language model for images is set to a predetermined number, so that the specified visual language model can perform efficient object detection on a predetermined number of frame images in parallel. The first thread of the present invention runs on the CPU, and the second thread runs on the GPU, so that the resources of the CPU and GPU are maximized. Through multiple threads, multiple queues, and a specified visual language model with accelerated processing, object detection can be performed on a predetermined number of frame images in parallel, so as to achieve the purpose of improving the efficiency of object detection and meet the needs of efficient object detection.

[0067] In the following, in conjunction with the accompanying drawings, a visual language model deployment optimization method provided by the present invention is exemplarily introduced.

[0068] like Figure 1 As shown, a visual language model deployment optimization method provided by an embodiment of the present invention may include the following steps:

[0069] S101: acquiring an image through a first thread running on a CPU and caching each acquired frame of the image into a first queue;

[0070] The present invention acquires images by running a first thread on the CPU, for example, by using the OpenCV (open source computer vision library) streaming technology to read images from a video through an independent first thread; since the first thread running on the CPU and the second thread running on the GPU are both independent threads, when a subsequent visual language model performs object detection, there will be no increasing delay due to the slow detection speed of the visual language model.

[0071] When acquiring images through the first thread, it is likely that factors such as network factors or hardware resource allocation (for example, the CPU allocated to the first thread may change, and other threads may snatch the CPU resources allocated to the first thread) may cause the first thread to acquire images at an unstable speed. In the present invention, after the first thread acquires the image, each frame of the acquired image can be cached in the first queue, so that the second thread can subsequently evenly acquire images from the first queue for object detection.

[0072] Among them, the first thread can cache each frame of image obtained to the first queue in frame order. Whenever the first thread obtains a frame of image, it writes the frame of image to the end of the first queue; and the first thread can control the length of the first queue. For example, the maximum number of images that the first queue can cache is 25 frames. Whenever there is a new frame of image that needs to be written to the first queue, if there is a frame of image that has been read by the second thread in the first queue, the read frame of image can be removed and the new frame of image can be written to the first queue. In this way, if the subsequent visual language model's object detection speed is slow, resulting in more and more images being read, the delay of the entire object detection process is reduced.

[0073] S102: reading a predetermined number of frame images from the first queue through a second thread running on the GPU, inputting the read predetermined number of frame images in parallel into a specified visual language model running on the GPU, and caching the object detection results for each frame image output by the specified visual language model in a second queue;

[0074] The designated visual language model is a visual language model obtained after a predetermined acceleration process is performed on the target visual language model, and the number of parallel inputs of the designated visual language model for images is set to the predetermined number;

[0075] The present invention can reasonably utilize the resources of the CPU and GPU. By running a second thread on the GPU, a predetermined number of frame images are read from the first queue, and the predetermined number of frame images are input in parallel to a specified visual language model running on the GPU, so that the specified visual language model performs object detection on the predetermined number of frame images in parallel, and outputs the object detection results of each frame image. Since the specified visual language model can perform object detection on the predetermined number of frame images in parallel, rather than uniformly serial object detection, in order to present the object detection results of each frame image at a uniform speed, the present invention caches the object detection results of each frame image through the second queue, and the object detection results of each frame image can be read from the second queue at a uniform speed later.

[0076] Similarly, the second thread can cache the object detection result of each frame of image to the second queue according to the frame sequence, and whenever the object detection results of a predetermined number of frames of image are obtained, the object detection result of the first frame of image in the frame sequence is written to the end of the second queue according to the frame sequence; and the second thread can control the length of the second queue, for example: the maximum number of object detection results of images that can be cached by the second queue is 3 frames (that is, the second queue can cache object detection results of a maximum of 3 frames of images), and whenever there is an object detection result of a new frame of image that needs to be written to the second queue, if there is an object detection result of a frame of image that has been read by the third thread in the second queue, the object detection result of the read frame of image can be eliminated, and the object detection result of the new frame of image can be written to the second queue.

[0077] It should be noted that the predetermined number is the number of images that can be detected in parallel by the specified visual language model in the present invention. The predetermined number can be 2, 4, 8, etc., and can be adjusted according to actual needs. The present invention does not limit this. In addition, the first queue and the second queue are queues for real-time continuous caching of images and object detection results of images, and have a certain queue length. When any queue is full, if an image or an object detection result of an image needs to be written into the queue, the earliest frame of image or the object detection result of a frame of image in the queue can be discarded.

[0078] Exemplarily, reading a predetermined number of image frames from the first queue includes:

[0079] Taking out a predetermined number of frame images as the head of the first queue from the first queue; wherein the arrangement order of the frame images in the first queue is: an arrangement order from the beginning to the end according to the frame sequence;

[0080] or,

[0081] A predetermined number of frame images are read from the first queue in frame sequence, and when there is a target frame image to be written into the first queue and the first queue is full, the frame image at the front of the frame sequence is removed from the first queue to write the target frame image into the first queue.

[0082] In the present invention, the arrangement order of each frame image in the first queue is the arrangement order from the beginning to the end according to the frame sequence. When the second thread reads a predetermined number of frame images from the first queue, it can directly take out the predetermined number of frame images as the head of the team from the first queue. The first thread can continue to write each frame image into the first queue, and the second thread can continue to take out the predetermined number of frame images as the head of the team from the first queue; or, the second thread can read a predetermined number of frame images from the first queue according to the frame sequence. At this time, it is only reading. After the reading is completed, the predetermined number of frame images still exist in the first queue. When the first thread needs to write a target frame image into the first queue, if the first queue is full, it is necessary to first remove the frame image belonging to the front of the frame sequence from the first queue (the front of the frame sequence has been read by the second thread), and then write the target frame image into the first queue. Similarly, when the third thread subsequently reads the object detection results of each frame image from the second queue, the object detection results of each frame image in the second queue are arranged in order from the beginning to the end according to the frame sequence, and the object detection results of each frame image can also be read in a similar manner. The object detection results of one frame image can be read continuously from the second queue, or the object detection results of multiple frames of images can be read at the same time, which will not be elaborated here.

[0083] In addition, in order to implement object detection on an image by using a specified visual language model, the image may be preprocessed first, wherein a predetermined number of frames of images read are input in parallel into a specified visual language model running on a GPU, including:

[0084] Invoking the GPU to perform image preprocessing on the predetermined number of image frames;

[0085] The predetermined number of frames of images after image preprocessing are input into the designated visual language model in parallel.

[0086] When a predetermined number of image frames are input in parallel to a specified object detection model running on a GPU, the GPU can be called to perform image preprocessing on the predetermined number of image frames, for example, adjusting the size and format of the images to adapt to the input of a specified visual language model, and then the predetermined number of image frames after image preprocessing are input in parallel to the specified visual language model to perform object detection in parallel.

[0087] Among them, image preprocessing can perform the same processing on a predetermined number of frames of images, which may include: adjusting the image width and height to a predetermined width and a predetermined height, and adjusting the image to an RGB format image to adapt to the input of a specified visual language model, which is not limited in the present invention.

[0088] S103: Reading the object detection result of each frame image from the second queue through the third thread to obtain the object detection result of each frame image.

[0089] The present invention can read the object detection results of each frame image from the second queue at a uniform speed through the third thread to obtain the object detection results of each frame image; then, the object detection results of each frame image can be uniformly presented in the push stream results (i.e., pushed to other receiving ends for display, etc.). The third thread can run on the CPU, which is not limited by the present invention.

[0090] Optionally, in an implementation manner, reading the object detection result of each frame image from the second queue to obtain the object detection result of each frame image includes:

[0091] Reading the object detection result of each frame image from the second queue;

[0092] The object detection result of each frame image is processed by the audio and video processing module to obtain a streaming result including the object detection result of each frame image.

[0093] After reading the object detection results of each frame of images from the second queue, the object detection results of each frame of images can be decoded, encoded or format converted through an audio and video processing module (for example, ffpmeg, an open source audio and video processing tool) to obtain a streaming result containing the object detection results of each frame of images, thereby adapting other receiving ends to display the object detection results of each frame of images.

[0094] The visual language model deployment optimization method provided by the embodiment of the present invention acquires images through a first thread running on the CPU, and caches each frame of the acquired image in a first queue; through a second thread running on the GPU, a predetermined number of frame images are read from the first queue, and input in parallel to a specified visual language model running on the GPU, and the object detection results for each frame of the image output by the specified visual language model are cached in the second queue; and then through a third thread, the object detection results of each frame of the image are read from the second queue to obtain the object detection results of each frame of the image. Among them, the specified visual language model in the embodiment of the present invention is a visual language model obtained after a predetermined acceleration process is performed on the target visual language model, that is, the specified object detection model is a deployment optimized visual language model, and the number of parallel inputs of the specified visual language model for images is set to a predetermined number, so that the specified visual language model can perform efficient object detection on a predetermined number of frame images in parallel. The first thread of the present invention runs on the CPU, and the second thread runs on the GPU, so that the resources of the CPU and GPU are maximized. Through multiple threads, multiple queues, and a specified visual language model with accelerated processing, object detection can be performed on a predetermined number of frame images in parallel, so as to achieve the purpose of improving the efficiency of object detection and meet the needs of efficient object detection.

[0095] Optionally, in another embodiment of the present invention, the target visual language model includes an image processing network and a text processing network, and the image processing network includes a feature extraction module, a feature fusion module, and an object detection module;

[0096] The text processing network is used to extract features from the target text to obtain text features, and input the text features into the feature fusion module, wherein the target text is a text that represents the object detection requirements;

[0097] The feature extraction module is used to extract features from each frame of the input predetermined number of frames of images, obtain image features of the frame of image, and input the features to the feature fusion module;

[0098] The feature fusion module is used to fuse the image features of each frame of image received with the text features to obtain the fusion features corresponding to the frame of image and input the fusion features into the object detection module;

[0099] The object detection module is used to perform object detection according to the fusion features corresponding to each received frame image, and obtain the object detection result corresponding to the frame image.

[0100] The target visual language model may include an image processing network and a text processing network. The text processing network may be used to extract features from the target text to obtain text features, and input the text features into a feature fusion module to act on the object detection of the image. The target text may be a text representing the object detection requirements, such as: "Where is the lion in the image?", "Where are the trees in the image?", etc.; and, for each frame image obtained from the same video stream, the target text used may be the same; or, the predetermined number of frame images may be a group of images, and each image in each group of images uses the same target text for object detection, and the target texts used by different groups of images may be different; or, the target texts used by different images may also be different, and the present invention does not limit this.

[0101] The image processing network includes a feature extraction module, a feature fusion module and an object detection module. The feature extraction module can perform feature extraction on each frame image of a predetermined number of input frames of images to obtain the image features of the frame image and input them into the feature fusion module; the feature fusion module fuses the image features and text features of the frame image to obtain the fusion features corresponding to the frame image and inputs them into the object detection module; the object detection module uses the fusion features corresponding to the frame image to perform object detection and obtain the object detection result corresponding to the frame image.

[0102] Among them, the text processing network in the target visual language model can extract features of the target text that represents the object detection requirements, and apply the extracted text features to the object detection process of the image to obtain the required object detection results for the image.

[0103] Optionally, the specified input parameter of the image processing network of the specified visual language model is a dynamic parameter; wherein the specified input parameter is a parameter used to indicate the number of parallel image inputs;

[0104] The setting method of setting the number of parallel inputs of the specified visual language model to the predetermined number includes:

[0105] setting the number of specified input parameters of the image processing network to the predetermined number;

[0106] The characteristic dimension parameters of the text processing network are dynamic parameters; the method further includes:

[0107] The feature dimension parameter of the text processing network is set to a number matching the predetermined number; wherein the feature dimension parameter is a parameter used to indicate the number of dimensions of the extracted text features.

[0108] The designated input parameter of the image processing network of the target visual language model is 1 and cannot be adjusted, so the target visual language model can perform serial object detection on one frame of image. In the present invention, in order to realize object detection on multiple frames of image in parallel, the designated input parameter is adjusted to a dynamic parameter, and the number of designated input parameters of the image processing network is set to a predetermined number, so that the designated visual language model obtained after the predetermined acceleration processing can perform object detection on a predetermined number of frames of image in parallel and efficiently.

[0109] In addition, in order to adapt to a predetermined number of frame images for parallel object detection, the feature dimension parameters of the text processing network can also be set to a number that matches the predetermined number, so that the dimension of the text features extracted by the text processing network is adapted to the predetermined number. For example, before setting, the text features of the text processing network for 1 frame image are 1×256 dimensions. After setting, the text features of the text processing network for a predetermined number of frame images are a predetermined number×256 dimensions (for example: 4×256, that is, the predetermined number can be 4).

[0110] In order to increase the speed of object detection, the method of performing a predetermined acceleration process on the target visual language model includes:

[0111] The feature extraction module, feature fusion module and object detection module in the image processing network are accelerated by the open neural network exchange ONNX and the neural network reasoning library TensorRT.

[0112] When the target visual language model is subjected to predetermined acceleration processing, each module (feature extraction module, feature fusion module, and object detection module) in the image processing network of the target visual language model can be optimized by the Open Neural Network Exchange (ONNX), and the neural network inference library TensorRT (a high-performance deep learning inference optimizer that can provide low-latency, high-throughput deployment inference for deep learning applications) can be accelerated to obtain the specified visual language model and improve the efficiency of object detection using the specified visual language model. Of course, the text processing network can also be accelerated by the Open Neural Network Exchange (ONNX) and the neural network inference library TensorRT, which is not limited by the present invention.

[0113] In addition, when performing acceleration processing of the open neural network exchange ONNX and the neural network reasoning library TensorRT, the model file of the target visual language model can be input into the open neural network exchange ONNX and the neural network reasoning library TensorRT to perform predetermined acceleration processing on the target visual language model (for example: deleting redundant parameters and connections in the model, reducing the size of the model, and vertical integration of the network structure, horizontal integration of the network structure, reducing connection layers, etc., to improve reasoning efficiency) to obtain the specified visual language model.

[0114] The target visual language model can perform object detection on a single image through an image processing network and a text processing network. In the present invention, by setting dynamic parameters, the specified visual language model can perform object detection on a predetermined number of frames of images in parallel; and the target visual language model can be accelerated by the open neural network exchange ONNX and the neural network reasoning library TensorRT, so that the specified visual language model can perform object detection on multiple frames of images in parallel and efficiently. It can be seen that the present invention can meet the demand for efficient object detection.

[0115] In the following, a visual language model deployment optimization method provided by the present invention is introduced in conjunction with another embodiment.

[0116] The present invention also relates to the intersection of computer vision and natural language processing, in particular, a visual language model deployment method based on acceleration and optimization of a visual language model, which is suitable for application scenarios such as real-time processing of video streams, video stream reasoning, and multi-threaded parallel computing. The present invention combines model acceleration technologies such as ONNX and TensorRT to improve the reasoning speed and performance of large models (Large Language Model, referred to as LLM, English: Large Language Model; refers to an advanced artificial intelligence algorithm trained on a large amount of data, a natural language processing model with more than 100 billion parameters, and a large model can be used as a visual language model) on edge devices and servers to meet the needs of efficient object detection.

[0117] The present invention provides a method for optimizing the deployment of a visual language model, which optimizes the model reasoning (object detection) process by processing multiple frames of images in parallel through multi-threading, and improves the efficiency of the visual language model by combining ONNX and TensorRT technologies. Figure 2 As shown, the process of a visual language model deployment optimization method provided by the present invention is as follows:

[0118] 1. Video stream reading:

[0119] 1.1, through OpenCV streaming technology, the video stream is read in an independent thread 1 (i.e. the first thread mentioned above) running on the CPU, so as to ensure that the delay will not increase due to the slow reasoning speed of the visual language model.

[0120] 1.2, when thread 1 reads the video stream, it is usually affected by network factors or hardware resource snatching and other factors, resulting in unstable speed of thread 1, which in turn affects the speed of subsequent threads; the present invention sets queue 1 (i.e., the first queue mentioned above) between thread 1 and thread 2 (i.e., the second thread mentioned above), and caches each frame of the read image in queue 1, ensuring that thread 2 can get the image read by thread 1 from queue 1 at a uniform speed; at the same time, thread 1 controls the length of queue 1 when storing image frames in queue 1. When the number of image frames in queue 1 exceeds 25 frames, the oldest frame in queue 1 is discarded to ensure that the delay will not increase due to the slow reasoning speed of the visual language model (resulting in more and more accumulated image frames).

[0121] 2. Model reasoning acceleration:

[0122] 2.1. In an independent thread 2 running on the GPU, the visual language model is used for reasoning (i.e., object detection). As the model size grows, it is difficult for the existing visual language model (i.e., the target visual language model) to achieve real-time reasoning speed (25fps (Frames Per Second)). In the entire process design, the speed of thread 2 determines the speed upper limit of the entire process.

[0123] 2.2. In order to improve the speed of thread 2 and further improve the speed of the entire process, first, after reading multiple image frames from queue 1, use GPU acceleration to preprocess multiple image frames (that is, call GPU to preprocess a predetermined number of image frames), and input the preprocessed multiple image frames (such as: batch_size×3×H×W, batch_size represents the number of input images, that is, specify the input parameters, which can be set to a predetermined number, such as: 4; 3 represents that the image is an RGB format image, H and W are the height and width of the preprocessed image frame respectively) into the inference acceleration model (such as: the accelerated GroundingDINO model) for multi-frame parallel accelerated inference. The visual language model is accelerated using ONNX and TensorRT technology. At the same time, multi-frame parallel inference is used to improve the inference speed and reduce latency.

[0124] 2.3, the target visual language model for unscheduled acceleration is as follows Figure 3 As shown, the specified visual language model after ONNX and TensorRT scheduled acceleration processing Figure 4Specifically, the overall structure of the target visual language model or the specified visual language model mainly includes an image processing network and a text processing network. Figure 3 In the target visual language model in , the input of the image processing network is a frame of image, and the input of the text processing network is text (target text). The text processing network extracts features from the text to obtain text features and inputs them into the feature fusion module; the image processing network includes a feature extraction module, a feature fusion module and an object detection module; the feature extraction module extracts features from a frame of image, and then inputs the extracted image features into the feature fusion module; the feature fusion module fuses the image features and text features of the frame of image to obtain fused features; the object detection module uses the fused features to perform object detection and outputs the object detection results. Figure 4 In the specified visual language model, the image processing network is an image processing network for predetermined accelerated processing, and the specified input parameters of the image processing network for predetermined accelerated processing are dynamic parameters and are set to a predetermined number (such as 4, 4 is obtained through experiments based on speed requirements and hardware level, and can also be 2, 8, etc., and the present invention is not limited to this), so that the input of the image processing network for predetermined accelerated processing is a predetermined number of frame images; the feature dimension parameters of the text processing network are set to a number that matches the predetermined number, so that the dimension of the text feature adapts to the predetermined number, so that the specified visual language model can perform parallel object detection on the predetermined number of frame images;

[0125] Among them, the various features that can be extracted by the text processing network, the feature dimension parameters of each feature can be set to a number that matches a predetermined number, to achieve multi-frame parallel reasoning and improve reasoning efficiency.

[0126] 2.4, further, since thread 2 is based on multi-frame parallel reasoning, it is not uniform reasoning. Therefore, in order to present the reasoning results in the push results at a uniform speed, the present invention sets queue 2 (i.e. the second queue mentioned above) between thread 2 and thread 3 (i.e. the third thread mentioned above), and caches the object detection results of each frame image after thread 2 reasoning in queue 2, so as to ensure that thread 3 can read the object detection results of each frame image processed by thread 2 from queue 2 at a uniform speed; at the same time, when thread 2 stores the object detection results of each frame image in queue 2, it will control the length of queue 2. When there are more than 3 frames of object detection results in queue 2, the object detection result of the oldest frame image in queue 2 will be discarded to reduce the delay.

[0127] 3. Streaming:

[0128] 3.1, thread 3 obtains the object detection result of each frame image from queue 2 at a uniform speed, and uses ffpmeg to decode, encode or format convert the object detection result of each frame image to obtain the push stream result containing the object detection result of each frame image; that is, thread 3 pushes the push stream result to other receiving ends, and other receiving ends can obtain the video stream containing the object detection result of each frame image.

[0129] The present invention significantly improves the reasoning speed and throughput of the visual language model by introducing multi-threaded processing and large model acceleration technology. The real-time performance of the present invention is enhanced: by optimizing the multi-threaded architecture of video stream processing and model reasoning, it is ensured that the system can achieve high real-time performance and reduce computing delays when processing large amounts of video data. The reasoning efficiency of the present invention is improved: by using ONNX model optimization and TensorRT acceleration technology, the reasoning time is greatly shortened, especially multi-frame parallel reasoning, which significantly improves the reasoning speed. The resource utilization rate of the present invention is improved: in view of the inconsistent processing time of each frame of the image, through multi-threaded parallel processing and reasonable scheduling of tasks, the CPU and GPU resources are maximized, reducing the waste of hardware resources. The present invention has good scalability: the present invention is applicable to various hardware platforms and computing scenarios, including large-scale deployment on the server side and lightweight deployment on edge devices, and has good scalability and versatility.

[0130] Based on the above method embodiment, the present invention also provides a visual language model deployment optimization device, such as Figure 5 As shown, the device comprises:

[0131] An acquisition module 510 is used to acquire images by running a first thread on the CPU and cache each acquired frame of the image into a first queue;

[0132] The input module 520 is used to read a predetermined number of frame images from the first queue through a second thread running on the GPU, input the read predetermined number of frame images in parallel to a specified visual language model running on the GPU, and cache the object detection results for each frame image output by the specified visual language model in the second queue; wherein the specified visual language model is a visual language model obtained after a predetermined acceleration process is performed on the target visual language model, and the number of parallel inputs of the specified visual language model for images is set to the predetermined number;

[0133] The reading module 530 is used to read the object detection result of each frame image from the second queue through a third thread to obtain the object detection result of each frame image.

[0134] The visual language model deployment optimization device provided by the embodiment of the present invention acquires images through a first thread running on the CPU, and caches each frame of the acquired image in a first queue; reads a predetermined number of frame images from the first queue through a second thread running on the GPU, and inputs them in parallel to a specified visual language model running on the GPU, and caches the object detection results for each frame of the image output by the specified visual language model in the second queue; and then reads the object detection results of each frame of the image from the second queue through a third thread to obtain the object detection results of each frame of the image. Among them, the specified visual language model in the embodiment of the present invention is a visual language model obtained after a predetermined acceleration process is performed on the target visual language model, that is, the specified object detection model is a deployment optimized visual language model, and the number of parallel inputs of the specified visual language model for images is set to a predetermined number, so that the specified visual language model can perform efficient object detection on a predetermined number of frame images in parallel. The first thread of the present invention runs on the CPU, and the second thread runs on the GPU, so that the resources of the CPU and GPU are maximized. Through multiple threads, multiple queues, and a specified visual language model with accelerated processing, object detection can be performed on a predetermined number of frame images in parallel, so as to achieve the purpose of improving the efficiency of object detection and meet the needs of efficient object detection.

[0135] Optionally, the input module is specifically used to:

[0136] Invoking the GPU to perform image preprocessing on the predetermined number of image frames;

[0137] The predetermined number of frames of images after image preprocessing are input into the designated visual language model in parallel.

[0138] Optionally, the target visual language model includes an image processing network and a text processing network, and the image processing network includes a feature extraction module, a feature fusion module and an object detection module;

[0139] The text processing network is used to extract features from the target text to obtain text features, and input the text features into the feature fusion module, wherein the target text is a text that represents the object detection requirements;

[0140] The feature extraction module is used to extract features from each frame of the input predetermined number of frames of images, obtain image features of the frame of image, and input the features to the feature fusion module;

[0141] The feature fusion module is used to fuse the image features of each frame of image received with the text features to obtain the fusion features corresponding to the frame of image and input the fusion features into the object detection module;

[0142] The object detection module is used to perform object detection according to the fusion features corresponding to each received frame image, and obtain the object detection result corresponding to the frame image.

[0143] Optionally, the specified input parameter of the image processing network of the specified visual language model is a dynamic parameter; wherein the specified input parameter is a parameter used to indicate the number of parallel image inputs;

[0144] The setting method of setting the number of parallel inputs of the specified visual language model to the predetermined number includes:

[0145] setting the number of specified input parameters of the image processing network to the predetermined number;

[0146] The characteristic dimension parameters of the text processing network are dynamic parameters; the device also includes:

[0147] A setting module is used to set the feature dimension parameter of the text processing network to a number matching the predetermined number; wherein the feature dimension parameter is a parameter used to indicate the number of dimensions of the extracted text features.

[0148] Optionally, the reading module is specifically used to:

[0149] Reading the object detection result of each frame image from the second queue;

[0150] The object detection result of each frame image is processed by the audio and video processing module to obtain a streaming result including the object detection result of each frame image.

[0151] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented by software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present invention is generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions may be transmitted from one website site, computer, server or data center to another website site, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium may be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more available media integrated. The available medium may be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state drive Solid State Disk (SSD)), etc.

[0152] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0153] Each embodiment in this specification is described in a related manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0154] The above description is only a preferred embodiment of the present invention and is not intended to limit the protection scope of the present invention. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention are included in the protection scope of the present invention.

Claims

1. A visual language model deployment optimization method, characterized in that: The method comprises: By running a first thread on the CPU, image acquisition is performed and each acquired frame of the image is cached in a first queue; A predetermined number of frame images are read from the first queue by a second thread running on the GPU, and the read predetermined number of frame images are input in parallel to a specified visual language model running on the GPU, and the object detection results for each frame image output by the specified visual language model are cached in the second queue; wherein the specified visual language model is a visual language model obtained after a predetermined acceleration process is performed on the target visual language model, and the number of parallel inputs of the specified visual language model for images is set to the predetermined number; The object detection result of each frame image is read from the second queue through the third thread to obtain the object detection result of each frame image.

2. The method according to claim 1, characterized in that The step of inputting the read predetermined number of frames of images in parallel to a specified visual language model running on a GPU includes: Invoking the GPU to perform image preprocessing on the predetermined number of image frames; The predetermined number of frames of images after image preprocessing are input into the designated visual language model in parallel.

3. The method according to claim 1, characterized in that: The target visual language model includes an image processing network and a text processing network, and the image processing network includes a feature extraction module, a feature fusion module and an object detection module; The text processing network is used to extract features from the target text to obtain text features, and input the text features into the feature fusion module, wherein the target text is a text that represents the object detection requirements; The feature extraction module is used to extract features from each frame of the input predetermined number of frames of images, obtain image features of the frame of image, and input the features to the feature fusion module; The feature fusion module is used to fuse the image features of each frame of image received with the text features to obtain the fusion features corresponding to the frame of image and input the fusion features into the object detection module; The object detection module is used to perform object detection according to the fusion features corresponding to each received frame image, and obtain the object detection result corresponding to the frame image.

4. The method according to claim 3, characterized in that The specified input parameter of the image processing network of the specified visual language model is a dynamic parameter; wherein the specified input parameter is a parameter used to indicate the number of parallel image inputs; The setting method of setting the number of parallel inputs of the specified visual language model to the predetermined number includes: setting the number of specified input parameters of the image processing network to the predetermined number; The characteristic dimension parameters of the text processing network are dynamic parameters; the method further includes: The feature dimension parameter of the text processing network is set to a number matching the predetermined number; wherein the feature dimension parameter is a parameter used to indicate the number of dimensions of the extracted text features.

5. The method according to any one of claims 1 to 4, characterized in that: The step of reading the object detection result of each frame image from the second queue to obtain the object detection result of each frame image includes: Reading the object detection result of each frame image from the second queue; The object detection result of each frame image is processed by the audio and video processing module to obtain a streaming result including the object detection result of each frame image.

6. A visual language model deployment optimization device, characterized in that: The device comprises: An acquisition module, used for acquiring images by running a first thread on the CPU and caching each acquired frame of the image into a first queue; An input module, configured to read a predetermined number of frame images from the first queue through a second thread running on the GPU, input the read predetermined number of frame images in parallel to a specified visual language model running on the GPU, and cache the object detection results for each frame image output by the specified visual language model in the second queue; wherein the specified visual language model is a visual language model obtained after a predetermined acceleration process is performed on the target visual language model, and the number of parallel inputs of the specified visual language model for images is set to the predetermined number; The reading module is used to read the object detection result of each frame image from the second queue through a third thread to obtain the object detection result of each frame image.

7. The device according to claim 6, characterized in that The input module is specifically used for: Invoking the GPU to perform image preprocessing on the predetermined number of image frames; The predetermined number of frames of images after image preprocessing are input into the designated visual language model in parallel.

8. The device according to claim 6, characterized in that The target visual language model includes an image processing network and a text processing network, and the image processing network includes a feature extraction module, a feature fusion module and an object detection module; The text processing network is used to extract features from the target text to obtain text features, and input the text features into the feature fusion module, wherein the target text is a text that represents the object detection requirements; The feature extraction module is used to extract features from each frame of the input predetermined number of frames of images, obtain image features of the frame of image, and input the features to the feature fusion module; The feature fusion module is used to fuse the image features of each frame of image received with the text features to obtain the fusion features corresponding to the frame of image and input the fusion features into the object detection module; The object detection module is used to perform object detection according to the fusion features corresponding to each received frame image, and obtain the object detection result corresponding to the frame image.

9. The device according to claim 8, characterized in that The specified input parameter of the image processing network of the specified visual language model is a dynamic parameter; wherein the specified input parameter is a parameter used to indicate the number of parallel image inputs; The setting method of setting the number of parallel inputs of the specified visual language model to the predetermined number includes: setting the number of specified input parameters of the image processing network to the predetermined number; The characteristic dimension parameters of the text processing network are dynamic parameters; the device also includes: A setting module is used to set the feature dimension parameter of the text processing network to a number matching the predetermined number; wherein the feature dimension parameter is a parameter used to indicate the number of dimensions of the extracted text features.

10. The device according to any one of claims 6 to 9, characterized in that: The reading module is specifically used for: Reading the object detection result of each frame image from the second queue; The object detection result of each frame image is processed by the audio and video processing module to obtain a streaming result including the object detection result of each frame image.

Images