Screenshot method based on double locks

Abstract

The invention relates to the technical field of image processing, in particular to a double-lock-based screenshot method, which comprises the following steps of: framing a video to obtain a plurality of images; determining image parameters; creating a thread pool; creating an AI recognition thread and a screenshot thread, and putting the threads into a thread pool; creating two locks in the memory; creating two memory spaces in a memory; defining a monitor of the AI recognition result, initializing the monitor, and transmitting an anonymous instance of the monitor to an AI recognition program; and executing a screenshot thread and an AI recognition thread according to the states of the two locks, storing and reading the screenshot by using the two memory spaces, and continuously carrying out operations of storing and reading the screenshot in a loop iteration manner. According to the method, the problems that the speed and the precision cannot be compatible during continuous screenshot in AI identification, and the AI identification of different contents is long in time consumption and poor in performance are solved.

Description

Technical Field

[0001] The present invention relates to the technical field of image processing, and in particular to a screenshot method based on double locking. Background Art

[0002] The current medical industry is closely integrated with Internet technology. In many scenarios, continuous screenshots are required to provide fast and stable screenshot data. For example, the object of surgical AI recognition is the doctor. The real-time screen or video of the operation is converted into an image through frame processing, and AI recognition is performed on the image. Then the AI ​​recognition results, such as the coordinates, number, cleanliness, endoscope speed and other parameters of the lesion site, are displayed on the video screen. Therefore, the speed and accuracy of screenshots are crucial to surgery.

[0003] like Figure 2 As shown, there are currently two traditional screenshot methods:

[0004] 1. The instructions for AI recognition and screenshot are in the same thread. The screenshot instruction is executed once each time, and then the memory address is written. Then AI recognition reads the image from the memory, and then the second screenshot is taken, and the execution is repeated continuously. Although the data accuracy of this method is guaranteed, the screenshot speed is affected to a certain extent because each AI recognition needs to wait for the screenshot to be completed.

[0005] 2. AI recognition and screenshot are in different threads. First, a memory address is agreed upon for reading and writing screenshot data. During the continuous screenshot process, screenshots are taken continuously and the image data is written to the agreed memory address. AI recognition reads the image from the agreed memory address. Although this method solves the screenshot speed problem, it is very likely that when the screenshot thread is writing the image data, the AI ​​recognition thread is also reading the memory address, which may easily cause errors in the image data obtained by AI recognition.

[0006] In addition, AI recognition has different recognition speeds and return frequencies for different recognition contents. During one recognition process, the coordinates of a part may have been recognized, but the clarity has not yet been completed. At this time, you can only continue to wait. When all parts are recognized, the data is returned as a whole, resulting in a waste of time and untimely display on the front end. Summary of the invention

[0007] In view of the shortcomings of the existing methods, the present invention solves the problems of incompatibility between speed and accuracy when taking screenshots continuously in AI recognition and the time-consuming and poor performance of AI recognition of different contents.

[0008] The technical solution adopted by the present invention is: a screenshot method based on double lock comprises the following steps:

[0009] Step 1: Process the video by dividing it into frames to obtain several images;

[0010] Step 2: Determine image parameters;

[0011] As a preferred implementation of the present invention, the image parameters include: X-axis offset, Y-axis offset, image size and image format.

[0012] Step 3: Create a thread pool;

[0013] As a preferred implementation of the present invention, Executors.newCachedThreadPool is used to create a thread pool.

[0014] Step 4: Create AI recognition thread and screenshot thread and put them into the thread pool;

[0015] As a preferred implementation of the present invention, Runnable is used to create an AI recognition thread and a screenshot thread respectively.

[0016] Step 5. Create two locks in memory;

[0017] As a preferred embodiment of the present invention, two locks are created using ReentrantLock.

[0018] Step 6. Create two memory spaces in the memory;

[0019] Step 7: Define the listener of AI recognition results, initialize the listener, and pass the anonymous class instance of the listener to the AI ​​recognition program;

[0020] As a preferred implementation manner of the present invention, the AI ​​recognition result includes: recognition task id, recognition type and recognition value.

[0021] As a preferred implementation of the present invention, the listener receives the AI ​​recognition result using the callback interface onResult.

[0022] Step 8: Execute the screenshot thread and AI recognition thread according to the status of the two locks, and use the two memory spaces to store and read the screenshots, and continuously loop and iterate to store and read the screenshots;

[0023] As a preferred implementation of the present invention, step eight specifically includes:

[0024] Step 81, use the screenshot thread to take a screenshot: determine the status of the two locks, if one of the locks is unlocked, lock the lock; then define getBitmap according to the image parameters, take a screenshot of the current frame image, write the screenshot into a memory space, and unlock the lock after writing is completed;

[0025] Step 82, the AI ​​recognition thread performs a certain read screenshot: determines the status of the two locks, and if a lock is in an unlocked state, locks the lock, and after reading the screenshot in a certain memory space, returns the recognition task id, and unlocks the lock;

[0026] Step 83, continuously repeating steps 81 and 82 to perform screenshot and read screenshot operations.

[0027] As a preferred implementation of the present invention, the image recognition thread in the AI ​​recognition program transmits the AI ​​recognition result to a third-party application in an asynchronous manner.

[0028] Beneficial effects of the present invention:

[0029] 1. In order to solve the conflict between the AI ​​recognition thread and the continuous screenshot thread, a lock is added during the AI ​​recognition and screenshot process so that other threads will not interfere with the current AI recognition and screenshot threads. When the AI ​​recognition obtains the image, if it determines that the image in the current memory space is locked, it will read the image in another memory space instead; if the screenshot thread determines that the image in the current memory space is locked, it will put the screenshot in another memory space, which will not affect the AI ​​thread; but a single lock will cause a waste of time waiting between threads, so the time-consuming problem is solved by the double lock method;

[0030] 2. In order to solve the problem of long time and poor performance of AI recognizing different contents, asynchronous recognition is used. Each time AI reads an image and starts to recognize it, the callback interface is executed immediately, but only the current task id is returned, and the recognition value and recognition type are not returned. This will not block the current thread, so that the next image can be read. By defining a listener, when AI recognizes an image, if a certain content is successfully recognized, the recognition result is returned through the listener, and the recognized content is quickly returned in batches. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] Figure 1 It is a flow chart of the screenshot method based on double lock of the present invention;

[0032] Figure 2 This is a schematic diagram of existing AI recognition and screenshots;

[0033] Figure 3 It is a schematic diagram of AI recognition and screenshot of the present invention. DETAILED DESCRIPTION

[0034] The present invention is further described below in conjunction with the accompanying drawings and embodiments. This figure is a simplified schematic diagram, which only illustrates the basic structure of the present invention in a schematic manner, and therefore it only shows the components related to the present invention.

[0035] The present invention is described by taking Android platform and Java language as examples.

[0036] like Figure 1 , 3 As shown, a screenshot method based on double lock includes the following steps:

[0037] Step 1: Process the video by dividing it into frames to obtain several images;

[0038] TextureView is used to render the video and perform frame processing to obtain several images. For example, a gastroscopy and colonoscopy video with a frame rate of 60 is frame-processed to obtain 60 images. Since there may be lesions in the video, several frames containing lesions are obtained, and a lesion may exist in one of the frames.

[0039] This embodiment describes medical video, and the method of the present invention can also be applied to processing in other fields.

[0040] Step 2: Determine image parameters;

[0041] Image parameters include: X-axis offset, Y-axis offset, image size and image format;

[0042] The image size includes width and height; the image format includes definition.

[0043] According to the requirements of the AI ​​recognition program for the image, determine the image's X-axis offset, Y-axis offset, width, height, and image format; for example, if AI recognition requires low definition, use RGB_565 to increase the screenshot speed; if high definition is required, use ARGB_8888 to reduce the screenshot speed.

[0044] The AI ​​recognition program can be obtained by using an existing conventional image recognition model, for example, by performing image recognition based on a DeepMACT model or a deep learning network model (Yolo v5l, GhostNet v2, etc.) to obtain an AI recognition program.

[0045] Step 3: Create a thread pool;

[0046] You can use Executors.newCachedThreadPool to create a thread pool. Of course, you can also use other methods to create a thread pool.

[0047] Step 4: Create AI recognition thread and screenshot thread and put them into the thread pool;

[0048] Use Runnable to create AI recognition thread and screenshot thread respectively. AI recognition thread and screenshot thread can also be placed in different thread pools.

[0049] Step 5. Create two locks in memory;

[0050] For example, use ReentrantLock to create two locks, namely the first lock lockA and the second lock lockB.

[0051] Step 6. Create two memory spaces in the memory to store screenshots and AI recognition threads for screenshot reading;

[0052] That is, create the first memory space picA and the second memory space picB;

[0053] Step 7: Define the listener of AI recognition results, initialize the listener, and pass the anonymous class instance of the listener to the AI ​​recognition program;

[0054] The listener uses the callback interface onResult to receive the AI ​​recognition result by defining an interface as the listener; the AI ​​recognition result in onResult has three parameters, namely the recognition task id, the recognition type type and the recognition value result; an anonymous class instance is created, onResult is rewritten, and the anonymous class instance is passed to the AI ​​recognition program.

[0055] Among them, id is a natural number; type includes: scope withdrawal speed, cecum recognition, cleanliness score, site recognition type and lesion detection type; result returns different values ​​according to different types, for example, the scope withdrawal speed is how many minutes, the cleanliness score is how many points; and when type is cecum recognition, the returned result may be which section of the cecum; the results of the site recognition type may include: inverted scope, gastric angle, gastric antrum, cardia; the results of the lesion detection type may include polyps, early cancer, ulcers and the corresponding lesion coordinates, etc.

[0056] Step 8: Execute the screenshot thread and AI recognition thread according to the status of the two locks, and use the two memory spaces to store and read the screenshots, and continuously loop and iterate to store and read the screenshots;

[0057] Step 81, use the screenshot thread to take a screenshot: determine the status of the two locks, if one of the locks is unlocked, lock the lock; then define getBitmap according to the image parameters, take a screenshot of the current frame image, write the screenshot into a memory space, and unlock the lock after the current frame screenshot is completed;

[0058] For example: take the first screenshot using the screenshot thread, determine whether the first lock is locked, and if it is not locked, lock the first lock; then define getBitmap according to the image parameters; obtain the current frame image, write it into the first memory space, and unlock the first lock after the current frame image screenshot is written;

[0059] The screenshot thread starts to execute and determines whether lockA is locked. If it is not locked, the lock interface is called to lock lockA, and then Bitmap is defined according to the image parameters. The getBitmap interface of TextureView is used to obtain the current frame screenshot of the video and write it to the memory space picA. After the screenshot is completed, the unlock interface is called to unlock lockA.

[0060] getBitmap includes: initializing an image instance according to the image format and image size; for example: capturing an image of a valid area in a video according to the X and Y axis offset values, width, and height; because a screenshot may have black edges or other invalid information around the video screen.

[0061] Step 82, the AI ​​recognition thread performs a certain read screenshot: determines the status of the two locks, and if a lock is in an unlocked state, locks the lock, and after reading the screenshot in a certain memory space, returns the recognition task id, and unlocks the lock;

[0062] For example: When the AI ​​recognition thread reads the screenshot for the first time, it first determines whether the first lock is locked. If it is not locked, it locks the first lock, reads the first memory space, returns the recognition task id after reading the screenshot, and unlocks the first lock;

[0063] The AI ​​recognition thread starts to execute and determines whether lockA is locked. If it is not locked, lockA is locked, and then the memory space picA is read. After reading the first screenshot, the recognition task id is returned through the recognition interface aiDetect(), and lockA is unlocked.

[0064] Step 83, continuously repeating steps 81 and 82 to perform screenshot and read screenshot operations;

[0065] For example: use the screenshot thread to take a second screenshot, determine whether the first lock is locked, if it is locked; then determine whether the second lock is locked, if not locked, lock the second lock, save the second screenshot in the second space of the memory, and unlock the second lock after saving;

[0066] Next, the AI ​​recognition thread performs a second image recognition to determine whether the first lock is locked. If it is locked, it determines whether the second lock is locked. If it is not locked, it locks the second lock, reads the screenshot in the second space of the memory, returns the recognition task id, and unlocks the second lock.

[0067] The AI ​​recognition thread continues to execute and determines whether lockA is locked. If lockA is locked, lockB is locked, and then the memory space picB is read, and the recognition task id is returned. After completion, lockB is unlocked.

[0068] Use the image recognition thread in the AI ​​recognition program to perform content recognition on the screenshots read by the AI ​​recognition thread, and transmit the AI ​​recognition results to the third-party application in an asynchronous manner;

[0069] For example: the image recognition thread recognizes the content of the screenshot and obtains a certain AI recognition result. For example, if the recognition of the withdrawal speed is completed, the AI ​​recognition result corresponding to the withdrawal speed is passed to the third-party application through the listener. The third-party application, such as the rendering thread, renders the AI ​​recognition result into the video; when the image recognition thread completes the second AI recognition result, such as the cleanliness score, the AI ​​recognition result corresponding to the cleanliness score is passed to the third-party application through the listener. The third-party application corresponds through the task identification id and renders the AI ​​recognition result belonging to the same frame of screenshot into the video; if the two recognition task ids are different, that is, they correspond to different frames of images, the second AI recognition result will be rendered into the corresponding video frame according to the recognition task id.

[0070] The screenshot thread of the present invention can avoid processing the same screenshot at the same time as the AI ​​recognition thread by judging the double locks of the first lock lockA and the second lock lockB; and can ensure that the AI ​​recognition thread can read the latest screenshot each time; in addition, the image recognition thread transmits several AI recognition results to the front-end rendering thread in an asynchronous manner, so as to achieve the purpose of displaying the recognition results at the first time.

[0071] Based on the above ideal embodiments of the present invention, the relevant staff can make various changes and modifications without departing from the technical concept of the present invention through the above description. The technical scope of the present invention is not limited to the contents of the specification, and its technical scope must be determined according to the scope of the claims.

Claims

1. A screenshot method based on double lock, characterized in that: The following steps are involved: Step 1: Process the video by dividing it into frames to obtain several images; Step 2: Determine image parameters; Step 3: Create a thread pool; Step 4: Create AI recognition thread and screenshot thread and put them into the thread pool; Step 5. Create two locks in memory; Step 6. Create two memory spaces in the memory; Step 7: Define the listener of AI recognition results, initialize the listener, and pass the anonymous class instance of the listener to the AI ​​recognition program; Step 8. Execute the screenshot thread and AI recognition thread according to the status of the two locks, and use two memory spaces to store and read screenshots, and continuously loop and iterate to store and read screenshots.

2. The screenshot method based on double lock according to claim 1, characterized in that: Step 8 specifically includes: Step 81, use the screenshot thread to take a screenshot: determine the status of the two locks, if one of the locks is unlocked, lock the lock; then define getBitmap according to the image parameters, take a screenshot of the current frame image, write the screenshot into a memory space, and unlock the lock after writing is completed; Step 82, the AI ​​recognition thread performs a certain read screenshot: determines the status of the two locks, and if a lock is in an unlocked state, locks the lock, and after reading the screenshot in a certain memory space, returns the recognition task id, and unlocks the lock; Step 83, continuously repeating steps 81 and 82 to perform screenshot and read screenshot operations.

3. The screenshot method based on double lock according to claim 1, characterized in that: The image recognition thread in the AI ​​recognition program transmits the AI ​​recognition results to third-party applications asynchronously.

4. The screenshot method based on double lock according to claim 1, characterized in that: Image parameters include: X-axis offset, Y-axis offset, image size and image format.

5. The screenshot method based on double lock according to claim 1, characterized in that: Use Executors.newCachedThreadPool to create a thread pool.

6. The screenshot method based on double lock according to claim 1, characterized in that: Use Runnable to create AI recognition thread and screenshot thread respectively.

7. The screenshot method based on double lock according to claim 1, characterized in that: Create two locks using ReentrantLock.

8. The screenshot method based on double lock according to claim 1, characterized in that: AI recognition results include: recognition task ID, recognition type and recognition value.

9. The screenshot method based on double lock according to claim 1, characterized in that: The listener uses the callback interface onResult to receive AI recognition results.

10. Screenshot system based on double lock, characterized by: include: a memory for storing instructions executable by a processor; A processor, configured to execute instructions to implement the screenshot method based on double locking as described in any one of claims 1-9.

Images