Method for obtaining target content and related device

Abstract

The present disclosure provides a method for obtaining target content and related devices. The method includes: obtaining first image data with a first clarity and including the target content, and obtaining second image data with a second clarity and having the same content as the first image data, where the first clarity is less than the second clarity; determining, based on the first image data, whether the user has completed a preset gesture for the target content at the current moment; in response to the user completing the preset gesture at the current moment, determining whether there is a target image in the second image data where the target content is not occluded; and in response to there being a target image in the second image data, obtaining the target content based on the target image.

Description

Technical Field

[0001] The present disclosure relates to the technical field of data processing, and in particular, to a method for obtaining target content and related devices. Background Art

[0002] This section aims to provide background or context for the embodiments of the present disclosure described in the claims. The description herein is not admitted to be prior art merely by virtue of its inclusion in this section.

[0003] People usually hope to retain the content they are interested in for future viewing and research. Generally, people capture this content by taking pictures, but inevitably, other content will also be captured, and then the other content is deleted through post-processing, which brings a lot of inconvenience to people. Although some technologies have emerged that can use computer image processing technology to collect the content indicated by people's gestures without mixing in other content. However, in this scenario, the hand is prone to block the content to be collected, and the complete and clear content cannot be obtained, resulting in the failure of content collection and the inability to quickly, accurately, and completely collect the target content of interest. Summary of the Invention

[0004] In view of this, there is a great need for an improved method that can effectively improve the problem of occlusion of the content obtained for the target content.

[0005] An exemplary embodiment of the present disclosure provides a method for obtaining target content, including:

[0006] Obtaining first image data with a first clarity and including the target content, and obtaining second image data with a second clarity and having the same content as the first image data, where the first clarity is less than the second clarity;

[0007] Based on the first image data, determining whether the user has completed a preset gesture for the target content at the current moment;

[0008] In response to the user completing the preset gesture at the current moment, determining whether there is a target image in the second image data where the target content is not occluded;

[0009] In response to the existence of the target image in the second image data, obtaining the target content based on the target image.

[0010] In some embodiments, determining whether there is a target image in the second image data where the target content is not occluded includes:

[0011] Based on the first image data, determining the gesture fingertip position information at the current moment;

[0012] Determine the target area of the target content in the second image data based on the gesture fingertip position information and the second image data;

[0013] Judge whether the overlap rate between the target area and the hand area in the second image data is greater than or equal to a preset threshold;

[0014] In response to the overlap rate being less than the preset threshold, determine that there is the target image in the second image data.

[0015] In some embodiments, the second image data includes a plurality of third images at preset frame intervals, and the method further includes:

[0016] Perform target content detection and fingertip hand detection on the third image to obtain the target content information and fingertip hand position information in the third image;

[0017] Store the preset number of third image information before the current moment into a cache queue, where the third image information includes the third image, the target content position information corresponding to the third image, and the fingertip hand position information.

[0018] In some embodiments, judging whether there is a target image in the second image data where the target content is not occluded includes:

[0019] Determine the gesture fingertip position information at the current moment based on the first image data;

[0020] Determine the target area of the target content in the third image based on the gesture fingertip position information and the target content position information of the third image;

[0021] Judge whether the overlap rate between the target area and the hand area in the fingertip hand position information of the third image is greater than or equal to a preset threshold;

[0022] In response to the overlap rate being less than the preset threshold, determine that there is the target image in the third image.

[0023] In some embodiments, the method further includes:

[0024] In response to the overlap rate being greater than or equal to the preset threshold, determine that there is no target image in the third image;

[0025] Judge whether there is a target image in the newly added third image information in the cache queue after the current moment; wherein, when the newly added third image information in the cache queue, the third image information with the earliest removal time is removed.

[0026] In some embodiments, the method further includes:

[0027] In response to the user completing the preset gesture at the current moment, prompt the user to move the hand away so that the user's hand does not block the target content.

[0028] In some embodiments, the first image data includes a plurality of consecutive first image frames. Then, determining whether the user has completed the preset gesture for the target image based on the first image data includes:

[0029] Identify the fingertip position of the user in the first image frame to obtain the gesture fingertip position information of the first image frame;

[0030] Obtain a gesture fingertip position information stream based on the plurality of gesture fingertip position information;

[0031] Determine whether the user has completed the preset gesture based on the gesture fingertip position information stream.

[0032] In some embodiments, determining whether the user has completed the preset gesture based on the gesture fingertip position information stream includes:

[0033] Determine whether the gesture fingertip position information stream matches the preset gesture;

[0034] In response to the gesture fingertip position information stream matching the preset gesture, determine that the user has completed the preset gesture.

[0035] Based on the same inventive concept, an exemplary embodiment of the present disclosure further provides an apparatus for obtaining target content, including:

[0036] An acquisition module, configured to acquire first image data with a first clarity and including the target content, and acquire second image data with a second clarity and having the same content as the first image data, where the first clarity is less than the second clarity;

[0037] A gesture determination module, configured to determine whether the user has completed a preset gesture for the target content at the current moment based on the first image data;

[0038] An occlusion determination module, configured to, in response to the user completing the preset gesture at the current moment, determine whether there is a target image in the second image data where the target content is not occluded;

[0039] A target content acquisition module, configured to, in response to the existence of the target image in the second image data, acquire the target content based on the target image.

[0040] Based on the same inventive concept, an exemplary embodiment of the present disclosure further provides an electronic device, including a memory, a processor, and a computer program stored on the memory and executable on the processor. When the processor executes the program, the method for obtaining the target content described in any one of the above is implemented.

[0041] Based on the same inventive concept, an exemplary embodiment of the present disclosure further provides a non-transitory computer-readable storage medium. The non-transitory computer-readable storage medium stores computer instructions, and the computer instructions are used to cause a computer to execute the method for obtaining the target content described in any one of the above.

[0042] Based on the same inventive concept, an exemplary embodiment of the present disclosure further provides a computer program product, including computer program instructions. When the computer program instructions run on a computer, the computer is caused to execute the method for obtaining the target content described in any one of the above.

[0043] As can be seen from the above, for the method for obtaining the target content and related devices provided by the present disclosure, gesture detection is performed on the first image data with lower clarity to trigger the acquisition of the target content; and when the acquisition of the target content is triggered, occlusion judgment of the target content is performed based on the second image data with higher clarity to obtain an unoccluded target content image, so that it is possible to quickly, accurately and completely collect the target content through gestures. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] In order to more clearly illustrate the technical solutions in the present disclosure or related technologies, the following will briefly introduce the drawings required for use in the description of the embodiments or related technologies. Obviously, the drawings in the following description are only the embodiments of the present disclosure. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative efforts.

[0045] Figure 1 It is a schematic diagram of a scenario for obtaining the target content.

[0046] Figure 2 It is a schematic diagram of the architecture for obtaining the target content of an exemplary embodiment of the present disclosure.

[0047] Figure 3A It is a schematic diagram of the application scenario of the method for obtaining the target content of an exemplary embodiment of the present disclosure.

[0048] Figure 3B It is a schematic flowchart of the method for obtaining the target content of an exemplary embodiment of the present disclosure.

[0049] Figure 4 It is a schematic diagram of the principle of the method for obtaining the target content of an exemplary embodiment of the present disclosure.

[0050] Figure 5 The structural schematic diagram of the acquisition device for the target content of the exemplary embodiment of the present disclosure.

[0051] Figure 6 The structural schematic diagram of the electronic device of the exemplary embodiment of the present disclosure. Detailed implementation manners

[0052] Hereinafter, the principles and spirit of the present application will be described with reference to several exemplary implementation manners. It should be understood that these implementation manners are provided only to enable those skilled in the art to better understand and then implement the present disclosure, rather than limiting the scope of the present application in any way. On the contrary, these implementation manners are provided to make the present disclosure more thorough and complete, and to be able to convey the scope of the present disclosure to those skilled in the art completely.

[0053] According to the embodiments of the present disclosure, a method for acquiring target content and related devices are provided.

[0054] In this article, it should be understood that the number of any elements in the drawings is for illustration rather than limitation, and any naming is only for distinction and does not have any limiting meaning.

[0055] Hereinafter, with reference to several representative implementation manners of the present disclosure, the principles and spirit of the present application will be elaborated in detail. Summary of the Invention

[0057] The solution of the present disclosure aims to provide a method for acquiring target content and related devices to implement an improved solution for acquiring target content.

[0058] Currently, users can collect images of target content through an image acquisition device (such as a camera) to obtain the target content. However, the field of view of the image acquisition device is generally in a regular shape, such as a rectangle or a square. If the layout form of the target content is an irregular shape, such as Figure 1 as shown. Figure 1 The schematic diagram of a target content acquisition scenario is shown. Figure 1In this case, the content object 100 includes content 101, content 102, and content 103. When the user wants to obtain content 101, if the field of view 110 of the image acquisition device includes not only content 101 but also a part of content 102, then the obtained image will mix content 102 in addition to content 101, which will make the image obtained by the user inaccurate. At this time, if the user wants to obtain content 101, post-processing is required, such as some image processing operations like cropping the image captured by the field of view 110, which will bring great inconvenience to the user. In addition, due to the property limitations of the image acquisition device itself, such as the focal length or the aspect ratio of the field of view, it is also easy to mix in other content. For example, when the user wants to obtain content 102, since the aspect ratio of the field of view is fixed, the field of view 120 of the image acquisition device includes not only content 102 but also a part of content 103, and other content is also mixed in, resulting in the inability to obtain the accurate target content. The user needs post-production processing to obtain the accurate target content, which increases the user's operations and reduces the efficiency and accuracy of obtaining the target content.

[0059] Although there are some ways to obtain target content that can accurately determine the area of the target content through the user's gesture interaction. For example, the area where the target content is located can be determined through the user's gesture, and the acquisition of the target content can be triggered. However, in this case, an image including the hand may be obtained, and the hand will block the target content, resulting in an incomplete target content being obtained.

[0060] In the process of implementing the present disclosure, the inventors found that the above-mentioned existing technologies all have obvious deficiencies. The traditional ways of obtaining target content are prone to mixing in other content, making the obtained content inaccurate and requiring post-processing; or when triggered by gestures, the image is directly captured, and the target content is easily blocked by the hand, making the obtained content incomplete.

[0061] Based on the characteristics of the above target content acquisition process and aiming at the problems existing in the above-mentioned existing technologies, the present disclosure provides a method for obtaining target content and related devices. Gesture detection is performed through the first image data with lower clarity to trigger the acquisition of the target content; and when the acquisition of the target content is triggered, occlusion judgment of the target content is performed based on the second image data with higher clarity to obtain an unoccluded target content image, so as to be able to quickly, accurately and completely capture the target content through gestures.

[0062] After introducing the basic principle of the present disclosure, the following specifically introduces various non-limiting implementation manners of the present disclosure.

[0063] Overview of Application Scenarios

[0064] Refer to Figure 2, which is a schematic diagram of the acquisition architecture of the target content provided by the embodiments of the present disclosure. The acquisition architecture 200 of the target content includes a server 210, a terminal 220, and a network 230 providing a communication link. The server 210 and the terminal 220 can be connected through the wired or wireless network 230. Among them, the server 210 can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms.

[0065] The terminal 220 can be implemented by hardware or software. For example, when the terminal 220 is implemented by hardware, it can be various electronic devices with a display screen and supporting page display, including but not limited to smart phones, tablet computers, e-book readers, laptop computers, and desktop computers, etc. When the terminal 220 device is implemented by software, it can be installed in the above-listed electronic devices; it can be implemented as multiple software or software modules (such as software or software modules for providing distributed services), or it can be implemented as a single software or software module, which is not specifically limited here.

[0066] It should be noted that the method for acquiring the target content provided by the embodiments of the present application can be executed by the terminal 220 or by the server 210. It should be understood that Figure 2 the numbers of the terminal, network, and server in

[0067] are only for illustration and are not intended to limit them. According to the implementation requirements, there can be any number of terminals, networks, and servers. Figure 3A Figure 3A Figure 3AAs shown, the target content object 100 includes contents 101-103, and the target content that the user wants to obtain is content 101. The user can use the hand 310 to draw a trajectory 320 including content 101 on the target content object 100. The image acquisition device continuously acquires the original image data of the target content object 100 and parses the original image data into first image data with a first clarity and second image data with a second clarity, where the first clarity can be less than the second clarity. The first image data includes at least one frame of first images. Since the clarity of the first images is relatively low, the computational amount and consumption for image processing thereof are correspondingly low. Hand detection can be performed on each frame of the first images. When the detected gesture of the hand 310 is a preset gesture for the target content 101, a determination on whether the target content is occluded is triggered. The second image data includes at least one frame of second images. The second images have a higher clarity, that is, the second images contain clearer target content. However, due to hardware conditions, it is not possible to perform frame-by-frame image processing on each frame of the second images. Instead, gesture detection and detection on whether the target content is occluded are performed every certain number of frames. It can be detected whether the target content in the second images is occluded, and the second images in which the target content is not occluded are used as target images 330, so as to obtain an image with a higher clarity and including the target content, and accurate and complete target content can be obtained from the target image.

[0068] The method according to the embodiments of the present disclosure performs gesture detection through the first image data with a lower clarity to trigger an occlusion determination of the target content; and performs an occlusion determination of the target content based on the second image data with a higher clarity to obtain an unoccluded target content image, thereby enabling the rapid, accurate, and complete acquisition of the target content through gestures.

[0069] Next, in combination with Figure 3A the application scenario, the method for obtaining the target content according to the exemplary embodiments of the present disclosure will be described. It should be noted that the above application scenario is only shown for the convenience of understanding the spirit and principle of the present disclosure, and the embodiments of the present disclosure are not limited in this regard. On the contrary, the embodiments of the present disclosure can be applied to any applicable scenario.

[0070] Exemplary Method

[0071] First, the embodiments of the present disclosure provide a method for obtaining target content, which can be executed by a server, such as Figure 2 the server 210 in Figure 2The terminal 220 in; it can also be jointly executed by the server and the terminal. For example, after the terminal 220 acquires the first image data and the second image data, it can send the first image data and the second image data to the server 210 via the network 230. The server 210 makes a judgment on the preset gesture and whether it is blocked, and determines the target image that is not blocked. Then the target image can be sent to the terminal 220, and the terminal 220 acquires the target content from the target image; or after acquiring the target content from the target image, it sends the target content to the terminal 220. Refer to Figure 3B , the method 300 for acquiring the target content according to the embodiment of the present disclosure may further include the following steps.

[0072] In step S301, acquire first image data with a first clarity and including the target content, and acquire second image data with a second clarity and having the same content as the first image data, where the first clarity is less than the second clarity.

[0073] Among them, after the image acquisition device acquires the original image data (such as original video data) including the target content, the original image data can be parsed via the corresponding interface to obtain the first image data with the first clarity and the second image data with the second clarity. Refer to Figure 4 , Figure 4 is a schematic diagram of the principle of the method for acquiring the target content according to the embodiment of the present disclosure. Figure 4 In, after parsing different clarities based on the original image data, the first image data and the second image data are obtained. Since the first image data and the second image data are parsed from the same original image data, the data of the first image data and the second image data is consistent. That is to say, for the same moment, the corresponding image content in the first image data and the second image data is the same, and the clarity of the second image data is higher than that of the first image data.

[0074] In step S302, based on the first image data, determine whether the user has completed the preset gesture for the target content at the current moment.

[0075] Among them, since the clarity of the first image data is low, using the first image data as the data basis for gesture judgment has a low calculation cost, which is beneficial to quickly and accurately judging the preset gesture and accelerating the response speed.

[0076] In some embodiments, the preset gesture may include gesture information pre-set by the user. The gesture information may include coordinate information and / or trajectory information of the preset gesture. Further, the coordinate information includes coordinate stream information formed by a plurality of coordinate values of the preset gesture. For example, the preset gesture may include drawing a circle (e.g., drawing at least a partial contour of the target content), drawing a line (e.g., drawing a line segment below the target content), pointing (e.g., pointing to or clicking on the area where the target content is located), etc.

[0077] In some embodiments, the first image data includes a plurality of consecutive first image frames. Then, determining whether the user has completed a preset gesture for the target image based on the first image data includes:

[0078] Identifying the fingertip position of the user in the first image frame to obtain the gesture fingertip position information of the first image frame;

[0079] Obtaining a gesture fingertip position information stream based on a plurality of the gesture fingertip position information;

[0080] Determining whether the user has completed the preset gesture based on the gesture fingertip position information stream.

[0081] In some embodiments, determining whether the user has completed the preset gesture based on the gesture fingertip position information stream includes:

[0082] Determining whether the gesture fingertip position information stream matches the preset gesture;

[0083] In response to the gesture fingertip position information stream matching the preset gesture, determining that the user has completed the preset gesture.

[0084] In some embodiments, determining whether the user has completed the preset gesture based on the gesture fingertip position information stream further includes:

[0085] In response to the gesture fingertip position information stream not matching the preset gesture, determining that the user has not completed the preset gesture.

[0086] Specifically, such as Figure 4As shown, the first image data 410 can be a video stream with a frame rate of 25fps. The first image data 410 includes a plurality of first image frames 411, and the interval between each pair of first image frames 411 is 40ms. Detect the fingertip position in each first image frame 411, for example, the coordinate information of the fingertip, which is denoted as the gesture fingertip position information. The gesture fingertip position information of the plurality of first image frames 411 forms the gesture fingertip position information stream of the first image data 410. The gesture fingertip position information stream can be processed (such as coordinate alignment processing or coordinate transformation processing) to convert the gesture fingertip position information stream into the fingertip information stream in the reference coordinate system, that is, to align the starting position of the gesture fingertip position information stream with the preset fingertip information stream of the preset gesture in the reference coordinate system. Then, the fingertip information stream is compared with the preset fingertip information stream of the preset gesture in this reference coordinate system. If the two are the same or at least part of the user's fingertip information stream is the same as at least part of the preset fingertip information stream, it indicates that the user has completed the preset gesture. For example, if the user's fingertip information stream L1 in the reference coordinate system is (x1, y1) → (x2, y2) → …… → (xi, yi), and the preset fingertip information stream Lf is (x1, y1) → (x2, y2) → …… → (xj, yj), where j ≥ i and i and j are positive integers, and the fingertip information stream L1 is at least part of the preset fingertip information stream Lf, it indicates that the user has completed the preset gesture.

[0087] It is also possible to perform trajectory recognition based on the user's fingertip information stream to determine the user's actual gesture; then determine whether the actual gesture is a preset gesture. For example, based on the user's fingertip information stream L1 being (x1, y1) → (x2, y2) → …… → (xi, yi) → (x1, y1), it is determined that the user's actual gesture is drawing a circle, and drawing a circle belongs to the preset gesture, indicating that the user has completed the preset gesture. Among them, a gesture model can be trained to recognize the user's fingertip information stream. For example, multiple fingertip position information streams can be used as the input layer data, and the corresponding gestures can be used as the output layer data to train a neural network to obtain a gesture recognition model.

[0088] Figure 4 Among them, gesture judgment is performed based on the first image frame 411, and at time t0, it is determined that the user has completed the preset gesture for the target content, thereby triggering the occlusion judgment of the second image data.

[0089] At step S303, in response to the user completing the preset gesture at the current moment, it is determined whether there is a target image in the second image data where the target content is not occluded.

[0090] In some embodiments, determining whether there is a target image in the second image data where the target content is not occluded includes:

[0091] Determine the gesture fingertip position information at the current moment based on the first image data;

[0092] Determine the target area of the target content in the second image data based on the gesture fingertip position information and the second image data;

[0093] Judge whether the overlap rate between the target area and the hand area in the second image data is greater than or equal to a preset threshold;

[0094] In response to the overlap rate being less than the preset threshold, determine that the target image exists in the second image data.

[0095] Specifically, Figure 4 In 420, the second image data includes a plurality of second image frames (not shown in the figure). Each second image frame can correspond to a first image frame 411. Finger-tip hand detection can be performed on each second image frame to obtain the corresponding fingertip position information L2 in the second image frame. When the occlusion judgment of the second image data is triggered at time t0, the gesture fingertip position information L0 in the first image frame at time t0 can be determined. Since the image contents of the first image frame and the second image frame are the same, the fingertip position information is also the same. Combining the fingertip position information L0 and the fingertip position information L2 of each second image frame, the target area D2 in the second image frame can be determined.

[0096] When performing finger-tip hand detection on the second image frame, the position of the hand area in the second image frame can also be obtained. For example, since the hand area will inevitably be recognized during the gesture recognition process, finger-tip hand detection can be performed on the second image frame based on the gesture model to obtain the hand area in the second image frame. For example, the gesture model can output a positioning box including the hand area. When performing the occlusion judgment of the target content, the overlap rate between the hand area and the target area D2 can be calculated. The overlap rate can include the ratio D1 / D2 of the first area D1 of the hand area to the second area of the target area D2. Then, the overlap rate is compared with the preset threshold. If the overlap rate is greater than or equal to the preset threshold, it means that the hand occludes the target content; if the overlap rate is less than the preset threshold, it means that the hand does not occlude the target content. The second image frame in which the hand does not occlude the target content is used as the target image to obtain accurate and complete target content therefrom.

[0097] In some embodiments, the second image data includes a plurality of third images spaced apart by a preset number of frames, and method 300 may further include:

[0098] Perform target content detection and finger-tip hand detection on the third image to obtain the target content information and finger-tip hand position information in the third image;

[0099] Store the preset number of third image information before the current moment into a cache queue, where the third image information includes the third image, the target content position information corresponding to the third image, and the fingertip hand position information.

[0100] Among them, due to hardware limitations, sometimes it is not possible to perform fingertip hand detection and target content detection for each second image frame of the second image data 420 (such as a high-definition road video stream). Fingertip hand detection and target content detection can be performed at intervals of a preset number of frames f. For example, Figure 4 in, fingertip hand detection and target content detection are performed every 5 frames, that is, every 200 ms. The corresponding fingertip hand detection results (i.e., fingertip hand position information) and target content detection results (i.e., target content information) can be stored in the cache queue 430 together with the second image frame at that moment. The length of the cache queue 430 is n frames of cache frames, and n is a positive integer. For example, if n = 5, the longest cache time of the cache queue 430 is 200 ms * 5 = 1 s.

[0101] As Figure 4 shown, the cache queue 430 may include at least one cache frame cache-1, cache-2, cache-3, cache-4, cache-5. Each cache frame includes a second image frame 421 with second clarity, a fingertip hand detection result (such as the hand region and fingertip position information L2 in the embodiments of the present disclosure), and a target content detection result (such as the target region D2 in the embodiments of the present disclosure). The cache queue 430 has a fixed length. As time goes by, when a new cache frame is added to the tail of the cache queue 430, the cache frame at the head of the queue is discarded to control the resources occupied by the cache and prevent the situation where the target content cannot be effectively recognized due to the movement of the target content, etc., resulting in the failure to obtain the target content.

[0102] In some embodiments, determining whether there is a target image in the second image data where the target content is not occluded includes:

[0103] Determine the gesture fingertip position information at the current moment based on the first image data;

[0104] Determine the target region of the target content in the third image based on the gesture fingertip position information and the target content position information of the third image;

[0105] Judge whether the overlap rate between the target region and the hand region in the fingertip hand position information of the third image is greater than or equal to a preset threshold;

[0106] In response to the overlap rate being less than the preset threshold, determine that there is a target image in the third image.

[0107] Specifically, when the occlusion determination of the second image data is triggered at time t0, the target image can be determined from the third image in the cache queue. Since the cache frame stores the third image and its corresponding fingertip hand detection results (such as the hand region and fingertip position information of the embodiments of the present disclosure) and the target content detection results (such as the target region of the embodiments of the present disclosure). The overlap rate between the hand region of the third image and the target region D2 can be calculated, and the overlap rate can include the ratio D3 / D2 of the third area D3 of the hand region to the second area of the target region D2. Then, the overlap rate is compared with a preset threshold. If the overlap rate is greater than or equal to the preset threshold, it indicates that the hand occludes the target content; if the overlap rate is less than the preset threshold, it indicates that the hand does not occlude the target content. The third image in which the hand does not occlude the target content is used as the target image to obtain accurate and complete target content therefrom. It can be seen that the occlusion determination of the target content for the historical cache frames in the cache queue can be completely parallel, so the acquisition efficiency of the target content can be improved and the response time can be further reduced.

[0108] In some embodiments, method 300 may further include:

[0109] In response to the overlap rate being greater than or equal to the preset threshold, determining that the target image does not exist in the third image;

[0110] Determining whether the target image exists in the newly added third image information in the cache queue after the current moment; wherein, the earliest removed third image information is the newly added third image information in the cache queue.

[0111] In actual operation, it may occur that there is no unoccluded target image in the cache queue 430, which may be because the hand occludes the target region for a period of time before the user completes the prediction gesture. At this time, the target image can be determined from the second image data or the cache queue after the current detection of the completion of the prediction gesture. For example, when a new cache frame enters the cache queue, the earliest removed cache frame can be removed, and the occlusion determination can be performed on the new cache frame, that is, to determine whether the overlap rate between the hand region in the newly added third image in the cache queue and the target region D2 is greater than or equal to the preset threshold, and the third image in which the hand in the newly added third image does not occlude the target content is used as the target image. For example, when Figure 4When there is no unoccluded target image in the cache frames cache-1, cache-2, cache-3, cache-4, and cache-5, when a new cache frame enters the cache queue 430, the cache frame cache-5 with the earliest time can be removed, and it is calculated whether the overlap rate between the hand region and the target region in the new cache frame is greater than or equal to a preset threshold. If the new cache frame has a target image, the third image in the new cache frame is returned; if the new cache frame still has no target image, the subsequent newly added cache frames are continuously judged for occlusion, and the third image that does not occlude the target content is used as the target image.

[0112] In some embodiments, method 300 may further include: in response to the user completing the preset gesture at the current moment, prompting the user to move the hand away so that the user's hand does not occlude the target content.

[0113] Among them, in order to prevent the response time from increasing due to the inability to collect the target image within a relatively long time range, the user can be prompted to move the hand away after completing the preset gesture. This prompt can be before the step of obtaining the target content or during the step of obtaining the target content. When prompting the user before the step of obtaining the target content, it can be guiding the user in the product description, such as through step diagrams combined with text descriptions in the static product description, or through animation diagrams in the dynamic product description. When prompting the user during the step of obtaining the target content, the user can be guided to quickly move the hand away from the target area at any time through voice and / or image, such as after the user completes the preset gesture.

[0114] In some embodiments, determining whether there is an unoccluded target image of the target content in the second image data includes:

[0115] Determining whether there is a fourth image that does not include a hand image in the second image data;

[0116] In response to the fourth image existing in the second image data, determining the fourth image as the target image.

[0117] Among them, it is also possible to directly determine whether there is a hand image in the second image data to determine the target image. If there is no hand image in the second image data, there will surely be no occlusion of the target content. Therefore, the second image in the second image data without a hand image can be determined as the target image.

[0118] Similarly, for the cached frames in the cache queue 430, the target image can also be determined by checking whether there is a hand image. The third image without a hand image in the cache queue 430 (including the historical cached frames between the current time t0 and the newly added cached frames after the current time t0) can be determined as the target image.

[0119] At step S304, in response to the presence of the target image in the second image data, the target content is obtained based on the target image.

[0120] Among them, target recognition (such as text recognition, image recognition, etc.) can be performed based on the target image to obtain the corresponding target content. Thus, it is possible to quickly, accurately, and completely collect the target content through gestures.

[0121] In some embodiments, method 300 further includes:

[0122] Determine whether the target image does not exist in the second image data within a preset time range;

[0123] In response to the target image not existing in the second image data within the preset time range, prompt the user to perform the gesture operation again.

[0124] Specifically, when a preset gesture of the user is completed, a preset time range can be set. When the target image has not been detected beyond this preset time range, the user can be prompted to perform the gesture operation again and obtain the target content again according to the method of the embodiments of the present disclosure.

[0125] Exemplary Device

[0126] Reference Figure 5 , based on the same inventive concept as the above embodiments of any method for obtaining target content, embodiments of the present disclosure also provide an apparatus for obtaining target content. The apparatus for obtaining target content includes:

[0127] An acquisition module, configured to acquire first image data with a first clarity and including the target content, and acquire second image data with a second clarity and having the same content as the first image data, where the first clarity is less than the second clarity;

[0128] A gesture determination module, configured to determine based on the first image data whether the user has completed a preset gesture for the target content at the current moment;

[0129] An occlusion determination module, configured to, in response to the user completing the preset gesture at the current moment, determine whether there is a target image in the second image data where the target content is not occluded;

[0130] A target content acquisition module, configured to, in response to the existence of the target image in the second image data, acquire the target content based on the target image.

[0131] In some embodiments, the occlusion determination module is further configured to:

[0132] Determine the gesture fingertip position information at the current moment based on the first image data;

[0133] Determine the target area of the target content in the second image data based on the gesture fingertip position information and the second image data;

[0134] Determine whether the overlap rate between the target area and the hand area in the second image data is greater than or equal to a preset threshold;

[0135] In response to the overlap rate being less than the preset threshold, determine that the target image exists in the second image data.

[0136] In some embodiments, the second image data includes multiple third images spaced apart by a preset number of frames;

[0137] The apparatus further includes a cache module, configured to:

[0138] Perform target content detection and fingertip hand detection on the third image to obtain the target content information and the fingertip hand position information in the third image;

[0139] Store the preset number of third image information before the current moment into a cache queue, where the third image information includes the third image, the target content position information corresponding to the third image, and the fingertip hand position information.

[0140] In some embodiments, the occlusion determination module is further configured to:

[0141] Determine the gesture fingertip position information at the current moment based on the first image data;

[0142] Determine the target area of the target content in the third image based on the gesture fingertip position information and the target content position information of the third image;

[0143] Determine whether the overlap rate between the target area and the hand area in the fingertip hand position information of the third image is greater than or equal to a preset threshold;

[0144] In response to the overlap rate being less than the preset threshold, determine that the target image exists in the third image.

[0145] In some embodiments, the cache module is further configured to:

[0146] In response to the overlapping rate being greater than or equal to the preset threshold, it is determined that the target image does not exist in the third image;

[0147] Determine whether the target image exists in the newly added third image information in the cache queue after the current moment; wherein, the newly added third image information in the cache queue is the third image information with the earliest removal time.

[0148] In some embodiments, the device further includes:

[0149] A prompting module, configured to, in response to the user completing the preset gesture at the current moment, prompt the user to move the hand away so that the user's hand does not block the target content.

[0150] In some embodiments, the first image data includes a plurality of consecutive first image frames; the gesture determination module is further configured to:

[0151] Identify the fingertip position of the user in the first image frame to obtain the gesture fingertip position information of the first image frame;

[0152] Obtain a gesture fingertip position information stream based on a plurality of the gesture fingertip position information;

[0153] Determine whether the user has completed the preset gesture based on the gesture fingertip position information stream.

[0154] In some embodiments, the gesture determination module is further configured to:

[0155] Determine whether the gesture fingertip position information stream matches the preset gesture;

[0156] In response to the gesture fingertip position information stream matching the preset gesture, determine that the user has completed the preset gesture.

[0157] In some embodiments, the device further includes: a timeout module, configured to determine whether the target image does not exist in the second image data within a preset time range;

[0158] The prompting module is further configured to, in response to the target image not existing in the second image data within the preset time range, prompt the user to perform a gesture operation again.

[0159] The device in the above embodiments is used to implement the corresponding target content acquisition method in any one of the embodiments in the foregoing exemplary target content acquisition method section, and has the beneficial effects of the corresponding method embodiments, which will not be elaborated herein.

[0160] Based on the same inventive concept as the method embodiments for obtaining any of the above target content, an embodiment of the present disclosure further provides an electronic device, including a memory, a processor, and a computer program stored on the memory and executable on the processor. When the processor executes the program, the method for obtaining the target content described in any of the above embodiments is implemented.

[0161] Figure 6 FIG. shows a schematic structural diagram of an electronic device according to an exemplary embodiment of the present disclosure. The device may include: a processor 610, a memory 620, an input / output interface 630, a communication interface 640, and a bus 650. Among them, the processor 610, the memory 620, the input / output interface 630, and the communication interface 640 are communicatively connected to each other inside the device through the bus 650.

[0162] The processor 610 may be implemented in a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits, etc., and is used to execute relevant programs to implement the technical solutions provided in the embodiments of this specification.

[0163] The memory 620 may be implemented in the form of a ROM (Read Only Memory), a RAM (Random Access Memory), a static storage device, a dynamic storage device, etc. The memory 1020 may store an operating system and other application programs. When implementing the technical solutions provided in the embodiments of this specification through software or firmware, the relevant program codes are stored in the memory 620 and are called and executed by the processor 610.

[0164] The input / output interface 630 is used to connect to an input / output module to implement information input and output. The input / output module may be configured as a component in the device (not shown in the figure) or externally connected to the device to provide corresponding functions. Among them, the input device may include a keyboard, a mouse, a touch screen, a microphone, various sensors, etc., and the output device may include a display, a speaker, a vibrator, an indicator light, etc.

[0165] The communication interface 640 is used to connect to a communication module (not shown in the figure) to implement communication interaction between this device and other devices. Among them, the communication module may implement communication in a wired manner (such as USB, network cable, etc.) or in a wireless manner (such as mobile network, WIFI, Bluetooth, etc.).

[0166] The bus 650 includes a path for transmitting information between various components of the device, such as the processor 610, the memory 1020, the input / output interface 630, and the communication interface 640.

[0167] It should be noted that although only the processor 610, the memory 620, the input / output interface 630, the communication interface 640, and the bus 650 are shown in the above device, in the specific implementation process, the device may further include other components necessary for normal operation. In addition, those skilled in the art can understand that the above device may also only include the components necessary to implement the solution of the embodiments of the present specification, and does not necessarily include all the components shown in the figure.

[0168] The electronic device of the above embodiment is used to implement the acquisition method of the corresponding target content in any one of the foregoing exemplary method parts, and has the beneficial effects of the corresponding method embodiments, which will not be elaborated here.

[0169] Exemplary Program Product

[0170] Based on the same inventive concept as the above embodiment of the acquisition method of any target content, the embodiment of the present disclosure also provides a non-transitory computer-readable storage medium, which stores computer instructions for causing the computer to execute the acquisition method of the corresponding target content in any one of the foregoing exemplary method parts.

[0171] The above non-transitory computer-readable storage medium may be any available medium or data storage device accessible by the computer, including but not limited to magnetic memories (such as floppy disks, hard disks, magnetic tapes, magneto-optical disks (MO), etc.), optical memories (such as CDs, DVDs, BDs, HVDs, etc.), and semiconductor memories (such as ROMs, EPROMs, EEPROMs, non-volatile memories (NAND FLASH), solid state drives (SSD)).

[0172] The computer instructions stored in the storage medium of the above embodiment are used to cause the computer to execute the acquisition method of the target content as described in any one of the above exemplary method parts, and have the beneficial effects of the corresponding method embodiments, which will not be elaborated here.

[0173] As is known to those skilled in the art, the embodiments of the present invention can be implemented as a system, a method, or a computer program product. Therefore, the present disclosure can be specifically implemented in the following forms, namely: entirely hardware, entirely software (including firmware, resident software, microcode, etc.), or a combination of hardware and software, which is generally referred to as "circuit", "module", or "system" herein. In addition, in some embodiments, the present invention can also be implemented in the form of a computer program product in one or more computer-readable media, which contain computer-readable program code.

[0174] Any combination of one or more computer-readable media can be adopted. The computer-readable media can be a computer-readable signal medium or a computer-readable storage medium. The computer-readable storage medium can be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination of the above. More specific examples (non-exhaustive) of the computer-readable storage medium can include, for example: an electrical connection having one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In this document, the computer-readable storage medium can be any tangible medium that contains or stores a program, which can be used by or in combination with an instruction execution system, apparatus, or device.

[0175] The computer-readable signal medium can include a data signal propagated in a baseband or as part of a carrier wave, which carries the computer-readable program code. Such a propagated data signal can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. The computer-readable signal medium can also be any computer-readable medium other than the computer-readable storage medium, which can send, propagate, or transmit a program for use by or in combination with an instruction execution system, apparatus, or device.

[0176] The program code contained on the computer-readable medium can be transmitted by any appropriate medium, including but not limited to wireless, wire, optical cable, RF, etc., or any suitable combination of the above.

[0177] Computer program code for performing the operations of the present invention may be written in one or more programming languages or combinations thereof. The programming languages include object-oriented programming languages such as Java, Smalltalk, C++, and also include conventional procedural programming languages such as the "C" language or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, executed as a stand-alone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network (including a local area network (LAN) or a wide area network (WAN)), or may be connected to an external computer (e.g., through the Internet using an Internet service provider).

[0178] It should be understood that each block of the flowchart and / or block diagram, and combinations of blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus, thereby producing a machine, and these computer program instructions, when executed by the computer or other programmable data processing apparatus, produce means for implementing the functions / operations specified in the blocks of the flowchart and / or block diagram.

[0179] These computer program instructions may also be stored in a computer-readable medium that can direct a computer or other programmable data processing apparatus to operate in a particular manner, such that the instructions stored in the computer-readable medium produce a product including an instruction means for implementing the functions / operations specified in the blocks of the flowchart and / or block diagram.

[0180] The computer program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device, such that a series of operation steps are performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, so that the instructions executed on the computer or other programmable apparatus can provide a process for implementing the functions / operations specified in the blocks of the flowchart and / or block diagram.

[0181] In addition, although the operations of the method of the present invention are described in a specific order in the drawings, this does not require or imply that these operations must be performed in that specific order, or that all of the shown operations must be performed to achieve the desired result. On the contrary, the steps depicted in the flowchart may be changed in the order of execution. Additionally or alternatively, certain steps may be omitted, multiple steps may be combined into one step for execution, and / or one step may be decomposed into multiple steps for execution.

[0182] The use of the verbs “comprise”, “comprising” and their inflected forms in the application documents does not exclude the presence of elements or steps other than those recited in the application documents. The article “a” or “an” before an element does not exclude the presence of a plurality of such elements.

[0183] Although the spirit and principles of the present invention have been described with reference to several specific embodiments, it should be understood that the present invention is not limited to the specific embodiments disclosed, and the division of each aspect does not mean that the features in these aspects cannot be combined for benefits. Such division is only for convenience of expression. The present invention is intended to cover various modifications and equivalent arrangements included within the spirit and scope of the appended claims. The scope of the appended claims is to be accorded the broadest interpretation so as to encompass all such modifications and equivalent structures and functions.

Claims

1. A method for obtaining target content, characterized in that, Including: Obtaining first image data with a first clarity and including the target content, and obtaining second image data with a second clarity and having the same content as the first image data, where the first clarity is less than the second clarity; Judging, based on the first image data, whether the user has completed a preset gesture for the target content at the current moment; In response to the user completing the preset gesture at the current moment, judging whether there is a target image in the second image data where the target content is not occluded, including: determining the gesture fingertip position information at the current moment based on the first image data; determining the target area of the target content in the second image data based on the gesture fingertip position information and the second image data; judging whether the overlap rate between the target area and the hand area in the second image data is greater than or equal to a preset threshold; in response to the overlap rate being less than the preset threshold, determining that there is the target image in the second image data; In response to there being the target image in the second image data, obtaining the target content based on the target image; Wherein, the second image data includes multiple third images spaced apart by a preset number of frames, and further includes: performing target content detection and fingertip and hand detection on the third images to obtain the target content information and fingertip and hand position information in the third images; storing the preset number of third image information before the current moment into a cache queue, where the third image information includes the third image, the target content position information corresponding to the third image, and the fingertip and hand position information.

2. The method according to claim 1, characterized in that, Judging whether there is a target image in the second image data where the target content is not occluded, including: Determining the gesture fingertip position information at the current moment based on the first image data; Determining the target area of the target content in the third image based on the gesture fingertip position information and the target content position information of the third image; Judging whether the overlap rate between the target area and the hand area in the fingertip and hand position information of the third image is greater than or equal to a preset threshold; In response to the overlap rate being less than the preset threshold, determining that there is the target image in the third image.

3. The method according to claim 2, wherein Further including: In response to the overlap rate being greater than or equal to the preset threshold, determining that there is no target image in the third image; Judging whether there is a target image in the newly added third image information in the cache queue after the current moment; wherein, when there is newly added third image information in the cache queue, the third image information with the earliest removal time is removed.

4. The method according to claim 1, wherein Further including: In response to the user completing the preset gesture at the current moment, prompting the user to move the hand away so that the user's hand does not occlude the target content.

5. The method according to claim 1, characterized in that If the first image data includes multiple consecutive first image frames, then judging whether the user has completed a preset gesture for the target image based on the first image data includes: Identifying the fingertip position of the user in the first image frame to obtain the gesture fingertip position information of the first image frame; Obtaining a gesture fingertip position information stream based on multiple pieces of the gesture fingertip position information; Determine whether the user has completed the preset gesture based on the information flow of the gesture fingertip positions.

6. The method according to claim 5, wherein Determining whether the user has completed the preset gesture based on the information flow of the gesture fingertip positions includes: Determine whether the information flow of the gesture fingertip positions matches the preset gesture; In response to the information flow of the gesture fingertip positions matching the preset gesture, determine that the user has completed the preset gesture.

7. An apparatus for obtaining target content, characterized in that, Includes: An acquisition module, configured to acquire first image data with a first clarity and including the target content, and acquire second image data with a second clarity and having the same content as the first image data, where the first clarity is less than the second clarity; A gesture determination module, configured to determine whether the user has completed a preset gesture for the target content at the current moment based on the first image data; An occlusion determination module, configured to, in response to the user completing the preset gesture at the current moment, determine whether there is a target image in the second image data where the target content is not occluded: determine the gesture fingertip position information at the current moment based on the first image data; Determine the target area of the target content in the second image data based on the gesture fingertip position information and the second image data; determine whether the overlap rate between the target area and the hand area in the second image data is greater than or equal to a preset threshold; In response to the overlap rate being less than the preset threshold, determine that there is a target image in the second image data; A target content acquisition module, configured to, in response to there being a target image in the second image data, acquire the target content based on the target image; Wherein, the second image data includes a plurality of third images spaced apart by a preset number of frames, and the apparatus further includes: a cache module, configured to: perform target content detection and fingertip and hand detection on the third images to obtain the target content information and fingertip and hand position information in the third images; store the information of a preset number of third images before the current moment in a cache queue, where the third image information includes the third image, the target content position information corresponding to the third image, and the fingertip and hand position information.

Images