A method for determining the target by combining gestures and voice

Abstract

This invention discloses a method, electronic device, and computer-readable storage medium for jointly determining the target object using gestures and voice. The method includes: acquiring the gesture trajectory input by a user to a target image and the associated voice command; extracting trajectory features such as drawing speed, distance between the first and last points, aspect ratio, envelope area, and number of repeated strokes; calculating a first type of confidence score corresponding to precise objects, ambiguous ranges, linear regions, solid regions, and semantic objects based on the degree to which the trajectory features satisfy preset feature conditions; calculating a second type of confidence score based on the degree of matching between object descriptors in the voice command and a preset descriptor set; weighted fusion of the two types of confidence scores to determine the final target object type, and generating an object mask, region mask, or strip mask as representation data. This invention improves the accuracy of target object recognition in multimodal interactions.

Description

Technical Field

[0001] This invention relates to the fields of human-computer interaction and image processing technology, and in particular to a method for determining the target of action by combining gestures and voice, an electronic device, and a computer-readable storage medium. Background Technology

[0002] With the development of artificial intelligence technology, especially image processing models and multimodal large models, users increasingly want to intelligently process images displayed on electronic devices. This could involve deleting, replacing, or recoloring specific objects in an image, or recognizing, answering questions about, and interpreting specific content within an image. A fundamental and crucial step in such applications is determining the target of the AI ​​processing—that is, exactly which part of the image the user wants to manipulate.

[0003] When expressing their processing intentions, users typically need to first specify the target and then describe the processing operation. During the process of specifying the target, the user's true intention often varies in terms of the granularity of selection: sometimes users want to precisely select the complete outline of a specific object in an image; sometimes users only want to roughly define a region, without requiring precise boundaries; sometimes users want to mark a linear area, such as the edge or border of an object; sometimes users want to select the entire image; and sometimes users want to select an object with specific semantic meaning, whose precise boundaries are difficult for the user to manually delineate. Accurately determining the target and its selection granularity desired by the user in a given interaction directly affects whether the subsequent AI processing results meet the user's expectations.

[0004] Existing technologies mainly employ the following approaches to determine the target of artificial intelligence processing, but they still have shortcomings in identifying the target.

[0005] The first type of solution is a gesture recognition scheme based on camera vision. This type of solution captures images of the user's hands using a camera, segments and classifies these images, and then identifies several predefined categories of hand movements. However, this type of solution identifies categories of the user's hand movements, and its output is usually a preset category label for the movement, without determining the specific object of action based on the image content displayed on the electronic device. This type of solution also struggles to handle trajectories freely drawn by the user on the display interface, cannot distinguish the granularity of the user's desired selection, and is even less able to combine the user's voice and semantics to determine and correct the object of action.

[0006] The second type of solution is based on the processing area pointed to by the touch operation. This type of solution acquires the user's touch operation on the displayed image, determines the area pointed to by the touch operation as the area to be processed, and then extracts elements from this area or hands it over to a generative model for processing. However, this type of solution usually only determines a single area based on the pointing position of a single touch operation, making it difficult to determine the granularity of the selection desired by the user in this interaction. That is, it is difficult to distinguish whether the user wants to precisely select a specific object, roughly define a vague area, mark a linear area, or select the entire image or a semantic object. In addition, in some solutions, voice information is not involved in determining the type of object and the granularity of selection, making it difficult to accurately determine and correct the object based on voice semantics.

[0007] The third type of solution is based on directly generating the selection area from a hand-drawn area. This type of solution directly uses the area enclosed by the trajectory drawn by the user on the image, or its envelope, as the selection mask for the target object. However, the trajectory drawn by the user is inherently imprecise: when the user expects to precisely select a specific object, the hand-drawn trajectory often cannot closely follow the actual boundary of the object, resulting in a selection area that is too large or too small; when the user only wants to roughly define a range, using the trajectory envelope directly as the selection area may not reflect the tolerance intended by the user. Because this type of solution will select different areas... Figure 1 The rules are converted into selection areas in the same way, and it is impossible to use different selection area generation methods according to the user's selection granularity. Therefore, it is difficult to take into account different scenarios such as precise selection, fuzzy selection, and line marking, and the recognition accuracy of the target object is low.

[0008] In summary, the common problem with existing technologies is that, when determining the target of artificial intelligence processing, they rely solely on a single modal input, such as hand gestures, a single touch gesture, or a hand-drawn trajectory. They fail to combine the characteristics of the gesture's drawing process with the semantic information in the user's speech, making it difficult to accurately determine the user's desired selection granularity in different scenarios. This easily leads to misjudgment of the target and fails to meet users' needs for refined AI processing of images. Therefore, there is an urgent need for a human-computer interaction solution that can accurately determine the target and its selection granularity. Summary of the Invention

[0009] To address the problem that existing technologies, when determining the target of artificial intelligence processing, struggle to accurately determine the user's desired selection granularity based solely on a single modal input, and are prone to misjudging the target, this invention provides a method, electronic device, and computer-readable storage medium for jointly determining the target using gestures and voice.

[0010] To achieve the above objectives, the first aspect of the present invention provides a method for determining an object by combining gestures and voice, applied to an electronic device, wherein a target image is displayed on the display interface of the electronic device; comprising:

[0011] S1: Obtain the gesture trajectory and associated voice command input by the user for the target image;

[0012] S2: Extract the trajectory features of the gesture trajectory, including the drawing speed of the gesture trajectory, the distance between the first point and the last point, the aspect ratio of the trajectory, the area of ​​the trajectory envelope, and the number of repeated strokes;

[0013] S3: Match the trajectory features with the preset feature conditions corresponding to each of the precise object type, fuzzy range type, linear region type, whole region type and semantic object type, and obtain the normalized first type confidence vector based on the degree to which the trajectory features satisfy each preset feature condition;

[0014] S4: Perform semantic recognition on the voice command to obtain object descriptors, match the object descriptors with the preset descriptor sets corresponding to each type of object, and obtain a normalized second type confidence vector based on the degree of matching between the object descriptors and each preset descriptor set.

[0015] S5: Set first and second fusion weights for the first and second type confidence vectors respectively, and sum the first and second type confidences of the same target type to obtain a fusion type confidence vector. Determine the target type with the highest confidence as the final target type. When no object descriptor is identified or its semantic recognition confidence is lower than the threshold, increase the first fusion weight and decrease the second fusion weight.

[0016] S6: Generate representation data of the target object on the target image according to the final target object type, wherein the representation data is at least one of object mask, region mask or strip mask.

[0017] Further, in step S3, calculating the first type of confidence score based on the degree to which the trajectory features satisfy each of the preset feature conditions includes:

[0018] The confidence level of the first type corresponding to the precise object type increases as the degree to which the drawing speed is lower than the first speed threshold increases, and as the degree to which the distance between the first point and the last point is lower than the closing distance threshold increases.

[0019] The first type confidence level corresponding to the fuzzy range type increases as the drawing speed exceeds the first speed threshold and when the distance between the first point and the last point is less than the closure distance threshold.

[0020] The first type confidence level corresponding to the linear region type increases as the distance between the first point and the last point is greater than the closure distance threshold and as the aspect ratio of the trajectory is greater than the aspect ratio threshold.

[0021] The first type confidence level corresponding to the whole region type increases as the degree to which the drawing speed is higher than the second speed threshold increases, and as the degree to which the ratio of the trajectory envelope area to the target image area is higher than the area ratio threshold increases.

[0022] The confidence level of the first type corresponding to the semantic object type increases as the degree to which the number of repeated strokes exceeds the stroke count threshold increases;

[0023] Wherein, the second speed threshold is greater than the first speed threshold.

[0024] Further, in step S4, the second type of confidence score is calculated based on the matching degree between the object descriptor and each of the preset descriptor sets, including:

[0025] When the object descriptor is a noun referring to a specific object, the second type confidence corresponding to the semantic object type is increased; when the object descriptor contains both a noun referring to a specific object and a proximity indicator, the second type confidence corresponding to the semantic object type is increased, wherein the proximity indicator includes at least one of this, here, that place, and the circled object;

[0026] When the object descriptor is a proximity indicator that does not contain a specific object name, the second type confidence corresponding to the precise object type is increased;

[0027] When the object descriptor is a range indicator, increase the second type confidence corresponding to the fuzzy range type. The range indicator includes at least one of "this block", "this area", "surrounding", and "background".

[0028] When the object descriptor is a linear indicator, increase the second type confidence corresponding to the linear region type. The linear indicator includes at least one of the following: line, edge, and border.

[0029] When the object descriptor is a global indicator, the second type confidence corresponding to the whole region type is increased. The global indicator includes at least one of all, whole, all and whole sheet.

[0030] Furthermore, in step S2,

[0031] The drawing speed is the ratio of the distance between adjacent sampling points of the gesture trajectory to the sampling time interval, or the ratio of the total length of the gesture trajectory to the drawing time of the gesture trajectory;

[0032] The distance between the first point and the last point is the straight-line distance between the starting sampling point and the ending sampling point of the gesture trajectory;

[0033] The aspect ratio of the trajectory is the ratio of the longer side to the shorter side of the smallest bounding rectangle of the gesture trajectory;

[0034] When the gesture trajectory is a closed trajectory, the trajectory envelope area is the area enclosed by the closed trajectory; when the gesture trajectory is a non-closed trajectory, the trajectory envelope area is the convex hull area or the area of ​​the smallest circumscribed polygon of the gesture trajectory.

[0035] The number of repeated strokes refers to the number of times the gesture trajectory intersects or repeatedly passes through the same local area.

[0036] Furthermore, the trajectory features in step S2 also include pause points, which are sampling points corresponding to the duration of the continuous drawing speed being less than the pause speed threshold being greater than or equal to the pause duration threshold; in step S6, when the pause point exists in the gesture trajectory, the characterization data is generated by applying a higher fitting accuracy to the local trajectory within a preset range around the pause point than to the other trajectory segments.

[0037] Further, step S6 includes:

[0038] When the final target type is a precise target type, object recognition is performed on the region within the envelope of the gesture trajectory to obtain at least one candidate object, and the contour of the candidate object with the highest degree of fit with the gesture trajectory is determined as the object mask as the representation data.

[0039] When the final target type is a semantic object type, object recognition is performed on the region within the envelope of the gesture trajectory to obtain at least one candidate object, and the contour of the candidate object with the highest semantic matching degree with the object descriptor is determined as the object mask as the representation data.

[0040] When the final target type is a fuzzy range type or a whole region type, a region mask is generated as the representation data.

[0041] When the final target type is a linear region, a strip mask generated with the gesture trajectory as the center line and a preset width will be used as the representation data.

[0042] The degree of fit is determined based on at least one of the following parameters: the percentage of overlap between the outline of the candidate object and the envelope of the gesture trajectory; the average boundary distance between the outline of the candidate object and the gesture trajectory; the intersection-union ratio between the outline of the candidate object and the envelope of the gesture trajectory; the distance between the center point of the candidate object and the center point of the gesture trajectory envelope; and the percentage of the area of ​​the candidate object covered by the gesture trajectory envelope.

[0043] The semantic matching degree is determined based on the text similarity between the candidate object's category label, image recognition description text, or visual feature description text and the object's descriptive words.

[0044] Further, the region mask and / or strip mask are generated in the following manner:

[0045] When the final target type is a fuzzy range type, the region mask is generated by expanding the envelope of the gesture trajectory outward by a first tolerance.

[0046] When the final target type is a whole region type, the entire region of the target image is used as the region mask;

[0047] When the final target type is a linear region, the preset width of the strip mask is determined based on the stroke width of the gesture trajectory, the drawing speed, or the descriptive words representing the size of the range in the voice command, wherein the larger the stroke width, the larger the preset width.

[0048] Furthermore, step S4 further includes semantic recognition of the voice command to obtain an operation intent word; the method further includes: associating the final target type, the representation data, and the operation intent word and sending them to the artificial intelligence processing module, so that the artificial intelligence processing module performs the processing corresponding to the operation intent word on the target represented by the representation data; wherein, the gesture trajectory and the voice command are acquired within the same interaction cycle, and the same interaction cycle is determined by the input device's press event, touch event, or quick trigger event.

[0049] In some implementations, the first speed threshold is 200 to 600 pixels per second; the second speed threshold is 800 to 1500 pixels per second; the closure distance threshold is 2% to 5% of the diagonal length of the target image; the aspect ratio threshold is 3 to 6; the area ratio threshold is 60% to 90%; and the number of strokes threshold is 2 to 4.

[0050] A second aspect of the present invention provides an electronic device, the electronic device including a processor, a memory, and a computer program stored in the memory and executable on the processor, wherein the computer program, when executed by the processor, implements the method described in any of the above embodiments.

[0051] A third aspect of the present invention provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the method described in any of the above embodiments.

[0052] Compared with the prior art, the present invention has the following beneficial effects:

[0053] First, this invention extracts trajectory features reflecting the drawing process from the user's input gesture trajectory, such as drawing speed, distance between the first and last points, trajectory aspect ratio, trajectory envelope area, and number of repeated strokes. Based on these features, it helps determine the user's desired selection granularity, thereby distinguishing different object types such as precise objects, ambiguous ranges, linear regions, entire regions, and semantic objects. Compared to schemes that uniformly use the area enclosed by the hand-drawn trajectory as the selection area, this invention can use different object determination methods based on the user's intention during the drawing process. It can obtain objects that meet the user's expectations in different scenarios such as precise selection, ambiguous selection, and linear marking, thus improving the accuracy of object recognition.

[0054] Secondly, this invention not only calculates the first type of confidence score based on the trajectory features of the gesture, but also performs semantic recognition on the voice command to obtain object descriptive words and calculates the second type of confidence score. The two are then weighted and fused to determine the final target object type. Compared to schemes that determine the target object based solely on a single modal input, or where voice does not participate in target object determination, this invention, through the joint determination of the gesture dynamic feature channel and the voice semantic channel, allows for the supplementation and correction of the target object type using voice semantics when the gesture trajectory itself is insufficient to clearly define the selection granularity, further improving the accuracy of target object determination.

[0055] Third, when fusing the confidence levels of the two channels, if no object descriptor is identified or the semantic recognition confidence level of the object descriptor is low, the present invention increases the first fusion weight corresponding to the gesture trajectory channel and correspondingly decreases the weight of the speech semantic channel. Therefore, in cases where speech recognition is unreliable, the determination of the target object type can rely more heavily on the more reliable gesture trajectory channel, avoiding misjudgment of the target object due to speech recognition errors and improving the robustness of multimodal joint determination.

[0056] Fourth, this invention generates representational data for the target object in a manner appropriate to the final target object type: for precise and semantic target objects, object recognition is performed on the region within the envelope of the gesture trajectory or the region within the search range expanded based on object descriptors. Object masks are then determined based on the degree of fit with the gesture trajectory or the degree of semantic matching with the object descriptors, thereby converging the user's imprecise hand-drawn trajectory into a precise outline that fits the true boundary of the object. For fuzzy range and solid region types, corresponding region masks are generated. For linear region types, a strip mask centered on the gesture trajectory is generated. Thus, by generating representational data in different forms for different selection granularities, subsequent artificial intelligence processing can be applied to the target object range that matches the user's expectations, improving the consistency between the artificial intelligence processing results and the user's intent in multimodal interaction. Attached Figure Description

[0057] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0058] Figure 1 This is a flowchart illustrating the method for determining the target object using a combination of gestures and voice, as provided in an embodiment of the present invention.

[0059] Figure 2 A schematic diagram illustrating the coordinate mapping relationship between a gesture trajectory and a target image provided in an embodiment of the present invention;

[0060] Figure 3 A schematic diagram illustrating the principle of determining the final target type by weighted fusion of a first type of confidence vector and a second type of confidence vector, as provided in an embodiment of the present invention.

[0061] Figure 4 A schematic diagram illustrating the correspondence between five types of target objects, gesture trajectory patterns, and voice commands provided in embodiments of the present invention;

[0062] Figure 5 This is a schematic diagram illustrating the determination of an object mask based on candidate objects under precise object types or semantic object types, as provided in an embodiment of the present invention.

[0063] Figure 6 This is a schematic diagram illustrating the generation of region masks or strip masks under the fuzzy range type, whole region type, and linear region type provided in the embodiments of the present invention;

[0064] Figure 7 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present invention.

[0065] Explanation of reference numerals in the attached figures: 100 - target image; 110 - gesture trajectory; 120 - envelope of gesture trajectory; 130 - candidate object; 140 - object mask; 150 - region mask; 160 - strip mask; 200 - electronic device; 210 - processor; 220 - memory; 230 - computer program. Detailed Implementation

[0066] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of the embodiments of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0067] It should be noted that the meanings of several terms involved in the embodiments of the present invention are as follows: The target image 100 refers to an image with raw pixel data presented on the display interface of the electronic device by an image host application, such as an image viewer, image editor, document editor, spreadsheet editor, presentation editor, or browser. The gesture trajectory 110 refers to the sequence of trajectory points corresponding to the trajectory input by the user on the display interface to indicate the target. The target object refers to the object or region in the target image 100 that the user expects to be processed by the artificial intelligence processing module. The representation data refers to data used to represent the position and range of the target object in the target image 100, including object mask 140, region mask 150, or strip mask 160.

[0068] In this embodiment of the invention, the method by which the user inputs the gesture trajectory 110 does not constitute a limitation of the invention. In some embodiments, the user inputs the gesture trajectory 110 through a mouse connected to the electronic device, the gesture trajectory 110 being a sequence of positions formed by the movement of the mouse cursor on the display interface; in other embodiments, the user inputs the gesture trajectory 110 by sliding a stylus or finger on a touchscreen; in still other embodiments, the user inputs the gesture trajectory 110 through a touchpad or other pointing device. The voice command associated with the gesture trajectory 110 refers to an audio stream input by the user through a microphone within the same interaction cycle of inputting the gesture trajectory 110, or a text command obtained after speech recognition of the audio stream. The same interaction cycle can be determined by a press event, touch event, or quick trigger event of the input device, for example, taking the press event of the smart button of the input device as the start of the interaction cycle and the release event of the smart button as the end of the interaction cycle, and synchronously collecting the gesture trajectory 110 and the voice command within the interaction cycle.

[0069] Please see Figure 1 This invention provides a method for determining an object using a combination of gestures and voice. The method is applied to an electronic device whose display interface shows a target image 100. The method includes steps S1 to S6. Each step is described in detail below with reference to the accompanying drawings.

[0070] Step S1: Obtain the gesture trajectory 110 input by the user to the target image 100, and the voice command associated with the gesture trajectory 110.

[0071] Specifically, when the user inputs the gesture trajectory 110 on the display interface, the electronic device collects sampling points on the gesture trajectory 110 at a preset sampling rate to obtain the trajectory point sequence corresponding to the gesture trajectory 110; at the same time, it collects the audio stream corresponding to the voice command through a microphone. In some embodiments, the preset sampling rate is 60 frames per second to 120 frames per second.

[0072] Please see Figure 2Since the user input of the gesture trajectory 110 usually occurs in the display interface coordinate system, while the target image 100 is often displayed on the display interface after scaling and translation, the coordinates of the gesture trajectory 110 in the display interface coordinate system may not be consistent with its coordinates in the target image coordinate system of the target image 100. In some embodiments, the gesture trajectory 110 is a sequence of trajectory points in the target image coordinate system; in other embodiments, the electronic device first acquires the sequence of trajectory points in the display interface coordinate system, and then performs coordinate mapping on the sequence of trajectory points in the display interface coordinate system according to the display position and display scaling ratio of the target image 100 on the display interface to obtain the sequence of trajectory points in the target image coordinate system. For example, the display interface coordinates can be subtracted from the display offset of the target image 100 on the display interface, and then divided by the display scaling ratio to obtain the coordinates in the target image coordinate system. The coordinate mapping can also be further combined with canvas offset, device pixel ratio, or two-dimensional affine transformation matrix, which is not limited in this invention.

[0073] Step S2: Extract the trajectory features of the gesture trajectory 110. The trajectory features include the drawing speed of the gesture trajectory 110, the distance between the first point and the last point, the aspect ratio of the trajectory, the area of ​​the trajectory envelope, and the number of repeated strokes.

[0074] Specifically, the drawing speed is the ratio of the distance between adjacent sampling points of the gesture trajectory 110 to the sampling time interval, or the ratio of the total length of the gesture trajectory 110 to the drawing time of the gesture trajectory 110. For example, for two adjacent sampling points, the drawing speed at that point is obtained by dividing the Euclidean distance between the two sampling points by the sampling time interval between the two sampling points; the average drawing speed of the gesture trajectory 110 is obtained by averaging the drawing speeds at various points on the gesture trajectory 110.

[0075] The distance between the first and last points is the straight-line distance between the starting and ending sampling points of the gesture trajectory 110. The aspect ratio of the trajectory is the ratio of the longer side to the shorter side of the smallest bounding rectangle of the gesture trajectory 110. The trajectory envelope area is the area enclosed by the closed trajectory when the gesture trajectory 110 is a closed trajectory, and the convex hull area or the area of ​​the smallest bounding polygon of the gesture trajectory 110 when the gesture trajectory 110 is a non-closed trajectory. The number of repeated strokes is the number of times the gesture trajectory 110 intersects or repeatedly passes through the same local area; for example, when the gesture trajectory 110 repeatedly traces back and forth within a certain local area, the number of times it repeatedly passes through that local area is recorded as the number of repeated strokes.

[0076] In some embodiments, the trajectory features further include pause points. A pause point is a sampling point corresponding to a duration where the drawing speed is continuously less than a pause speed threshold for a duration greater than or equal to a pause duration threshold. For example, when a user slows down or pauses at a certain position while inputting the gesture trajectory 110, causing the drawing speed near that position to be continuously lower than the pause speed threshold for a duration equal to the pause duration threshold, the sampling point corresponding to that position is determined as a pause point. The pause point can be used to reflect the user's interactive intent to more precisely specify boundaries or emphasize that position.

[0077] Step S3: Match the trajectory features with the preset feature conditions corresponding to the precise object type, fuzzy range type, linear region type, whole region type and semantic object type respectively. Calculate the first type confidence degree corresponding to each object type based on the degree to which the trajectory features satisfy each preset feature condition, and obtain the normalized first type confidence degree vector.

[0078] In this embodiment of the invention, the five types of target objects have the following meanings: The precise object type corresponds to the situation where the user expects to precisely select the complete outline of a specific object in the target image 100; the fuzzy range type corresponds to the situation where the user expects to roughly define a range without requiring precise boundaries; the linear region type corresponds to the situation where the user expects to mark a linear region, such as the edge or border of an object; the entire region type corresponds to the situation where the user expects to select the entire region of the target image 100; and the semantic object type corresponds to the situation where the user expects to select an object with specific semantic meaning, but whose precise boundaries are difficult for the user to manually delineate.

[0079] Specifically, the relationship between the preset feature conditions corresponding to each type of target and the trajectory features is as follows, please refer to them together. Figure 4 .

[0080] For the precise object type, the preset characteristic condition is a low drawing speed and a nearly closed trajectory. When the drawing speed is lower than a first speed threshold and the distance between the first and last points is lower than a closure distance threshold, the trajectory feature is considered to meet the preset characteristic condition of the precise object type. The greater the degree to which the drawing speed is lower than the first speed threshold and the greater the degree to which the distance between the first and last points is lower than the closure distance threshold, the higher the first type confidence level corresponding to the precise object type. This is because in some interactive scenarios, when a user wants to precisely select a specific object, they usually slow down the drawing speed and draw a closed trajectory along the object's outline.

[0081] For the aforementioned fuzzy range type, the preset feature condition is that the trajectory is nearly closed but the drawing speed is relatively high. When the distance between the first and last points is less than the closing distance threshold, and the drawing speed is higher than the first speed threshold, the trajectory feature is considered to satisfy the preset feature condition of the fuzzy range type. Furthermore, the greater the degree to which the drawing speed exceeds the first speed threshold, the higher the confidence level of the first type corresponding to the fuzzy range type. This is because in some interactive scenarios, when users only want to roughly define a range, they usually draw a closed loop quickly without demanding precise boundaries.

[0082] For the linear region type, the preset characteristic condition is that the trajectory is not closed and has a long and thin shape. When the distance between the first and last points is higher than the closure distance threshold, and the aspect ratio of the trajectory is higher than the aspect ratio threshold, the trajectory feature is considered to meet the preset characteristic condition of the linear region type. The greater the degree to which the distance between the first and last points is higher than the closure distance threshold, and the greater the degree to which the aspect ratio of the trajectory is higher than the aspect ratio threshold, the higher the first type confidence of the linear region type. This is because in some interactive scenarios, when a user wants to mark a linear region, they usually draw a long and thin trajectory that is not closed.

[0083] For the whole-region type, the preset characteristic condition is high drawing speed and large trajectory envelope area. When the drawing speed is higher than a second speed threshold, and the ratio of the trajectory envelope area to the area of ​​the target image 100 is higher than an area ratio threshold, the trajectory feature is considered to meet the preset characteristic condition of the whole-region type. The greater the degree to which the drawing speed exceeds the second speed threshold and the greater the degree to which the ratio exceeds the area ratio threshold, the higher the first type confidence level corresponding to the whole-region type. This is because in some interactive scenarios, when a user wants to select the entire image, they usually quickly draw a large trajectory covering most of the image. The second speed threshold is greater than the first speed threshold.

[0084] For the semantic object type, the preset feature condition is the presence of repeated outlines. When the number of repeated outlines exceeds a threshold, the trajectory feature is considered to satisfy the preset feature condition of the semantic object type. Furthermore, the greater the degree to which the number of repeated outlines exceeds the threshold, the higher the confidence level of the first type corresponding to the semantic object type. This is because in some interaction scenarios, when a user wants to select a specific semantic object but finds it difficult to accurately outline its boundaries in one go, they often repeatedly trace the object.

[0085] In some embodiments, the first speed threshold is 200 to 600 pixels per second, for example, 400 pixels per second; the second speed threshold is 800 to 1500 pixels per second, for example, 1000 pixels per second; the closure distance threshold is 2% to 5% of the diagonal length of the target image, for example, 3%; the aspect ratio threshold is 3 to 6, for example, 4; the area ratio threshold is 60% to 90%, for example, 75%; and the number of strokes threshold is 2 to 4, for example, 2.

[0086] In one specific implementation, a scoring function can be pre-defined for each object type. This scoring function uses the degree to which the trajectory features satisfy the preset feature conditions of the object type as the independent variable and the initial score of the object type as the dependent variable; the higher the degree of satisfaction, the higher the initial score. The initial scores of the five object types are then normalized so that the sum of the five scores is one, resulting in the normalized first-type confidence vector. For example, the first-type confidence vector can be represented as P1=(p11, p12, p13, p14, p15), where p11 to p15 are the first-type confidence scores corresponding to the precise object type, fuzzy range type, linear region type, whole region type, and semantic object type, respectively, and p11+p12+p13+p14+p15=1.

[0087] Step S4: Perform semantic recognition on the voice command to obtain object descriptors. Match the object descriptors with the preset descriptor sets corresponding to each object type. Calculate the second type confidence for each object type based on the matching degree between the object descriptors and each preset descriptor set, and obtain a normalized second type confidence vector.

[0088] Specifically, the electronic device performs speech recognition on the voice command to obtain text, and then extracts object descriptors from the text to describe the target object. The preset descriptor sets corresponding to each target object type are as follows: The preset descriptor set corresponding to the semantic object type includes nouns referring to specific objects, such as "cup," "that person," and "table," as well as descriptions that simultaneously contain nouns referring to specific objects and proximity indicators, such as "this cup." The proximity indicators include at least one of "this," "here," "that place," and "the enclosed object." The preset descriptor set corresponding to the precise object type includes proximity indicators that do not contain nouns referring to specific objects, such as simply "this" or "here." The preset descriptor set corresponding to the fuzzy range type includes range indicators, which include at least one of "this piece," "this area," "surroundings," and "background." The preset descriptor set corresponding to the linear region type includes linear indicators, which include at least one of "this line," "edge," and "border." The preset descriptor set corresponding to the whole region type includes global indicators, which include at least one of "all," "entire," "all," and "the whole sheet."

[0089] When the object descriptor matches a preset descriptor set corresponding to a certain object type, the second type confidence score corresponding to that object type is increased. Similar to the first type confidence score vector, the second type confidence scores corresponding to the five object types are normalized to obtain a normalized second type confidence score vector P2=(p21, p22, p23, p24, p25), where p21+p22+p23+p24+p25=1. When no object descriptor is identified, the second type confidence score vector can be set to a uniform vector with the same confidence score for each object type, or the second fusion weight can be set to 0, so that the final object type is mainly determined by the first type confidence score vector.

[0090] Step S5: Set a first fusion weight and a second fusion weight for the first type confidence vector and the second type confidence vector respectively; perform a weighted sum of the first type confidence and the second type confidence corresponding to the same target type to obtain a fusion type confidence vector; and determine the target type with the highest confidence in the fusion type confidence vector as the final target type.

[0091] Please see Figure 3Specifically, let the first fusion weight be w1 and the second fusion weight be w2, and w1 + w2 = 1. For the i-th target type, its fusion type confidence pi = w1 × p1i + w2 × p2i, where i ranges from 1 to 5. After obtaining the fusion type confidence vector P = (p1, p2, p3, p4, p5), the target type corresponding to the component with the largest value is taken as the final target type.

[0092] In some embodiments, the initial values ​​of the first fusion weight w1 and the second fusion weight w2 are equal, for example, both are 0.5. When the object descriptor is not recognized, or the semantic recognition confidence of the object descriptor is less than the semantic confidence threshold, the first fusion weight w1 is increased and the second fusion weight w2 is decreased accordingly, for example, w1 is increased to 0.8 and w2 is decreased to 0.2, so that in the case of unreliable speech recognition, the determination of the final object type relies more on the more reliable gesture trajectory channel.

[0093] The following example illustrates the function of step S5. In one scenario, a user quickly draws a closed circle while saying, "Remove this cup." According to step S3, due to the high drawing speed and the closed trajectory, the first type confidence corresponding to the fuzzy range type is high, for example, the first type confidence vector P1 = (0.20, 0.55, 0.05, 0.05, 0.15); according to step S4, since the object descriptor "this cup" contains both a concrete object noun and a proximity indicator, the second type confidence corresponding to the semantic object type is high, for example, the second type confidence vector P2 = (0.15, 0.10, 0.05, 0.05, 0.65). Taking w1=w2=0.5, the fusion type confidence vector P=(0.175,0.325, 0.05, 0.05, 0.40), where the component corresponding to the semantic object type is the largest. Therefore, the final target object type is determined to be the semantic object type. Thus, even if the user's gesture trajectory itself tends to be judged as an ambiguous range, the target object can still be corrected to a semantic object by using speech semantics. This allows the object mask corresponding to the cup to be obtained through object recognition in subsequent steps, rather than simply using the area roughly circled by the user as the region mask.

[0094] Step S6: Generate representation data of the object on the target image 100 according to the final object type. The representation data is generated in a manner appropriate to different final object types, which will be explained below and referred to in detail. Figure 5 and Figure 6 .

[0095] Please see Figure 5 , Figure 5 (a) in the figure is an inaccurate hand-drawn trajectory diagram; Figure 5 (b) in the diagram is a schematic diagram of candidate objects obtained by object identification within the envelope; Figure 5 (c) is a schematic diagram of an object mask that conforms to the real boundary. When the final object type is a precise object type, object recognition is performed on the region within the envelope 120 of the gesture trajectory 110 to obtain at least one candidate object 130. The contour of the candidate object 130 with the highest degree of conformity to the gesture trajectory 110 is determined as the object mask 140, which is used as the representation data. The degree of conformity is determined based on at least one of the following parameters: the ratio of the overlapping area between the contour of the candidate object 130 and the envelope 120 of the gesture trajectory 110; the average boundary distance between the contour of the candidate object 130 and the gesture trajectory 110; the intersection-union ratio between the contour of the candidate object 130 and the envelope 120 of the gesture trajectory 110; the distance between the center point of the candidate object 130 and the center point of the envelope 120 of the gesture trajectory 110; and the proportion of the area of ​​the candidate object 130 covered by the envelope 120 of the gesture trajectory 110. Thus, the user's imprecise hand-drawn trajectory is converged into an object mask 140 that fits the real boundary of the specific object.

[0096] When the final target object type is a semantic object type, object recognition is performed on the region within the envelope 120 of the gesture trajectory 110, or on the region within the search range expanded based on the object descriptor, to obtain at least one candidate object 130. The contour of the candidate object 130 with the highest semantic matching degree with the object descriptor is determined as the object mask 140, which serves as the representation data. The semantic matching degree is determined based on the text similarity between the category label, image recognition description text, or visual feature description text of the candidate object 130 and the object descriptor. For example, when the object descriptor is "cup," the contour of the candidate object 130 identified as "cup" within the region is determined as the object mask 140. The expanded search range based on the object descriptor is used to expand the scope of object recognition when the object referred to by the object descriptor may not be completely located within the envelope range 120 of the gesture trajectory 110. For example, when the object descriptor is a semantic object word such as "sky", "road", or "water surface", the search range is expanded to relevant areas outside the envelope range 120 in the target image 100.

[0097] Please see Figure 6 , Figure 6 (a) in the diagram is a schematic diagram of generating a region mask under the blurred range type; Figure 6(b) in the diagram is a schematic diagram of generating a region mask under the whole region type; Figure 6 (c) is a schematic diagram of generating a strip mask under the linear region type. When the final target type is a blurred range type, the envelope of the gesture trajectory 110 is expanded outward by a first tolerance to generate a region mask 150, which is used as the representation data. When the final target type is a whole region type, the entire region of the target image 100 is used as the region mask 150, which is used as the representation data. When the final target type is a linear region type, a strip mask 160 is generated with the gesture trajectory 110 as the center line and a preset width, which is used as the representation data. The preset width is determined according to the stroke width of the gesture trajectory 110, the drawing speed, or the descriptive words representing the size of the range in the voice command, wherein the larger the stroke width, the larger the preset width.

[0098] In some embodiments, when the pause point exists in the gesture trajectory 110, the representation data is generated by applying a higher fitting accuracy to the local trajectory within a preset range around the pause point than to the other trajectory segments. For example, when generating the object mask 140 or the region mask 150, denser sampling or a smaller boundary tolerance is applied to the local region around the pause point, so that the representation data fits the user's expected boundary better in that local region.

[0099] In some embodiments, step S4 further includes semantic recognition of the voice command to obtain an operation intent word. The method further includes associating the final target type, the representation data, and the operation intent word and sending them to an artificial intelligence processing module, so that the artificial intelligence processing module performs the processing corresponding to the operation intent word on the target represented by the representation data. The operation intent word is used to indicate the processing performed on the target, such as deletion, replacement, color change, recognition, question answering, or explanation. The artificial intelligence processing module can be located locally on the electronic device or on a server communicatively connected to the electronic device.

[0100] Five specific application examples are given below to further illustrate the implementation of the present invention.

[0101] Example 1: Precise Object Type. The target image 100 is a photograph containing a person, and the user wants to precisely select a watch in the photograph. The user slowly draws a nearly closed circle along the outline of the watch, but the voice command lacks a clear object descriptor or fails to recognize a valid object descriptor. According to steps S2 and S3, the drawing speed is lower than a first speed threshold and the distance between the first and last points is lower than a closing distance threshold, resulting in the highest confidence level for the first type corresponding to the precise object type. According to steps S4 and S5, since no valid object descriptor is recognized, the second fusion weight is set to 0, and the final object type is determined as the precise object type by the first type confidence vector. According to step S6, object recognition is performed on the region within the envelope 120, and the watch outline that best matches the gesture trajectory 110 is determined as the object mask 140.

[0102] Example 2, Blurred Range Type. The target image 100 is a landscape photograph. The user wants to adjust a region in the sky but does not require precise boundaries. The user quickly draws a closed circle and says, "Brighten this area a bit." According to step S3, the first type of the blurred range type has a high confidence level; according to step S4, the object descriptor "this area" matches the range indicator, and the second type of the blurred range type has a high confidence level; according to step S5, the final applied object type after fusion is the blurred range type. According to step S6, the envelope of the gesture trajectory 110 is expanded outward by a first tolerance to generate a region mask 150. Combined with the operation intent word "brighten," the artificial intelligence processing module performs brightening processing on the region represented by the region mask 150.

[0103] Example 3, Linear Region Type. The target image 100 is a photograph of a building, and the user wants to mark an edge of the roof. The user draws a thin, non-closed trajectory and says, "Trace this edge." According to step S3, the distance between the first and last points is higher than the closure distance threshold and the aspect ratio of the trajectory is higher than the aspect ratio threshold, so the first type confidence of the linear region type is high; according to step S4, the object descriptors "this" and "edge" match the linear indicator words, so the second type confidence of the linear region type is high; according to step S5, the final object type is the linear region type. According to step S6, a strip mask 160 is generated with the gesture trajectory 110 as the center line and a preset width.

[0104] Example 4: Whole Region Type. The target image 100 is a single image, and the user desires to stylize the entire image. The user quickly draws a large trajectory covering most of the image and states, "Turn the whole image into an ink painting style." According to step S3, the drawing speed is higher than the second speed threshold and the envelope area ratio is higher than the area ratio threshold, indicating a high confidence level for the first type corresponding to the whole region type. According to step S4, the object descriptor "whole image" matches the global indicator, indicating a high confidence level for the second type corresponding to the whole region type. According to step S5, the final target type is the whole region type. According to step S6, the entire region is used as a region mask 150. Combined with the operation intent phrase "turn into an ink painting style," the artificial intelligence processing module performs stylization processing on the entire image.

[0105] Example 5: Semantic Object Type, embodying the joint correction of gesture and voice channels. The target image 100 is a photo containing multiple objects, and the user wants to remove one of them, a cup. The user quickly draws a closed circle containing both the cup and part of the tabletop, and says, "Remove this cup." According to step S3, due to the high drawing speed and closed trajectory, the first type of the blurred range type has a high confidence level. According to step S4, the object descriptor "this cup" contains both a specific object noun and a proximity indicator, so the second type of the semantic object type has a high confidence level. According to step S5, following the aforementioned fusion method, the final object type is corrected to a semantic object type. According to step S6, object recognition is performed on the region within the envelope 120, and the outline of the candidate object 130 with the highest semantic matching degree to "cup" is determined as the object mask 140. Combined with the operation intent word "remove," the artificial intelligence processing module only performs deletion and content filling processing on the object mask 140 corresponding to the cup, without affecting the tabletop portion within the circle.

[0106] Please see Figure 7 This invention also provides an electronic device 200, which includes a processor 210, a memory 220, and a computer program 230 stored in the memory 220 and executable on the processor 210. When the processor 210 executes the computer program 230, it implements the method described in any of the above embodiments. The electronic device 200 can be a desktop computer, a laptop computer, a tablet computer, an all-in-one computer, or other terminals connected to the above devices; this invention does not limit this.

[0107] This invention also provides a computer-readable storage medium storing a computer program 230 thereon, which, when executed by a processor, implements the method described in any of the above embodiments. The computer-readable storage medium may be a read-only memory, random access memory, magnetic disk, optical disk, flash memory, or solid-state drive, etc.

[0108] It should be noted that the technical features in the above embodiments can be combined arbitrarily without contradiction. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as the combination of these technical features does not contradict each other, they should be considered to be within the scope of this specification.

[0109] The embodiments described above are merely illustrative of several implementations of the present invention, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of protection of the present invention. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of the present invention, and these modifications and improvements all fall within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the appended claims.

Claims

1. A method for determining an object by combining gestures and voice, applied to an electronic device, wherein a target image is displayed on the display interface of the electronic device; characterized in that, include: S1: Obtain the gesture trajectory and associated voice command input by the user for the target image; S2: Extract the trajectory features of the gesture trajectory, including the drawing speed of the gesture trajectory, the distance between the first point and the last point, the aspect ratio of the trajectory, the area of ​​the trajectory envelope, and the number of repeated strokes; S3: Match the trajectory features with the preset feature conditions corresponding to each of the precise object type, fuzzy range type, linear region type, whole region type and semantic object type, and obtain the normalized first type confidence vector based on the degree to which the trajectory features satisfy each preset feature condition; S4: Perform semantic recognition on the voice command to obtain object descriptors, match the object descriptors with the preset descriptor sets corresponding to each type of object, and obtain a normalized second type confidence vector based on the degree of matching between the object descriptors and each preset descriptor set. S5: Set first and second fusion weights for the first and second type confidence vectors respectively, and sum the first and second type confidences of the same target type to obtain the fusion type confidence vector, and determine the target type with the highest confidence as the final target type; When no object descriptor is identified or its semantic recognition confidence is below a threshold, the first fusion weight is increased and the second fusion weight is decreased. S6: Generate representation data of the target object on the target image according to the final target object type, wherein the representation data is at least one of object mask, region mask or strip mask.

2. The method according to claim 1, characterized in that, In step S3, the first type of confidence score is calculated based on the degree to which the trajectory features satisfy each of the preset feature conditions, including: The confidence level of the first type corresponding to the precise object type increases as the degree to which the drawing speed is lower than the first speed threshold increases, and as the degree to which the distance between the first point and the last point is lower than the closing distance threshold increases. The first type confidence level corresponding to the fuzzy range type increases as the drawing speed exceeds the first speed threshold and when the distance between the first point and the last point is less than the closure distance threshold. The first type confidence level corresponding to the linear region type increases as the distance between the first point and the last point is greater than the closure distance threshold and as the aspect ratio of the trajectory is greater than the aspect ratio threshold. The first type confidence level corresponding to the whole region type increases as the degree to which the drawing speed is higher than the second speed threshold increases, and as the degree to which the ratio of the trajectory envelope area to the target image area is higher than the area ratio threshold increases. The confidence level of the first type corresponding to the semantic object type increases as the degree to which the number of repeated strokes exceeds the stroke count threshold increases; Wherein, the second speed threshold is greater than the first speed threshold.

3. The method according to claim 1, characterized in that, In step S4, the second type of confidence score is calculated based on the matching degree between the object descriptor and each preset descriptor set, including: When the object descriptor is a noun referring to a specific object, the second type confidence corresponding to the semantic object type is increased; when the object descriptor contains both a noun referring to a specific object and a proximity indicator, the second type confidence corresponding to the semantic object type is increased, wherein the proximity indicator includes at least one of this, here, that place, and the circled object; When the object descriptor is a proximity indicator that does not contain a specific object name, the second type confidence corresponding to the precise object type is increased; When the object descriptor is a range indicator, increase the second type confidence corresponding to the fuzzy range type. The range indicator includes at least one of "this block", "this area", "surrounding", and "background". When the object descriptor is a linear indicator, increase the second type confidence corresponding to the linear region type. The linear indicator includes at least one of the following: line, edge, and border. When the object descriptor is a global indicator, the second type confidence corresponding to the whole region type is increased. The global indicator includes at least one of all, whole, all and whole sheet.

4. The method according to claim 1, characterized in that, In step S2, The drawing speed is the ratio of the distance between adjacent sampling points of the gesture trajectory to the sampling time interval, or the ratio of the total length of the gesture trajectory to the drawing time of the gesture trajectory; The distance between the first point and the last point is the straight-line distance between the starting sampling point and the ending sampling point of the gesture trajectory; The aspect ratio of the trajectory is the ratio of the longer side to the shorter side of the smallest bounding rectangle of the gesture trajectory; When the gesture trajectory is a closed trajectory, the trajectory envelope area is the area enclosed by the closed trajectory; when the gesture trajectory is a non-closed trajectory, the trajectory envelope area is the convex hull area or the area of ​​the smallest circumscribed polygon of the gesture trajectory. The number of repeated strokes refers to the number of times the gesture trajectory intersects or repeatedly passes through the same local area.

5. The method according to claim 1, characterized in that, The trajectory features in step S2 also include pause points, which are sampling points corresponding to the duration of the continuous drawing speed being less than the pause speed threshold being greater than or equal to the pause duration threshold; in step S6, when the pause point exists in the gesture trajectory, the characterization data is generated by applying a higher fitting accuracy to the local trajectory within a preset range around the pause point than to the other trajectory segments.

6. The method according to claim 1, characterized in that, Step S6 includes: When the final target type is a precise target type, object recognition is performed on the region within the envelope of the gesture trajectory to obtain at least one candidate object, and the contour of the candidate object with the highest degree of fit with the gesture trajectory is determined as the object mask as the representation data. When the final target type is a semantic object type, object recognition is performed on the region within the envelope of the gesture trajectory to obtain at least one candidate object, and the contour of the candidate object with the highest semantic matching degree with the object descriptor is determined as the object mask as the representation data. When the final target type is a fuzzy range type or a whole region type, a region mask is generated as the representation data. When the final target type is a linear region, a strip mask generated with the gesture trajectory as the center line and a preset width will be used as the representation data. The degree of fit is determined based on at least one of the following parameters: the percentage of overlap between the outline of the candidate object and the envelope of the gesture trajectory; the average boundary distance between the outline of the candidate object and the gesture trajectory; the intersection-union ratio between the outline of the candidate object and the envelope of the gesture trajectory; the distance between the center point of the candidate object and the center point of the gesture trajectory envelope; and the percentage of the area of ​​the candidate object covered by the gesture trajectory envelope. The semantic matching degree is determined based on the text similarity between the candidate object's category label, image recognition description text, or visual feature description text and the object's descriptive words.

7. The method according to claim 6, characterized in that, The region mask and / or strip mask are generated in the following manner: When the final target type is a fuzzy range type, the region mask is generated by expanding the envelope of the gesture trajectory outward by a first tolerance. When the final target type is a whole region type, the entire region of the target image is used as the region mask; When the final target type is a linear region, the preset width of the strip mask is determined based on the stroke width of the gesture trajectory, the drawing speed, or the descriptive words representing the size of the range in the voice command, wherein the larger the stroke width, the larger the preset width.

8. The method according to claim 1, characterized in that, Step S4 further includes semantic recognition of the voice command to obtain an operation intent word; the method further includes: associating the final target type, the representation data and the operation intent word and sending them to the artificial intelligence processing module, so that the artificial intelligence processing module performs the processing corresponding to the operation intent word on the target represented by the representation data; wherein, the gesture trajectory and the voice command are acquired within the same interaction cycle, and the same interaction cycle is determined by the input device's press event, touch event or quick trigger event.

9. An electronic device, characterized in that, The electronic device includes a processor, a memory, and a computer program stored in the memory and executable on the processor, wherein the computer program, when executed by the processor, implements the method as described in any one of claims 1 to 8.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the method as described in any one of claims 1 to 8.

Images