RPA robot state judgment method and device based on multi-modal recognition
By matching the images of the soft phone login interface with the pre-stored images, and using template matching and feature point recognition algorithms to determine the login status, the problem of soft phone login failure detection is solved, ensuring the continuity of outbound call services.
Patent Information
- Application Number
- CN202411037812.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-31
- Publication Date
- 2024-11-19
AI Technical Summary
During soft phone login, in the case of network failure, etc., it is impossible to quickly detect whether the login is successful, resulting in an outgoing call service interruption.
By responding to the login command, the image of the soft phone login interface is obtained and matched with the pre-stored images to determine whether the login is successful or not, and an image recognition algorithm based on template matching and feature points is used to improve accuracy and handle login failures in a timely manner.
It realizes quick judgment of the login status of soft phones, ensures the normal execution of outbound call services, and reduces the number of interruptions and re-login times caused by login failure.
Smart Images

Figure CN118984352A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of mobile communication technology, and in particular to a method and device for determining the state of an RPA robot based on multimodal recognition. Background Art
[0002] With the continuous development of mobile technology, there are more and more softphone products. You can use softphone products to automatically make calls. Before making a call, you can log in to your user account. Only after the login is successful can you make a call. However, in some cases, such as network failure, you cannot log in normally. How to quickly detect whether the login is successful is an urgent problem that needs to be solved. Summary of the invention
[0003] The purpose of some embodiments of the present application is to provide an RPA robot state judgment method and device based on multimodal recognition. Through the technical solution of the embodiments of the present application, by responding to the login instruction, according to the softphone login interface, a first image of the softphone login interface is obtained; according to the first image and the pre-stored second image, whether the first image contains the second image is judged; if the first image contains the second image, it is determined that the softphone login is successful. The embodiments of the present application capture an image of the softphone login interface and match it with a pre-stored image. If the captured image of the softphone login interface contains the pre-stored image, it means that the softphone login is successful. By judging whether the login is successful, the unsuccessful login situation can be handled in time, so that the outbound call service can be executed normally.
[0004] In a first aspect, some embodiments of the present application provide an RPA robot state judgment method based on multimodal recognition, including:
[0005] In response to the login instruction, acquiring a first image of the softphone login interface according to the softphone login interface;
[0006] According to the first image and a pre-stored second image, determining whether the first image contains the second image;
[0007] If the first image includes the second image, it is determined that the softphone login is successful.
[0008] Some embodiments of the present application capture an image of the softphone login interface and match it with a pre-stored image. If the captured image of the softphone login interface contains the pre-stored image, it means that the softphone login is successful. By judging whether the login is successful, the situation of unsuccessful login can be handled in time, so that the outbound call service can be executed normally.
[0009] Optionally, judging, according to the first image and a pre-stored second image, whether the first image contains the second image includes:
[0010] According to a pre-stored image recognition algorithm, the first image is recognized to obtain a recognition result corresponding to the first image; wherein the pre-stored image recognition algorithm at least includes an image recognition algorithm based on a template matching algorithm and an image recognition algorithm based on a feature point;
[0011] The recognition result and the second image are matched.
[0012] Some embodiments of the present application use a pre-stored image recognition algorithm to recognize the content in the captured first image, and then determine whether the recognition result contains the second image, thereby improving the recognition accuracy.
[0013] Optionally, the method further comprises:
[0014] If the first image does not include the second image, it is determined that the softphone login is unsuccessful.
[0015] In some embodiments of the present application, the first image and the second image are matched. If the first image does not contain the second image, it means that the softphone has not been successfully logged in, and the screenshot process needs to be re-performed.
[0016] Optionally, the method further comprises:
[0017] If the softphone login fails, re-screenshot the softphone login interface to obtain a third image of the softphone login interface;
[0018] The third image is matched with the pre-stored second image.
[0019] In some embodiments of the present application, the first image and the second image are matched. If the first image does not contain the second image, it means that the softphone has not logged in successfully, and the screenshot process needs to be performed again until the login is successful.
[0020] Optionally, the method further comprises:
[0021] If the softphone login fails within the preset login times, a login failure message is sent to the server device.
[0022] In some embodiments of the present application, if a login attempt is unsuccessful once, a screenshot is taken again and matched with a pre-stored image. If a login attempt is unsuccessful within a preset number of times, a login failure message is sent to the server device to find the cause of the failure.
[0023] In a second aspect, some embodiments of the present application provide an RPA robot state judgment device based on multimodal recognition, including:
[0024] An acquisition module, configured to acquire a first image of the softphone login interface according to the softphone login interface in response to the login instruction;
[0025] a judging module, configured to judge whether the first image contains the second image according to the first image and a pre-stored second image;
[0026] The determination module is configured to determine that the softphone login is successful if the first image includes the second image.
[0027] Some embodiments of the present application capture an image of the softphone login interface and match it with a pre-stored image. If the captured image of the softphone login interface contains the pre-stored image, it means that the softphone login is successful. By judging whether the login is successful, the situation of unsuccessful login can be handled in time, so that the outbound call service can be executed normally.
[0028] Optionally, the judging module is used to:
[0029] According to a pre-stored image recognition algorithm, the first image is recognized to obtain a recognition result corresponding to the first image; wherein the pre-stored image recognition algorithm at least includes an image recognition algorithm based on a template matching algorithm and an image recognition algorithm based on a feature point;
[0030] The recognition result and the second image are matched.
[0031] Some embodiments of the present application use a pre-stored image recognition algorithm to recognize the content in the captured first image, and then determine whether the recognition result contains the second image, thereby improving the recognition accuracy.
[0032] Optionally, the determining module is used to:
[0033] If the first image does not include the second image, it is determined that the softphone login is unsuccessful.
[0034] In some embodiments of the present application, the first image and the second image are matched. If the first image does not contain the second image, it means that the softphone has not been successfully logged in, and the screenshot process needs to be re-performed.
[0035] Optionally, the judging module is further used for:
[0036] If the softphone login fails, re-screenshot the softphone login interface to obtain a third image of the softphone login interface;
[0037] The third image is matched with the pre-stored second image.
[0038] In some embodiments of the present application, the first image and the second image are matched. If the first image does not contain the second image, it means that the softphone has not logged in successfully, and the screenshot process needs to be performed again until the login is successful.
[0039] Optionally, the judging module is used to:
[0040] If the softphone login fails within the preset login times, a login failure message is sent to the server device.
[0041] In some embodiments of the present application, if a login attempt is unsuccessful once, a screenshot is taken again and matched with a pre-stored image. If a login attempt is unsuccessful within a preset number of times, a login failure message is sent to the server device to find the cause of the failure.
[0042] In a third aspect, some embodiments of the present application provide an electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the RPA robot state judgment method based on multimodal recognition as described in any embodiment of the first aspect can be implemented.
[0043] In a fourth aspect, some embodiments of the present application provide a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, can implement the RPA robot state judgment method based on multimodal recognition as described in any embodiment of the first aspect.
[0044] In a fifth aspect, some embodiments of the present application provide a computer program product, comprising a computer program, wherein when the computer program is executed by a processor, it can implement the RPA robot state judgment method based on multimodal recognition as described in any embodiment of the first aspect. BRIEF DESCRIPTION OF THE DRAWINGS
[0045] In order to more clearly illustrate the technical solutions of some embodiments of the present application, the drawings required for use in some embodiments of the present application will be briefly introduced below. It should be understood that the following drawings only show some embodiments of the present application and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.
[0046] Figure 1 A flowchart of a method for determining the state of an RPA robot based on multimodal recognition provided in an embodiment of the present application;
[0047] Figure 2 A flowchart of another RPA robot state judgment method based on multimodal recognition provided in an embodiment of the present application;
[0048] Figure 3 A schematic diagram showing a softphone login interface provided in an embodiment of the present application;
[0049] Figure 4 Another display schematic diagram provided for an embodiment of the present application;
[0050] Figure 5 Another display schematic diagram provided for an embodiment of the present application;
[0051] Figure 6 A schematic diagram of the structure of an RPA robot state judgment device based on multimodal recognition provided in an embodiment of the present application;
[0052] Figure 7 A schematic diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION
[0053] The technical solutions in some embodiments of the present application will be described below in conjunction with the drawings in some embodiments of the present application.
[0054] It should be noted that similar reference numerals and letters represent similar items in the following drawings, so once an item is defined in one drawing, it does not need to be further defined and explained in subsequent drawings. At the same time, in the description of this application, the terms "first", "second", etc. are only used to distinguish the description and cannot be understood as indicating or implying relative importance.
[0055] With the continuous development of mobile technology, there are more and more softphone products. Calls can be made automatically through softphone products. Before making a call, you can log in to your user account. You can make a call only after the login is successful. However, in some cases, such as network failure, you cannot log in normally. In view of this, some embodiments of the present application provide an RPA robot state judgment method based on multimodal recognition, the method comprising: responding to a login instruction, according to the softphone login interface, obtaining a first image of the softphone login interface; judging whether the first image contains the second image according to the first image and a pre-stored second image; if the first image contains the second image, it is determined that the softphone login is successful. The embodiment of the present application captures an image of the softphone login interface and matches it with a pre-stored image. If the captured image of the softphone login interface contains the pre-stored image, it means that the softphone login is successful. By judging whether the login is successful, the unsuccessful login situation can be handled in time, so that the outbound call service can be performed normally.
[0056] like Figure 1As shown, an embodiment of the present application provides an RPA robot state judgment method based on multimodal recognition, the method comprising:
[0057] S101, in response to a login instruction, acquiring a first image of a softphone login interface according to a softphone login interface;
[0058] Specifically, a softphone is installed on the terminal device, which is a software for performing outbound call services. After Airtest connects to the softphone, it can simulate a person operating the softphone, enter the account and password, and click the login button. The terminal device responds to the login instruction and takes a screenshot of the softphone login interface to obtain the first image of the softphone login interface.
[0059] S102, judging whether the first image contains the second image according to the first image and the pre-stored second image;
[0060] Specifically, the second image pre-stored on the terminal device may be any part of the softphone login interface. After acquiring the first image, the terminal device identifies the first image and determines whether the first image contains the second image.
[0061] S103: If the first image includes the second image, it is determined that the softphone login is successful.
[0062] Specifically, if the terminal device determines that the first image contains the second image, it means that the softphone login is successful.
[0063] For example, after clicking the login button on the softphone, Airtest will take a screenshot and then Figure 5 Match in the captured pictures and see Figure 5 Is it part of the captured image? If it can be matched, it means that the softphone login is successful.
[0064] Some embodiments of the present application capture an image of the softphone login interface and match it with a pre-stored image. If the captured image of the softphone login interface contains the pre-stored image, it means that the softphone login is successful. By judging whether the login is successful, the situation of unsuccessful login can be handled in time, so that the outbound call service can be executed normally.
[0065] Another embodiment of the present application further supplements the RPA robot state judgment method based on multimodal recognition provided in the above embodiment.
[0066] Optionally, judging, according to the first image and a pre-stored second image, whether the first image contains the second image includes:
[0067] According to a pre-stored image recognition algorithm, the first image is recognized to obtain a recognition result corresponding to the first image; wherein the pre-stored image recognition algorithm at least includes an image recognition algorithm based on a template matching algorithm and an image recognition algorithm based on a feature point;
[0068] The recognition result is matched with the second image.
[0069] Specifically, the terminal device uses a pre-stored image recognition algorithm to identify the first image. In the process of identifying the first image, Airtest uses the SURFMatching, TemplateMatching, and BRISKMatching algorithms to search. For example, in the execution result of the TemplateMatching algorithm, there is a "confidence": 0.4586929976940155, which is the credibility of the algorithm;
[0070] When the program is performing image recognition, it will return a matching result, which is the credibility. As the name suggests, it is the similarity of the images. The value range is [0, 11]. The threshold is an attribute in each image script, which is used to determine whether the image is successfully recognized. If it is not changed manually, the default value is 0.7, and the value range is also [0, 1].
[0071] When the image recognition algorithm is executed, a credibility will be calculated. When the credibility > threshold value, the program determines that the best match result has been found and stops matching. When the credibility < threshold value, it will be considered that the best match result has not been found and other algorithms will continue to match. When different recognition algorithms do not meet the requirements, the program returns the error message of TargetNotFoundError recognition failure.
[0072] If the credibility of the algorithm recognition (that is, confidence) exceeds the set threshold (default 0.7), the recognition is considered successful, otherwise it is considered a failure. The image recognition here uses the open source OpenCV library, and the default algorithm is CVSTRATEGY = ["surf", "tpl", "brisk"]. Every time when searching for an image, Airtest will execute it in the order of the set algorithms until it finds a recognition result that meets the set threshold, or it will keep searching in a loop until it times out.
[0073] Different types of image recognition algorithms are integrated into the Airtest framework, including template matching (i.e. Templatematching) and feature point-based image recognition methods (i.e. suRFMatching and BRIskMatching).
[0074] Some embodiments of the present application use a pre-stored image recognition algorithm to recognize the content in the captured first image, and then determine whether the recognition result contains the second image, thereby improving the recognition accuracy.
[0075] Optionally, the method further comprises:
[0076] If the first image does not include the second image, it is determined that the softphone login is unsuccessful.
[0077] Specifically, after Airtest is connected to the softphone on the terminal device, you can simulate a person to operate the softphone, enter the account and password, and then click the login button. After waiting for a preset time, for example, 3 seconds, you can log in successfully. Figure 4 If the account password is incorrect or there are other abnormalities, it will stay at Figure 3 The softphone interface of Airtest is based on image recognition. After connecting to the softphone, you can capture pictures. After clicking the login button, wait for 3 seconds and take a screenshot. The screenshot may be Figure 3 , or it could be Figure 4 .
[0078] In some embodiments of the present application, the first image and the second image are matched. If the first image does not contain the second image, it means that the softphone has not been successfully logged in, and the screenshot process needs to be re-performed.
[0079] Optionally, the method further comprises:
[0080] If the softphone login fails, re-screenshot the softphone login interface to obtain a third image of the softphone login interface;
[0081] The third image is matched with the pre-stored second image.
[0082] in, Figure 5 It is pre-login successful interface Figure 1 A part of the interception can be Figure 4 Any part of the image is only the only sign of successful softphone login. This part is stored in advance to facilitate RPA automated business processes to check whether this part of the image is included in the image captured by Airtest.
[0083] In some embodiments of the present application, the first image and the second image are matched. If the first image does not contain the second image, it means that the softphone has not logged in successfully, and the screenshot process needs to be performed again until the login is successful.
[0084] Optionally, the method further comprises:
[0085] If the softphone login fails within the preset login times, a login failure message is sent to the server device.
[0086] Specifically, if the captured image does not match the pre-stored image, the steps for logging into the softphone will be repeated up to 10 times. If the login is not successful after 10 times, the server IP and account information will be pushed to the enterprise WeChat group, and then the R&D personnel will log in to the corresponding server to check the specific situation to avoid affecting the outbound call operation.
[0087] In some embodiments of the present application, if a login attempt is unsuccessful once, a screenshot is taken again and matched with a pre-stored image. If a login attempt is unsuccessful within a preset number of times, a login failure message is sent to the server device to find the cause of the failure.
[0088] Figure 2 A flowchart of another RPA robot state judgment method based on multimodal recognition provided in an embodiment of the present application is shown in FIG. Figure 2 As shown, the RPA robot state judgment method based on multimodal recognition includes:
[0089] S201, open the softphone and display the login interface;
[0090] S202. Open Airtest and connect the softphone;
[0091] S203, Airtest simulates mouse and keyboard operation to input account and password;
[0092] S204, click the login button;
[0093] Specifically, open the softphone, display the login interface, open Airtest (a cross-platform UI automation testing framework) and connect to the softphone application window, then Airtest simulates mouse and keyboard operations to enter the number and password, and click the login button.
[0094] S205, wait for 3 seconds, and use Airtest to capture the softphone interface. Figure 5 The picture is matched with the captured picture. If it can be matched, it means that the softphone has logged in successfully. Otherwise, repeat the above steps up to 10 times. If the login is not successful after 10 times, the server IP and account information will be pushed to the enterprise WeChat group, and then the R&D personnel will log in to the corresponding server to check the specific situation to avoid affecting the outbound call operation.
[0095] After clicking the softphone login button, Airtest will capture the image. If the account is incorrect or there is no response after clicking the login button, the image captured by Airtest is Figure 3 and Figure 4 , Figure 5 It is a part of the interface after the softphone successfully logs in. It is fixed and will not change. Figure 3 and Figure 4 The pictures are respectively Figure 5 If the pictures are compared, Figure 5 If the image contains Figure 5 , it means that the login was unsuccessful.
[0096] Among them, Airtest is a UI automation testing framework based on image recognition. Its main function is to operate applications and software by simulating user operations. RPA (Robotic Process Automation) is an automated business process, and Airtest is one of the frameworks and modules used to write RPA.
[0097] It should be noted that each implementable method in this embodiment may be implemented separately, or may be implemented in combination in any combination without conflict, and this application is not limited thereto.
[0098] Another embodiment of the present application provides an RPA robot state judgment device based on multimodal recognition, which is used to execute the RPA robot state judgment method based on multimodal recognition provided by the above embodiment.
[0099] like Figure 6 , which is a schematic diagram of the structure of the RPA robot state judgment device based on multimodal recognition provided by an embodiment of the present application. The RPA robot state judgment device based on multimodal recognition includes an acquisition module 601, a judgment module 602 and a determination module 603, wherein:
[0100] The acquisition module 601 is used to acquire a first image of the softphone login interface according to the softphone login interface in response to the login instruction;
[0101] The determination module 602 is used to determine whether the first image contains the second image according to the first image and the pre-stored second image;
[0102] The determination module 603 is configured to determine that the softphone login is successful if the first image includes the second image.
[0103] Regarding the device in this embodiment, the specific manner in which each module performs operations has been described in detail in the embodiment of the method, and will not be elaborated here.
[0104] Some embodiments of the present application capture an image of the softphone login interface and match it with a pre-stored image. If the captured image of the softphone login interface contains the pre-stored image, it means that the softphone login is successful. By judging whether the login is successful, the situation of unsuccessful login can be handled in time, so that the outbound call service can be executed normally.
[0105] Another embodiment of the present application further supplements the RPA robot state judgment device based on multimodal recognition provided in the above embodiment.
[0106] Optionally, the judgment module is used to:
[0107] According to a pre-stored image recognition algorithm, the first image is recognized to obtain a recognition result corresponding to the first image; wherein the pre-stored image recognition algorithm at least includes an image recognition algorithm based on a template matching algorithm and an image recognition algorithm based on a feature point;
[0108] The recognition result is matched with the second image.
[0109] Some embodiments of the present application use a pre-stored image recognition algorithm to recognize the content in the captured first image, and then determine whether the recognition result contains the second image, thereby improving the recognition accuracy.
[0110] Optionally, a module is determined to:
[0111] If the first image does not include the second image, it is determined that the softphone login is unsuccessful.
[0112] In some embodiments of the present application, the first image and the second image are matched. If the first image does not contain the second image, it means that the softphone has not been successfully logged in, and the screenshot process needs to be re-performed.
[0113] Optionally, the judging module is further used for:
[0114] If the softphone login fails, re-screenshot the softphone login interface to obtain a third image of the softphone login interface;
[0115] The third image is matched with the pre-stored second image.
[0116] In some embodiments of the present application, the first image and the second image are matched. If the first image does not contain the second image, it means that the softphone has not logged in successfully, and the screenshot process needs to be performed again until the login is successful.
[0117] Optionally, the judgment module is used to:
[0118] If the softphone login fails within the preset login times, a login failure message is sent to the server device.
[0119] In some embodiments of the present application, if a login attempt is unsuccessful once, a screenshot is taken again and matched with a pre-stored image. If a login attempt is unsuccessful within a preset number of times, a login failure message is sent to the server device to find the cause of the failure.
[0120] Regarding the device in this embodiment, the specific manner in which each module performs operations has been described in detail in the embodiment of the method, and will not be elaborated here.
[0121] It should be noted that each implementable method in this embodiment may be implemented separately, or may be implemented in combination in any combination without conflict, and this application is not limited thereto.
[0122] An embodiment of the present application also provides a computer-readable storage medium on which a computer program is stored. When the program is executed by a processor, it can implement the operations corresponding to any embodiment of the RPA robot state judgment method based on multimodal recognition provided in the above embodiments.
[0123] An embodiment of the present application also provides a computer program product, which includes a computer program, wherein when the computer program is executed by a processor, it can implement the operations corresponding to any embodiment of the RPA robot state judgment method based on multimodal recognition provided in the above embodiment.
[0124] like Figure 7 As shown, some embodiments of the present application provide an electronic device 700, which includes: a memory 710, a processor 720, and a computer program stored in the memory 710 and executable on the processor 720, wherein the processor 720 reads the program from the memory 710 through a bus 730 and executes the program to implement a method of any embodiment of the RPA robot state judgment method based on multimodal recognition as described above.
[0125] Processor 720 can process digital signals and can include various computing structures, such as complex instruction set computer structure, reduced instruction set computer structure, or a structure that implements a combination of multiple instruction sets. In some examples, processor 720 can be a microprocessor.
[0126] The memory 710 may be used to store instructions executed by the processor 720 or data related to the execution of instructions. These instructions and / or data may include codes for implementing some or all functions of one or more modules described in the embodiments of the present application. The processor 720 of the disclosed embodiment may be used to execute instructions in the memory 710 to implement the method shown above. The memory 710 includes a dynamic random access memory, a static random access memory, a flash memory, an optical memory, or other memory known to those skilled in the art.
[0127] The above are only embodiments of the present application and are not intended to limit the scope of protection of the present application. For those skilled in the art, the present application may have various changes and variations. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present application should be included in the scope of protection of the present application. It should be noted that similar reference numerals and letters represent similar items in the following drawings, so once an item is defined in one drawing, it does not need to be further defined and explained in the subsequent drawings.
[0128] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.
[0129] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.
Claims
1. A method for determining the state of an RPA robot based on multimodal recognition, characterized in that: The method comprises: In response to the login instruction, acquiring a first image of the softphone login interface according to the softphone login interface; According to the first image and a pre-stored second image, determining whether the first image contains the second image; If the first image includes the second image, it is determined that the softphone login is successful.
2. The RPA robot state judgment method based on multimodal recognition according to claim 1 is characterized in that: The determining, based on the first image and a pre-stored second image, whether the first image contains the second image includes: According to a pre-stored image recognition algorithm, the first image is recognized to obtain a recognition result corresponding to the first image; wherein the pre-stored image recognition algorithm at least includes an image recognition algorithm based on a template matching algorithm and an image recognition algorithm based on a feature point; The recognition result and the second image are matched.
3. The RPA robot state judgment method based on multimodal recognition according to claim 1 is characterized in that: The method further comprises: If the first image does not include the second image, it is determined that the softphone login is unsuccessful.
4. The RPA robot state judgment method based on multimodal recognition according to claim 3 is characterized in that: The method further comprises: If the softphone login fails, re-screenshot the softphone login interface to obtain a third image of the softphone login interface; The third image is matched with the pre-stored second image.
5. The RPA robot state judgment method based on multimodal recognition according to claim 1 is characterized in that: The method further comprises: If the softphone login fails within the preset login times, a login failure message is sent to the server device.
6. An RPA robot state judgment device based on multimodal recognition, characterized in that: The device comprises: An acquisition module, configured to acquire a first image of the softphone login interface according to the softphone login interface in response to the login instruction; a judging module, configured to judge whether the first image contains the second image according to the first image and a pre-stored second image; The determination module is configured to determine that the softphone login is successful if the first image includes the second image.
7. The RPA robot state judgment device based on multimodal recognition according to claim 6 is characterized in that: The judging module is used for: According to a pre-stored image recognition algorithm, the first image is recognized to obtain a recognition result corresponding to the first image; wherein the pre-stored image recognition algorithm at least includes an image recognition algorithm based on a template matching algorithm and an image recognition algorithm based on a feature point; The recognition result and the second image are matched.
8. The RPA robot state judgment device based on multimodal recognition according to claim 6 is characterized in that: The determining module is used to: If the first image does not include the second image, it is determined that the softphone login is unsuccessful.
9. An electronic device, characterized in that: It includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the RPA robot state judgment method based on multimodal recognition as described in any one of claims 1 to 5 can be implemented.
10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, wherein when the program is executed by a processor, the RPA robot state judgment method based on multimodal recognition as described in any one of claims 1 to 5 can be implemented.
