Information determination method, device and equipment and computer readable storage medium

Abstract

The embodiment of the invention discloses an information determination method, and the method comprises the steps: receiving an alarm request sent by an alarm object through a terminal, and obtaining the call information of the alarm object based on the alarm request; sending target page address information corresponding to the alarm request to the terminal based on the call information; acquiring target image information of the alarm object under the condition that response information which is sent by the alarm object and aims at the target page address information is received; and sign language recognition is performed on the target image information based on the target feature recognition model to obtain the target alarm information of the alarm object, so that the alarm process is more convenient, and the alarm efficiency is higher. The embodiment of the invention further discloses an information determination device and equipment and a computer readable storage medium.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to an information determination method, device, equipment and computer-readable storage medium. Background Art

[0002] Usually, the deaf-mute people report an alarm through the alarm help application (Application, APP) installed in the alarm help mobile phone. Specifically, the deaf-mute people first send an alarm request to the alarm help platform through the alarm help App, and then the alarm help platform sends a response message of the alarm request to the alarm help App, and the response message is displayed in the form of sign language video, that is, the alarm help platform communicates with the deaf-mute people through video sign language to achieve the purpose of the deaf-mute people reporting an alarm for help; however, in the alarm method of using the Alarm Help App, the Alarm Help App must be installed in the alarm help mobile phone in order for the deaf-mute people to report an alarm, and the Alarm Help App needs to be updated later, which makes the alarm process more complicated. Summary of the invention

[0003] In order to solve the above technical problems, the embodiments of the present application hope to provide an information determination method, device, equipment and computer-readable storage medium, which can solve the problem of complex alarm process in related technologies, make the alarm process more convenient and the alarm efficiency higher.

[0004] The technical solution of this application is implemented as follows:

[0005] A method for determining information, the method comprising:

[0006] Receiving an alarm request sent by an alarm object through a terminal, and acquiring call information of the alarm object based on the alarm request;

[0007] Based on the call information, sending the target page address information corresponding to the alarm request to the terminal;

[0008] Upon receiving the response information sent by the alarm object with respect to the target page address information, acquiring the target image information of the alarm object;

[0009] The target image information is subjected to sign language recognition based on a target feature recognition model to obtain target alarm information of the alarm object.

[0010] In the above solution, the step of sending the target page address information corresponding to the alarm request to the terminal based on the call information includes:

[0011] In the case that the target content is not extracted from the call information of the target call duration, the target page address information is sent to the terminal.

[0012] In the above solution, the step of obtaining the target image information of the alarm object includes:

[0013] Sending selection information for determining whether the alarmer has a physical dysfunction to the terminal;

[0014] When confirmation information of the alarm target in response to the selection information is received, target image information of the alarm target is acquired.

[0015] In the above solution, upon receiving confirmation information from the alarm object regarding the selection information, obtaining target image information of the alarm object includes:

[0016] Receiving first image information of the alarm object sent by the terminal; wherein the first image information is collected by the terminal using a first image collection module;

[0017] In a case where the content of the first image information meets the target image content, determining the first image information as the target image information;

[0018] In the case that the content of the first image information does not satisfy the target image content, updated second image information of the alarm object collected by the terminal is obtained, and the target image information is determined based on the second image information.

[0019] In the above solution, the step of acquiring updated second image information of the alarm object collected by the terminal and determining the target image information based on the second image information includes:

[0020] Sending an adjustment instruction for adjusting the posture and / or position of the alarm object to the terminal;

[0021] When the first sub-image information of the alarm object sent by the terminal is received within the target time period and the content of the first sub-image information meets the target image content, determining the first sub-image information as the target image information; wherein the second image information includes the first sub-image information;

[0022] If the first sub-image information is not received within the target time period, or the content of the first sub-image information does not meet the target image content, sending a switching instruction for switching the image acquisition module to the terminal;

[0023] Receive the second sub-image information of the alarm object collected by the second image acquisition module sent by the terminal, and determine that the second sub-image information is the target image information; wherein, the second image acquisition module is the image acquisition module after the terminal switches based on the switching instruction; the second image information includes the second sub-image information.

[0024] In the above scheme, the method further comprises:

[0025] Collect multiple sample images of hands and torsos of various hand modes of sample subjects in different environments; wherein the multiple sample images are images of hand gestures and torso images of the sample subjects in communication;

[0026] Based on multiple sample images of each hand pattern, the first feature extraction network and the second feature extraction network in the initial feature recognition model are trained to obtain a target feature recognition model corresponding to each hand pattern; wherein the first feature extraction network is a network capable of extracting features in time and space dimensions; and the second feature extraction network is a convolutional neural network.

[0027] In the above scheme, the first feature extraction network and the second feature extraction network in the initial feature recognition model are trained based on multiple sample images of each hand pattern to obtain the target feature recognition model corresponding to each hand pattern, including:

[0028] For the multiple sample images of each hand pattern, each sample image of each hand pattern is identified based on a target recognition algorithm, and a target area of ​​a key part of the alarm object in each sample image is determined;

[0029] Extracting the image of the target area in each sample image of each hand pattern to obtain a plurality of to-be-processed images of each hand pattern;

[0030] Based on the multiple images to be processed corresponding to each hand pattern and the multiple sample images of each hand pattern, model training is performed on the first feature extraction network and the second feature extraction network to obtain a target feature recognition model corresponding to each hand pattern.

[0031] In the above scheme, the first feature extraction network and the second feature extraction network are trained based on the multiple images to be processed of each hand pattern and the multiple sample images of each hand pattern to obtain the target feature recognition model corresponding to each hand pattern, including:

[0032] Based on each layer of the time recurrent neural network, feature extraction is performed on each image to be processed of each of the hand patterns to obtain T first feature images; wherein the first feature extraction network is composed of T layers of time recurrent neural networks;

[0033] Based on each convolution layer in the second feature extraction network, feature extraction is performed on each sample image of each hand pattern to obtain T second feature images; wherein the second feature extraction network includes T convolution layers;

[0034] Using the fully connected layer in the initial feature recognition model to splice each of the first feature images and each of the second feature images that have a corresponding relationship, to obtain T spliced ​​feature images;

[0035] The first feature extraction network and the second feature extraction network are trained based on the T spliced ​​feature images to obtain a target feature recognition model corresponding to each hand pattern.

[0036] In the above scheme, the target feature recognition model performs sign language recognition on the target image information to obtain the target alarm information of the alarm object, including:

[0037] Determine a feature recognition model to be selected from a plurality of target feature recognition models based on the sign language recognition mode determination information;

[0038] Performing sign language recognition on the target image information based on the selected feature recognition model to obtain pre-selected alarm information of the alarm object;

[0039] Receive the target alarm information corrected by the alarm object with respect to the pre-selected alarm information; wherein the target alarm information is obtained after the pre-selected alarm information is corrected.

[0040] In the above scheme, the target image information is processed based on the target feature recognition model to obtain the alarm information of the alarm object, including:

[0041] When it is determined that the network bandwidth of the terminal meets the target network bandwidth and has an idle recognition module that carries the target feature recognition model, the target image information is recognized using the target feature recognition model carried by the idle recognition module to obtain the alarm information.

[0042] In the above scheme, the method further comprises:

[0043] In the case that the response information sent by the alarm object to the target page information is not received, the return visit information for returning the alarm object is generated and sent to the return visit information server.

[0044] In the above scheme, the method further comprises:

[0045] Based on the alarm request, obtaining a first position and a second position of the alarm object; wherein the first position is determined by a base station corresponding to the terminal, and the second position is determined by a positioning module set on the terminal;

[0046] Based on the first position and the second position, a target position of the alarm object is determined.

[0047] An information determination device, the device comprising:

[0048] A receiving unit, configured to receive an alarm request sent by an alarm object through a terminal, and obtain call information of the alarm object based on the alarm request;

[0049] A sending unit, configured to send target page address information corresponding to the alarm request to the terminal based on the call information;

[0050] A processing unit, configured to obtain target image information of the alarm object upon receiving response information sent by the alarm object with respect to the target page address information;

[0051] The determination unit is used to perform sign language recognition on the target image information based on a target feature recognition model to obtain target alarm information of the alarm object.

[0052] An information determination device, the device comprising: a processor, a memory, and a communication bus;

[0053] The communication bus is used to realize the communication connection between the processor and the memory;

[0054] The processor is used to execute the information determination program stored in the memory to implement the steps of the above-mentioned information determination method.

[0055] A computer-readable storage medium stores one or more programs, and the one or more programs can be executed by one or more processors to implement the steps of the above-mentioned information determination method.

[0056] The information determination method, apparatus, device and computer-readable storage medium provided in the embodiments of the present application first receive an alarm request sent by an alarm object through a terminal, and obtain the call information of the alarm object based on the alarm request, and send the target page address information corresponding to the alarm request to the terminal based on the call information, and then obtain the target image information of the alarm object upon receiving the response information sent by the alarm object for the target page address information, and then perform sign language recognition on the target image information based on the target feature recognition model to obtain the target alarm information of the alarm object. In this way, it is possible to directly determine whether the target page address information is to be sent to the alarm object through the call information, and the alarm object can directly make an alarm through the target page address information after receiving the target page address information, without installing an alarm help App, thereby reducing the complexity of the alarm process, making the alarm process more convenient and the alarm efficiency higher, and further, the target image information of the alarm object is recognized through the target feature recognition model to improve the accuracy of the alarm information. BRIEF DESCRIPTION OF THE DRAWINGS

[0057] Figure 1 A flowchart of an information determination method provided in an embodiment of the present application;

[0058] Figure 2 A flowchart of another information determination method provided in an embodiment of the present application;

[0059] Figure 3 A schematic diagram of an alarm process in an information determination method provided in an embodiment of the present application;

[0060] FIG4( a) is a schematic diagram of model training in an information determination method provided in an embodiment of the present application;

[0061] FIG4( b) is a schematic diagram of model training in an information determination method provided in an embodiment of the present application;

[0062] Figure 5 A flowchart of another information determination method provided in an embodiment of the present application;

[0063] Figure 6 A schematic diagram of obtaining target image information in an information determination method provided in an embodiment of the present application;

[0064] Figure 7 A schematic diagram of the structure of an information determination device provided in an embodiment of the present application;

[0065] Figure 8 A schematic diagram of the structure of an information determination device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0066] The technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application.

[0067] It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0068] The present application embodiment provides an information determination method, which can be applied to an information determination device, referring to Figure 1 As shown, the method comprises the following steps:

[0069] Step 101: Receive an alarm request sent by an alarm object through a terminal, and obtain call information of the alarm object based on the alarm request.

[0070] Among them, the terminal may refer to a terminal with multiple image acquisition modules and can browse web pages developed based on the fifth-generation HyperText Markup Language (HTML) technology; call information may refer to the communication information between the alarm object and the alarm center in the information determination device after sending an alarm request.

[0071] In an embodiment of the present application, the alarm object can send an alarm request by dialing an alarm phone through a terminal. For example, when there is an emergency, the alarm object can dial an alarm phone through a terminal such as a mobile phone or a landline phone, such as dialing 120 or 110 to report an emergency. The dialed alarm phone is connected to the nearest base station according to the alarm object's mobile phone or landline phone. The base station then sends the alarm request sent by the alarm object through the terminal to the information determination device. The information determination device receives the alarm request sent by the alarm object through the terminal and obtains the call information corresponding to the alarm phone. In a feasible implementation method, the base station may refer to an operator base station.

[0072] Step 102: Send the target page address information corresponding to the alarm request to the terminal based on the call information.

[0073] In an embodiment of the present application, the call content in the call information can be analyzed, and based on the analysis results, the target page address information corresponding to the alarm request can be sent to the terminal in a targeted manner. After that, the alarm object corresponding to the terminal can click on the target page address information to jump to the target page for subsequent alarm.

[0074] It should be noted that the alarm object can enter the alarm page by clicking on the target page address information. When entering the alarm page for the first time, it is necessary to verify the alarm mobile phone number, which can be automatically obtained through the operator, and the alarm mobile phone number can also be verified by SMS verification code.

[0075] In a feasible implementation method, the target method may refer to a short message method, the target method may refer to a WeChat message method, the target method may refer to a WeChat public account method, and the target method may also refer to a 5G message method; the target page address information may be in the form of a link and may refer to a Uniform Resource Locator (URL).

[0076] Step 103: upon receiving the response information sent by the alarm object with respect to the target page address information, the target image information of the alarm object is acquired.

[0077] In an embodiment of the present application, the response information refers to the reply information of the alarm object to the target page address information; the target image information may refer to the video information of the upper body of the alarm object, and is sent to the information determination device after being acquired by the terminal through the image acquisition module; in this way, since only the video information of the upper body of the alarm object is collected instead of the whole body video information, the amount of data transmitted by the video information is reduced and the speed of video information transmission is increased.

[0078] In an embodiment of the present application, it can be determined whether a response message to the target page address information sent by the alarm object is received within a preset time. If a response message to the target page address information sent by the alarm object is received within the preset time, the terminal can obtain the target image information of the alarm object in real time through the image acquisition module, and send the acquired target image information to the information determination device in real time; it should be noted that the image acquisition module can include at least a first image acquisition module and a second image acquisition module; in a feasible implementation method, the preset time can be 5 minutes.

[0079] Step 104: Perform sign language recognition on the target image information based on the target feature recognition model to obtain target alarm information of the alarm object.

[0080] In an embodiment of the present application, the target feature recognition model may refer to a model obtained by training an initial feature recognition model using a sample image, and the initial feature recognition model may include a first feature extraction network and a second feature extraction network; the target alarm information is obtained by processing the target image information using the target feature recognition model; the target image information can be input into the target feature recognition model for sign language recognition to obtain the alarm information of the alarm object.

[0081] The information determination method provided in the embodiment of the present application can directly determine whether to send the target page address information to the alarm object through call information, and the alarm object can directly alarm through the target page address information after receiving the target page address information, without the need to install an alarm help App, thereby reducing the complexity of the alarm process, making the alarm process more convenient and the alarm efficiency higher, and further, the target image information of the alarm object is identified through the target feature recognition model, thereby improving the accuracy of the alarm information.

[0082] Based on the above embodiments, the present application provides an information determination method, referring to Figure 2 and Figure 3 As shown, the method comprises the following steps:

[0083] Step 201: The information determination device receives an alarm request sent by an alarm object through a terminal, and obtains call information of the alarm object based on the alarm request.

[0084] Step 202: When the target content is not extracted from the call information of the target call duration, the information determination device sends the target page address information to the terminal.

[0085] In an embodiment of the present application, the target call duration may be determined based on historical experience; failure to extract the target content from the call information of the target call duration may include two situations: one is that no communication content is extracted; the other is that the extracted communication content is such as "ah ah ah" and belongs to the object of physical dysfunction. In these cases, the target page address information can be directly sent to the terminal; it should be noted that failure to extract the target content from the call information of the target call duration may indicate that the alarm object is a person with physical dysfunction or a normal user who can sign language is in a coerced scenario; in a feasible implementation method, the target call duration may be 10 seconds.

[0086] Step 203: Upon receiving the response information sent by the alarm object with respect to the target page address information, the information determination device sends selection information for determining whether the alarmer has a physical dysfunction to the terminal.

[0087] In the embodiment of the present application, the selection information can be used by the alarm object to determine whether it has a physical dysfunction; Figure 3 As shown, after the terminal receives the target page address information sent by the information determination device, the alarm object can click on the target page address information through the terminal. After clicking, the information determination device will send selection information for the target page address information to the terminal for the alarm object to choose, that is, the alarm object is asked to confirm his or her identity again (that is, confirm whether he or she has a physical dysfunction). In this way, the identity of the alarmer can be verified once again, thereby improving the accuracy of determining the identity of the alarm object.

[0088] Step 204: When receiving confirmation information from the alarm object regarding the selected information, the information determination device acquires the target image information of the alarm object.

[0089] Among them, the confirmation information may refer to the reply information of the alarm object to the selection information; the selection information may refer to the alarm person's selection of whether he or she has a physical dysfunction; the confirmation information may be "yes" or "no".

[0090] In the embodiments of this application, refer to Figure 3 As shown, when the confirmation information is "yes", the information determination device sends the login page of the object with physical dysfunction (i.e., the deaf-mute person) to the terminal, and at the same time, the information determination device obtains the target image information of the alarm object; it should be noted that the login page can be an accessible login page or a login page with relatively low verification complexity; when the confirmation information is "no", the alarm can be made through the alarm process of the object without physical dysfunction (i.e., the normal object), and the page of the text alarm can be switched to the bronze drum text to make an alarm.

[0091] It should be noted that step 204 can be implemented in the following ways:

[0092] Step 204a1: The information determination device receives the first image information of the alarm object sent by the terminal.

[0093] The first image information is collected by the terminal using a first image collection module.

[0094] In the embodiment of the present application, the first image acquisition module may refer to an ordinary camera; the information determination device may receive the first image information of the alarm object sent by the terminal in real time.

[0095] Step 204a2: When the content of the first image information meets the target image content, the information determination device determines that the first image information is the target image information.

[0096] The target image content may include the shape of the hand, the position of the hand, the movement trajectory of the hand, the orientation of the palm and the facial expression (or facial posture), etc.

[0097] In an embodiment of the present application, the content of the first image information can be analyzed to determine whether the content of the first image information includes the hand shape, hand position, hand movement trajectory, palm orientation and facial expression (or facial posture) of the alarm object. If the content of the first image information includes the hand shape, hand position, hand movement trajectory, palm orientation and facial expression (or facial posture) of the alarm object, the first image information is directly determined to be the target image information.

[0098] Step 204a3: When the content of the first image information does not satisfy the target image content, the information determination device obtains updated second image information of the alarm object collected by the terminal, and determines the target image information based on the second image information.

[0099] In an embodiment of the present application, the content of the first image information not satisfying the target image content may mean that the content of the first image information does not include the hand shape, hand position, hand movement trajectory, palm orientation and facial expression (or facial posture) of the alarm object, or only includes part of the above information; the second image information may refer to the updated image information of the alarm object collected by the terminal, and the second image information may be the updated image information of the alarm object collected by the terminal based on the adjustment instruction and the switching instruction; the acquired second image information can be compared with the target image content, and the target image information can be determined based on the comparison result.

[0100] Step 205: The information determination device collects multiple sample images of multiple hand patterns of sample objects in different environments.

[0101] The multiple sample images are images of hand gestures and torso images of the sample subjects during communication; the hand structure used in each hand mode is the same.

[0102] In an embodiment of the present application, different environments may include a car accident environment, an earthquake environment, etc.; sample objects may refer to objects of different heights, weights and genders; sample images with the same hand structure used in communication are determined as sample images corresponding to a hand mode (that is, sample images of left-hand communication are determined as the first hand mode, sample images of right-hand communication are determined as the second hand mode, and sample images of both hands communication are determined as the third hand mode); images of hand gestures and torso images of sample objects of different heights, weights and genders in environments such as car accidents and earthquakes are collected when only the left hand is used for communication, images of hand gestures and torso images when only the right hand is used for communication, and images of hand gestures and torso images when both hands are used for communication in a communication, and the images collected under the three hand modes are labeled, and the labeled hand gesture images and torso images are used as sample images; it should be noted that the label includes the Chinese content of the sign language expression and the description of the environment.

[0103] Step 206: The information determination device performs model training on the first feature extraction network and the second feature extraction network in the initial feature recognition model based on multiple sample images of each hand pattern to obtain a target feature recognition model corresponding to each hand pattern.

[0104] Among them, the first feature extraction network is a network that extracts features in time dimension and space dimension; the second feature extraction network is a convolutional neural network.

[0105] In an embodiment of the present application, the second feature extraction network may refer to a convolutional neural network that can extract the time dimension; for multiple sample images of each hand pattern, the multiple sample images may be respectively input into the first feature extraction network and the second feature extraction network for model training to obtain a target feature recognition model; specifically, the multiple sample images of the first hand pattern may be respectively input into the first feature extraction network and the second feature extraction network for model training to obtain a target feature recognition model corresponding to the first hand pattern, the multiple sample images of the second hand pattern may be respectively input into the first feature extraction network and the second feature extraction network for model training to obtain a target feature recognition model corresponding to the second hand pattern, the multiple sample images of the third hand pattern may be respectively input into the first feature extraction network and the second feature extraction network for model training to obtain a target feature recognition model corresponding to the second hand pattern 4(a) is used to perform model training on the first feature extraction network and the second feature extraction network in the initial feature recognition model; and the first feature extraction network and the second feature extraction network in the initial feature recognition model are trained by using the images of hand postures of right hand communication and the trunk images of body trunk. The first feature extraction network and the second feature extraction network in the initial feature recognition model are trained by using the images of hand postures of right hand communication and the trunk images of body trunk. The second hand pattern and the plurality of sample images may be images of hand postures of left hand communication and the trunk images of body trunk. The first feature extraction network and the second feature extraction network in the initial feature recognition model are trained by using the images of hand postures of left hand communication and the trunk images of body trunk.

[0106] It should be noted that step 206 can be implemented in the following ways:

[0107] Step 206b1: The information determination device identifies each sample image of each hand pattern based on a target recognition algorithm for multiple sample images of each hand pattern, and determines the target area of ​​the key part of the alarm object in each sample image.

[0108] In an embodiment of the present application, a target recognition algorithm can be used to identify key parts of an alarm object in a sample image; a target area refers to the area in the sample image where the key parts of an alarm object are located; each sample image may contain multiple key parts of an alarm object; key parts of the human body are divided into three parts, namely, key parts of the left hand, key parts of the right hand and key parts of the human torso, among which 24 key parts of the human body are identified, including the five senses; 21 key parts of the left hand (or right hand) are identified according to the finger joints; in a feasible implementation method, the target recognition algorithm may refer to a human posture recognition algorithm (OpenPose).

[0109] Step 206b2: The information determination device extracts the image of the target area in each sample image of each hand pattern to obtain a plurality of to-be-processed images of each hand pattern.

[0110] In the embodiment of the present application, the image to be processed refers to the image of the area corresponding to the key part of the alarm object in the sample image; for each hand pattern, the target recognition algorithm can extract the image of the target area in each sample image of each hand pattern to obtain multiple images to be processed; in a feasible implementation method, refer to Figure 4(a) and 4(b) As shown, each sample image is processed by OpenPose to obtain multiple images to be processed.

[0111] Step 206b3: The information determination device performs model training on the first feature extraction network and the second feature extraction network based on multiple to-be-processed images of each hand pattern and multiple sample images of each hand pattern to obtain a target feature recognition model corresponding to each hand pattern.

[0112] In an embodiment of the present application, for each hand pattern, each image to be processed can be input into the first feature extraction network for model training, and each sample image can be input into the second feature extraction network for model training to obtain a target feature recognition model.

[0113] Step 207: Perform sign language recognition on the target image information based on the target feature recognition model to obtain target alarm information of the alarm object.

[0114] In an embodiment of the present application, after obtaining the target alarm information, the target alarm information can also be synchronously transmitted to the operator (i.e., the alarm seat). The operator receives and summarizes the alarm information, and then manually makes a ticket, and the alarm ends.

[0115] Step 208: When no response information regarding the target page address information is received from the alarm object, the information determination device generates return visit information for returning the alarm object, and sends the return visit information to the server.

[0116] In an embodiment of the present application, it can be determined whether a response message sent by the alarm object for the target page address information is received within a preset time. If a response message sent by the alarm object for the target page address information is not received within the preset time, a follow-up message for a follow-up visit to the alarm object is generated to remind the manual customer service on the server side to make a follow-up call to the alarm object.

[0117] It should be noted that the above embodiment may further include the following steps:

[0118] Step 209: The information determination device obtains the first position and the second position of the alarm object based on the alarm request.

[0119] The first position is determined by a base station corresponding to the terminal, and the second position is determined by a positioning module set on the terminal.

[0120] In the embodiment of the present application, the first position and the second position both refer to geographic locations; the information determination device can obtain the first position of the alarm object based on the alarm telephone number of the alarm object obtained previously, and then determine the location of the alarm telephone number through a base station. At the same time, the terminal can send the second position determined by its own positioning module to the information determination device in real time; it should be noted that the position of the alarm object is the position of the terminal.

[0121] In a feasible implementation manner, the base station may refer to an operator base station; and the positioning module may refer to a global satellite positioning system (GPS).

[0122] Step 210: The information determination device determines the target location of the alarm object based on the first location and the second location.

[0123] In an embodiment of the present application, the target position is the position of the alarm object determined based on the first position and the second position; the information determination device determines the position of the alarm object by combining the first position determined by its own positioning module and the second position reported by the terminal, thereby improving the accuracy of determining the target position.

[0124] The information determination method provided in the embodiment of the present application can directly determine whether to send the target page address information to the alarm object through call information, and the alarm object can directly alarm through the target page address information after receiving the target page address information, without the need to install an alarm help App, thereby reducing the complexity of the alarm process, making the alarm process more convenient and the alarm efficiency higher, and further, the target image information of the alarm object is identified through the target feature recognition model, thereby improving the accuracy of the alarm information.

[0125] Based on the above embodiments, the present application provides an information determination method, referring to Figure 5 As shown, the method comprises the following steps:

[0126] Step 301: The information determination device receives an alarm request sent by an alarm object through a terminal, and obtains call information of the alarm object based on the alarm request.

[0127] Step 302: When the target content is not extracted from the call information of the target call duration, the information determination device sends the target page address information to the terminal.

[0128] Step 303: Upon receiving the response information sent by the alarm object with respect to the target page address information, the information determination device sends selection information for determining whether the alarmer has a physical dysfunction to the terminal.

[0129] Step 304: When receiving confirmation information from the alarm object regarding the selected information, the information determining device receives the first image information of the alarm object sent by the terminal.

[0130] The first image information is collected by the terminal using a first image collection module.

[0131] Step 305: When the content of the first image information meets the target image content, the information determination device determines that the first image information is the target image information.

[0132] Step 306: When the content of the first image information does not satisfy the target image content, the information determination device sends an adjustment instruction for adjusting the posture and / or position of the alarm object to the terminal.

[0133] The adjustment instruction may be used to instruct the alarm object to adjust its own posture and / or position.

[0134] In an embodiment of the present application, a half-body image frame can be displayed on the terminal, and then an adjustment instruction can be used to indicate the alarm object to adjust its own position and posture in the half-body image frame so that the alarm object is completely framed in the half-body image frame, so that the image acquisition module in the terminal can see the half-body image of the alarm object, and then the image acquisition module can capture the half-body image of the alarm object in the half-body image frame (including the shape of the hand, the position of the hand, the movement trajectory of the hand, the orientation of the palm and the facial expression (or facial posture)); it should be noted that an adjustment duration can also be set.

[0135] In a feasible implementation, the adjustment duration may be set to 30 seconds.

[0136] Step 307: When the first sub-image information of the alarm object is received from the terminal within the target duration and the content of the first sub-image information meets the target image content, the information determination device determines that the first sub-image information is the target image information.

[0137] The second image information includes the first sub-image information.

[0138] In an embodiment of the present application, the target duration may refer to the adjustment duration of adjusting the posture of the alarm object based on the adjustment instruction; the first sub-image information may refer to the image captured after the terminal adjusts the posture of the alarm object based on the adjustment instruction; the content of the received first sub-image information of the alarm object can be compared with the target image content, and when the content of the first sub-image information meets the target image content, the first sub-image information is directly determined to be the target image information; in a feasible implementation method, the target duration may be 30 seconds.

[0139] Step 308: If the first sub-image information is not received within the target time period, or the content of the first sub-image information does not meet the target image content, the information determination device sends a switching instruction for switching the image acquisition module to the terminal.

[0140] In the embodiment of the present application, the switching instruction is used to instruct the terminal to switch the image acquisition module, that is, it can switch from one image acquisition module to another image acquisition module.

[0141] Step 309: The information determination device receives the second sub-image information of the alarm object collected by the second image collection module and sent by the terminal, and determines the second sub-image information as the target image information.

[0142] The second image acquisition module is the image acquisition module switched by the terminal based on the switching instruction; the second image information includes the second sub-image information.

[0143] In an embodiment of the present application, the capture viewing angle of the second image acquisition module is greater than that of the first image acquisition module; the second sub-image information is image information captured by the terminal through the second image acquisition module; when the first sub-image information captured after adjusting the posture of the alarm object based on the adjustment instruction still does not meet the target image content, the second sub-image information captured by the second image acquisition module after the image acquisition module is switched is obtained, and the second sub-image information is determined to be the target image information; in this way, the viewing angle effect can be improved to capture more and more effective image information of the alarm object.

[0144] In a feasible implementation, when the first image acquisition module is an ordinary camera, the second image acquisition module may be a wide-angle camera; when the first image acquisition module is a wide-angle camera with a first viewing angle, the second image acquisition module may be a wide-angle camera with a second viewing angle, and the second viewing angle is greater than the first viewing angle.

[0145] Step 310: The information determination device collects multiple sample images of multiple hand patterns of sample objects in different environments.

[0146] The multiple sample images are images of hand gestures and torso images of the sample objects during communication; the hand structure used in each hand mode is the same.

[0147] Step 311: The information determination device identifies each sample image of each hand pattern based on a target recognition algorithm for multiple sample images of each hand pattern, and determines a target area of ​​a key part of the alarm object in each sample image.

[0148] It should be noted that there are two abnormal situations in the key part identification in the relevant technology. In the first situation, the entire key part of the hand is missing in the video information, such as the posture of the left arm and the left hand key parts are not detected, which is mainly due to the injury of the left and right hands in the accident or the left and right hands are outside the picture; In the second situation, some key parts are missing in the video information due to obstruction by body clothing or other foreign objects or local injuries, for example, there is an immovable obstacle in front of the body in the accident or only the hand is injured in the accident without affecting the arm movement.

[0149] In the embodiment of the present application, for situation 1, if the target recognition algorithm fails to recognize the key parts of the left hand and left arm and the key parts of the right hand and right arm, it can automatically switch to the corresponding single-hand recognition model, select the single-hand recognition model for targeted recognition, and improve the recognition accuracy. For situation 2, a key point repair mechanism is added (i.e., using T-layer LSTM to collect spatial dimension features) to supplement the missing key part information, thereby improving the recognition accuracy.

[0150] Step 312: The information determination device extracts the image of the target area in each sample image of each hand pattern to obtain a plurality of to-be-processed images of each hand pattern.

[0151] Step 313: The information determination device extracts features from each image to be processed based on the time recurrent neural network of each layer to obtain T first feature images.

[0152] Among them, the first feature extraction network is composed of a T-layer time recurrent neural network.

[0153] In an embodiment of the present application, the first feature image is obtained by performing feature extraction on the image to be processed through a T-layer time recurrent neural network (Long Short-Term Memory, LSTM); for each image to be processed, the image to be processed can be input into a T-layer time recurrent neural network, and each layer of the time recurrent neural network will output a first feature image, thus obtaining T first feature images.

[0154] It should be noted that the first feature extraction network is composed of T layers of LSTM, which can perform feature transmission not only in the time dimension, but also in the space dimension, that is, bidirectional LSTM is used for transmission in the time dimension, which can combine the semantic coherence of the previous and the next, and avoid the situation where the extracted features are too few and cannot be recognized due to incomplete hand gesture display, thereby improving the recognition accuracy.

[0155] In a feasible implementation, as shown in Figure 4, T layers of LSTM are used to extract features from the image to be processed, wherein the first feature image output by the first layer of LSTM can be recorded as R1 and B1, the first feature image output by the second layer of LSTM can be recorded as R2 and B2, and the first feature image output by the Tth layer of LSTM can be recorded as Rt and Bt; it should be noted that R is the feature obtained from the image extracted from the key parts of the right hand, and B is the feature obtained from the image extracted from the key parts of the torso.

[0156] Step 314: The information determination device performs feature extraction on each sample image based on each convolutional layer in the second feature extraction network to obtain T second feature images.

[0157] Wherein, the second feature extraction network includes T convolutional layers;

[0158] In an embodiment of the present application, the second feature image is obtained by performing feature extraction on the sample image in each convolution layer; for each sample image, the sample image can be input into the second feature extraction network, and the feature extraction of the sample image is performed through the convolution layer in the second feature extraction network, and each convolution layer outputs a result (i.e., the second feature image), and T second feature images are obtained.

[0159] In a feasible implementation, the second feature extraction network may refer to a Visual Geometry Group (VGG) network. The sample image may be input into the VGG network, and the first 16 layers (i.e., 16 convolutional layers) of VGG-19 may be used to perform feature extraction on the sample image. As shown in FIG4 , the second feature image output by the first convolutional layer is denoted as F1, the second feature image output by the second convolutional layer is denoted as F2, and the second feature image output by the 16th convolutional layer is denoted as F16.

[0160] Step 315: The information determination device uses the fully connected layer in the initial feature recognition model to splice each first feature image and each second feature image that have a corresponding relationship to obtain T spliced ​​feature images.

[0161] Among them, the fully connected layer (FC) can be used to splice the first feature image and the second feature image that have a corresponding relationship to obtain a spliced ​​feature image.

[0162] In the embodiment of the present application, the first feature image and the second feature image having a corresponding relationship refer to the case where the first feature image is the feature output by the first layer of the temporal recurrent neural network, and the second image feature having a corresponding relationship with the first feature image refers to the image feature output by the first convolutional layer performing feature extraction on the sample image.

[0163] In a feasible implementation, as shown in FIG. 4 , R1 and B1 may be spliced ​​with F1, R2 and B2 may be spliced ​​with F2, and Rt and Bt may be spliced ​​with Ft, thereby obtaining T spliced ​​feature images.

[0164] Step 316: The information determination device trains the first feature extraction network and the second feature extraction network based on the T spliced ​​feature images to obtain a target feature recognition model corresponding to each hand pattern.

[0165] In a feasible implementation, as shown in Figure 4, the obtained T spliced ​​feature images can be input into a random dropout layer. The function of the Dropout layer is to avoid overfitting, and its dropout rate can be set to 0.5; then, the features output by the Dropout layer are input into the loss function (Softmax) layer, which can assign a probability value to each output; finally, the entire initial feature recognition model is trained using a cross entropy loss function and an optimizer (Adam) to obtain a target feature recognition model; it should be noted that in order to process the nonlinearity of each extracted first feature image and second feature image, an activation function (ReLU) can be used in the LSTM of adjacent layers and the convolutional layer of adjacent layers.

[0166] Step 317: The information determination device determines a sub-target feature recognition model from multiple target feature recognition models based on the selection information input by the user.

[0167] In an embodiment of the present application, the selection information may refer to information used by the user to select a target feature recognition model under different hand modes; the multiple target feature recognition models include a target feature recognition model corresponding to the first hand mode, a target feature recognition model corresponding to the second hand mode, and a target feature recognition model corresponding to the third hand mode; the sub-target feature recognition model may refer to a recognition model selected from multiple target feature recognition models based on the selection information; the information determination device may analyze the user's selection information, and based on the analysis results, select a sub-target feature recognition model from the target feature recognition model corresponding to the first hand mode, the target feature recognition model corresponding to the second hand mode, and the target feature recognition model corresponding to the third hand mode, and use the sub-target feature recognition model for sign language recognition.

[0168] It should be noted that sign language generally requires images of both hands + face to accurately recognize the meaning of the sign language. In the alarm scenario, the caller may generally hold the mobile phone with one hand or may be unable to use one hand due to injury. For this special scenario, the information determination device can set target feature recognition models corresponding to three hand modes, one is a two-hand sign language recognition mode, one is a left-hand sign language recognition mode, and the other is a right-hand sign language recognition mode. The accuracy of the single-hand sign language recognition mode (left-hand sign language recognition mode and right-hand sign language recognition mode) may be lower than that of the two-hand sign language recognition mode, so the information determination device can push selection information to the terminal, allowing the alarm object to choose to use the left-hand sign language recognition mode, the right-hand sign language recognition mode, or the two-hand sign language recognition mode, and then use the corresponding recognition mode for sign language recognition.

[0169] Step 318: The information determination device performs sign language recognition on the target image information based on the sub-target feature recognition model to obtain pre-selected alarm information of the alarm object.

[0170] In an embodiment of the present application, the pre-selected alarm information may refer to the recognition result obtained by directly using the sub-target feature recognition model to perform sign language recognition on the target image information; the target image information may be input into the sub-target feature recognition model for sign language recognition to obtain the pre-selected alarm information of the alarm object; in a feasible implementation method, the three sign language translation words with the highest probability in the recognition result may be selected as the pre-selected alarm information.

[0171] It should be noted that step 318 can be implemented in the following ways:

[0172] Step 318a1: When it is determined that the network bandwidth of the terminal meets the target network bandwidth and has an idle recognition module that carries a sub-target feature recognition model, the information determination device uses the sub-target feature recognition model carried by the idle recognition module to recognize the target image information and obtain alarm information.

[0173] The target network bandwidth may be determined based on historical experimental data.

[0174] In the embodiments of this application, refer to Figure 3 As shown, before the terminal collects the target image information, it is necessary to judge whether the network bandwidth of the terminal and the recognition module carrying the target feature recognition model are idle. When the network bandwidth of the terminal meets the target network bandwidth and there is an idle recognition module carrying the target feature recognition model, the sub-target feature recognition model carried by the idle recognition module is used to recognize the target image information to obtain the pre-selected alarm information; in a feasible implementation method, the network bandwidth meeting the target network bandwidth can refer to that the terminal's network can meet the bandwidth of the video.

[0175] In other embodiments of the present application, when the network bandwidth of the terminal does not meet the target network bandwidth, or there is no idle recognition module carrying the target feature recognition model, the willingness of the alarm object can be determined, and when the alarm object is willing to wait, the network bandwidth of the terminal and the recognition module carrying the sub-target feature recognition model can continue to be monitored to see if they are idle, until the network bandwidth meets the target network bandwidth and there is an idle recognition module carrying the sub-target feature recognition model, the sub-target feature recognition model carried by the idle recognition module is used to identify the target image information to obtain the alarm information; when the alarm object is unwilling to wait, refer to Figure 3 As shown, a text alarm page can be sent to the terminal so that the alarm object can alarm in text form through the text alarm page in the terminal, thereby obtaining target alarm information.

[0176] Step 319: The information determination device receives the correction information of the alarm object for the pre-selected alarm information, and determines the target alarm information based on the correction information.

[0177] In an embodiment of the present application, the pre-selected alarm information can be pushed to the client for confirmation by the alarm object. If the alarm object believes that there is an error between the pre-selected alarm information and the meaning it wants to express, the alarm object can manually correct the pre-selected alarm information, and then generate the target alarm information based on the corrected alarm information; it should be noted that the content of the modification is generally small, so as to ensure the timeliness of the alarm and the accuracy of sign language recognition as much as possible.

[0178] In a feasible implementation method, it can be pushed after several hand movements are completed, or pushed to the client for confirmation by the reporting object after all hand movements are completed. In this way, for the one-handed sign language recognition mode, the information determination device (alarm platform) can iteratively optimize the one-handed sign language recognition model according to the correct sign language translation words selected by the alarm object and the corrected sign language translation words. Through the cyclic model iterative update and optimization, the recognition accuracy of the one-handed sign language recognition model can be further improved, and the recognition effect can be improved. Corresponding processing has been carried out for the one-handed sign language needs in the alarm scenario, which can be more suitable for sign language recognition in the alarm scenario of the deaf and mute, with better applicability and better sign language recognition effect.

[0179] Step 320: If no response information regarding the target page address information sent by the alarm object is received, generate return visit information for returning the alarm object.

[0180] In other embodiments of the present application, refer to Figure 6As shown, the alarm object needs to start the terminal before sending an alarm request, and then can send the alarm request by dialing the alarm phone; it should be noted that the alarm object can set the interaction method and the priority of the interaction method with the information determination device in the privacy of the terminal in advance. The interaction method may include voice, text and gesture, and the priority order of the interaction method can be text, gesture and voice. Such a setting will take care of the alarm objects who do not like voice and are willing and can communicate in text. Afterwards, the information determination device can extract voice information (i.e., target content) from the call information corresponding to the alarm phone, and if no voice information is extracted, send selection information for the target page address information to the terminal in text / voice / gesture mode. If the selection information for the target page address information is sent to the terminal in text mode, the alarm object can click on the selection information displayed in text form in the terminal. If the alarm object determines that he or she is deaf-mute (i.e., confirmation information), the terminal enters the target image information acquisition mode. If the content of the first image information meets the target image content (i.e., the bust of the alarm object can be seen), the information determination device directly sends the first image information to the target feature recognition model carried by the idle recognition module after receiving the first image information; if the content of the first image information does not meet the target image content (i.e., the bust of the alarm object cannot be seen), the alarm object is instructed to adjust its posture based on the adjustment instruction to obtain the first sub-image information. image information; when the first sub-image information still does not meet the target image content, the terminal is instructed to switch the image acquisition module based on the switching instruction to obtain the second sub-image information. When the second sub-image information meets the target image content, the information determination device directly sends the first image information to the target feature recognition model carried by the idle recognition module after receiving the first image information; when the second sub-image information still does not meet the target image content, a prompt message can be displayed on the terminal page to prompt that the bust of the alarm object cannot be captured at present, and prompt the alarm object to adjust the distance from itself to the top, bottom, left, right, front and back of the terminal (the prompt information can be in text or graphic form); after the prompt is completed, wait for the alarm object to adjust. Within the timeout period, if the bust of the alarm object can be seen, the information determination device directly sends it to the target feature recognition model carried by the idle recognition module for recognition after receiving the bust of the alarm object; otherwise, the conversation between the alarm object and the alarm operator is connected, and human intervention is required for communication.

[0181] The information determination method provided in the embodiment of the present application can directly determine whether to send the target page address information to the alarm object through call information, and the alarm object can directly alarm through the target page address information after receiving the target page address information, without the need to install an alarm help App, thereby reducing the complexity of the alarm process, making the alarm process more convenient and the alarm efficiency higher, and further, the target image information of the alarm object is identified through the target feature recognition model, thereby improving the accuracy of the alarm information.

[0182] Based on the above embodiments, the embodiments of the present application provide an information determination device, which can be applied to Figure 1 , 2 In the information determination method provided in the embodiment corresponding to 5, refer to Figure 7 As shown, the information determination device 4 may include: a receiving unit 41, a sending unit 42, a processing unit 43 and a determination unit 44, wherein:

[0183] The receiving unit 41 is used to receive an alarm request sent by the alarm object through the terminal, and obtain the call information of the alarm object based on the alarm request;

[0184] The sending unit 42 is used to send the target page address information corresponding to the alarm request to the terminal based on the call information;

[0185] The processing unit 43 is used to obtain the target image information of the alarm object when receiving the response information sent by the alarm object for the target page address information;

[0186] The determination unit 44 is used to perform language recognition on the target image information based on the target feature recognition model to obtain target alarm information of the alarm object.

[0187] In the embodiment of the present application, the sending unit 42 is further configured to perform the following steps:

[0188] When the target content is not extracted from the call information of the target call duration, the target page address information is sent to the terminal.

[0189] In the embodiment of the present application, the processing unit 43 is further configured to perform the following steps:

[0190] Sending selection information for determining whether the alarmer has a physical dysfunction to the terminal;

[0191] When confirmation information of the alarm target in response to the selection information is received, target image information of the alarm target is acquired.

[0192] In the embodiment of the present application, the processing unit 43 is further configured to perform the following steps:

[0193] Receiving first image information of an alarm object sent by a terminal; wherein the first image information is collected by the terminal using a first image collection module;

[0194] In a case where the content of the first image information satisfies the target image content, determining the first image information as the target image information;

[0195] In the case that the content of the first image information does not satisfy the target image content, updated second image information of the alarm object collected by the terminal is obtained, and the target image information is determined based on the second image information.

[0196] In the embodiment of the present application, the processing unit 43 is further configured to perform the following steps:

[0197] Sending an adjustment instruction for adjusting the posture and / or position of the alarm object to the terminal;

[0198] When the first sub-image information of the alarm object sent by the terminal is received within the target time period and the content of the first sub-image information meets the target image content, the first sub-image information is determined to be the target image information; wherein the second image information includes the first sub-image information;

[0199] If the first sub-image information is not received within the target time period, or the content of the first sub-image information does not meet the target image content, sending a switching instruction for switching the image acquisition module to the terminal;

[0200] The receiving terminal sends the second sub-image information of the alarm object collected by the second image acquisition module, and determines the second sub-image information as the target image information; wherein the second image acquisition module is the image acquisition module switched by the terminal based on the switching instruction; the second image information includes the second sub-image information.

[0201] In the embodiment of the present application, the determining unit 44 is further configured to perform the following steps:

[0202] Collect multiple sample images of hands and torsos of multiple hand modes of sample subjects in different environments; wherein the multiple sample images are images of hand gestures and torso images of the sample subjects in communication; the hand structure used in each hand mode is the same;

[0203] Based on multiple sample images of each hand pattern, the first feature extraction network and the second feature extraction network in the initial feature recognition model are trained to obtain a target feature recognition model corresponding to each hand pattern; wherein the first feature extraction network is a network capable of extracting features in the time dimension and the space dimension; and the second feature extraction network is a convolutional neural network.

[0204] In the embodiment of the present application, the determining unit 44 is further configured to perform the following steps:

[0205] For multiple sample images of each hand pattern, each sample image of each hand pattern is identified based on a target recognition algorithm, and a target area of ​​a key part of the alarm object in each sample image is determined;

[0206] Extracting an image of a target area in each sample image of each hand pattern to obtain a plurality of images to be processed of each hand pattern;

[0207] Based on a plurality of to-be-processed images of each hand pattern and a plurality of sample images of each hand pattern, model training is performed on the first feature extraction network and the second feature extraction network to obtain a target feature recognition model corresponding to each hand pattern.

[0208] In the embodiment of the present application, the determining unit 44 is further configured to perform the following steps:

[0209] Based on each layer of the time recurrent neural network, feature extraction is performed on each image to be processed to obtain T first feature images; wherein the first feature extraction network is composed of T layers of time recurrent neural networks;

[0210] Based on each convolution layer in the second feature extraction network, feature extraction is performed on each sample image to obtain T second feature images; wherein the second feature extraction network includes T convolution layers;

[0211] Using the fully connected layer in the initial feature recognition model to splice each first feature image and each second feature image having a corresponding relationship, to obtain T spliced ​​feature images;

[0212] The first feature extraction network and the second feature extraction network are trained based on T spliced ​​feature images to obtain a target feature recognition model corresponding to each hand pattern.

[0213] In the embodiment of the present application, the determining unit 44 is further configured to perform the following steps:

[0214] Determining a sub-target feature recognition model from a plurality of target feature recognition models based on selection information input by a user;

[0215] Based on the sub-target feature recognition model, the target image information is recognized by sign language to obtain the pre-selected alarm information of the alarm object;

[0216] Receive correction information of the alarm object for the pre-selected alarm information, and determine the target alarm information based on the correction information.

[0217] In the embodiment of the present application, the determining unit 44 is further configured to perform the following steps:

[0218] When it is determined that the network bandwidth of the terminal meets the target network bandwidth and has an idle recognition module that carries a sub-target feature recognition model, the sub-target feature recognition model carried by the idle recognition module is used to recognize the target image information to obtain target alarm information.

[0219] In the embodiment of the present application, the processing unit 43 is further configured to perform the following steps:

[0220] In the case that the response information for the target page address information sent by the alarm object is not received, the return visit information for returning the alarm object is generated, and the return visit information is sent to the terminal.

[0221] In the embodiment of the present application, the determining unit 44 is further configured to perform the following steps:

[0222] Based on the alarm request, obtaining a first position and a second position of the alarm object; wherein the first position is determined by a base station corresponding to the terminal, and the second position is determined by a positioning module set on the terminal;

[0223] Based on the first position and the second position, a target position of the alarm object is determined.

[0224] It should be noted that the specific implementation process of the steps performed by each unit in this embodiment can be referred to Figure 1 , 2 The implementation process of the information determination method provided in the embodiment corresponding to 5 will not be repeated here.

[0225] The information determination device provided in the embodiment of the present application can directly determine whether to send the target page address information to the alarm object through call information, and the alarm object can directly alarm through the target page address information after receiving the target page address information, without the need to install an alarm help App, thereby reducing the complexity of the alarm process, making the alarm process more convenient and the alarm efficiency higher, and further, the target image information of the alarm object is identified through the target feature recognition model, thereby improving the accuracy of the alarm information.

[0226] Based on the above embodiments, the embodiments of the present application provide an information determination device, which can be applied to Figure 1 , 2 The information determination method provided in the embodiment corresponding to 5 is referred to Figure 8 As shown, the information determination device 5 may include a processor 51, a memory 52 and a communication bus 53;

[0227] The communication bus 53 is used to realize the communication connection between the processor 51 and the memory 52;

[0228] The processor 51 is used to execute the information determination program stored in the memory 52 to implement the following steps:

[0229] Receive an alarm request sent by an alarm object through a terminal, and obtain call information of the alarm object based on the alarm request;

[0230] Based on the call information, the target page address information corresponding to the alarm request is sent to the terminal;

[0231] When receiving the response information sent by the alarm object for the target page address information, obtaining the target image information of the alarm object;

[0232] Based on the target feature recognition model, sign language recognition is performed on the target image information to obtain the target alarm information of the alarm object.

[0233] In other embodiments of the present application, the processor 51 is used to execute the memory 52 to send the target page address information corresponding to the alarm request to the terminal based on the call information, so as to implement the following steps:

[0234] When the target content is not extracted from the call information of the target call duration, the target page address information is sent to the terminal.

[0235] In other embodiments of the present application, the processor 51 is used to execute the acquisition of target image information of the alarm object in the memory 52 to implement the following steps:

[0236] Sending selection information for determining whether the alarmer has a physical dysfunction to the terminal;

[0237] When confirmation information of the alarm target in response to the selection information is received, target image information of the alarm target is acquired.

[0238] In other embodiments of the present application, the processor 51 is used to execute the steps in the memory 52 to obtain the target image information of the alarm object when receiving the confirmation information of the alarm object for the selection information, so as to implement the following steps:

[0239] Receiving first image information of an alarm object sent by a terminal; wherein the first image information is collected by the terminal using a first image collection module;

[0240] In a case where the content of the first image information satisfies the target image content, determining the first image information as the target image information;

[0241] In the case that the content of the first image information does not satisfy the target image content, updated second image information of the alarm object collected by the terminal is obtained, and the target image information is determined based on the second image information.

[0242] In other embodiments of the present application, the processor 51 is used to execute the updated second image information of the alarm object collected by the terminal in the memory 52, and determine the target image information based on the second image information, so as to achieve the following manner:

[0243] Sending an adjustment instruction for adjusting the posture and / or position of the alarm object to the terminal;

[0244] When the first sub-image information of the alarm object sent by the terminal is received within the target time period and the content of the first sub-image information meets the target image content, the first sub-image information is determined to be the target image information; wherein the second image information includes the first sub-image information;

[0245] If the first sub-image information is not received within the target time period, or the content of the first sub-image information does not meet the target image content, sending a switching instruction for switching the image acquisition module to the terminal;

[0246] The receiving terminal sends the second sub-image information of the alarm object collected by the second image acquisition module, and determines the second sub-image information as the target image information; wherein the second image acquisition module is the image acquisition module switched by the terminal based on the switching instruction; the second image information includes the second sub-image information.

[0247] In other embodiments of the present application, the processor 51 is used to execute the information determination program stored in the memory 52 to implement the following steps:

[0248] Collect multiple sample images of hands and torsos of multiple hand modes of sample subjects in different environments; wherein the multiple sample images are images of hand gestures and torso images of the sample subjects in communication; the hand structure used in each hand mode is the same;

[0249] Based on multiple sample images of each hand pattern, the first feature extraction network and the second feature extraction network in the initial feature recognition model are trained to obtain a target feature recognition model corresponding to each hand pattern; wherein the first feature extraction network is a network capable of extracting features in the time dimension and the space dimension; and the second feature extraction network is a convolutional neural network.

[0250] In other embodiments of the present application, the processor 51 is used to execute the model training of the first feature extraction network and the second feature extraction network in the initial feature recognition model based on multiple sample images of each hand pattern in the memory 52 to obtain the target feature recognition model corresponding to each hand pattern, so as to implement the following steps:

[0251] For multiple sample images of each hand pattern, each sample image of each hand pattern is identified based on a target recognition algorithm, and a target area of ​​a key part of the alarm object in each sample image is determined;

[0252] Extracting an image of a target area in each sample image of each hand pattern to obtain a plurality of images to be processed of each hand pattern;

[0253] Based on a plurality of to-be-processed images of each hand pattern and a plurality of sample images of each hand pattern, model training is performed on the first feature extraction network and the second feature extraction network to obtain a target feature recognition model corresponding to each hand pattern.

[0254] In other embodiments of the present application, the processor 51 is used to execute the model training of the first feature extraction network and the second feature extraction network based on the multiple images to be processed of each hand pattern and the multiple sample images of each hand pattern in the memory 52 to obtain the target feature recognition model corresponding to each hand pattern, so as to implement the following steps:

[0255] Based on each layer of the time recurrent neural network, feature extraction is performed on each image to be processed to obtain T first feature images; wherein the first feature extraction network is composed of T layers of time recurrent neural networks;

[0256] Based on each convolution layer in the second feature extraction network, feature extraction is performed on each sample image to obtain T second feature images; wherein the second feature extraction network includes T convolution layers;

[0257] Using the fully connected layer in the initial feature recognition model to splice each first feature image and each second feature image having a corresponding relationship, to obtain T spliced ​​feature images;

[0258] The first feature extraction network and the second feature extraction network are trained based on T spliced ​​feature images to obtain a target feature recognition model corresponding to each hand pattern.

[0259] In other embodiments of the present application, the processor 51 is used to execute the target feature recognition model in the memory 52 to perform sign language recognition on the target image information to obtain the target alarm information of the alarm object, so as to implement the following steps:

[0260] Determining a sub-target feature recognition model from a plurality of target feature recognition models based on selection information input by a user;

[0261] Based on the sub-target feature recognition model, the target image information is recognized by sign language to obtain the pre-selected alarm information of the alarm object;

[0262] Receive correction information of the alarm object for the pre-selected alarm information, and determine the target alarm information based on the correction information.

[0263] In other embodiments of the present application, the processor 51 is used to execute the sub-target feature recognition model in the memory 52 to perform sign language recognition on the target image information to obtain the pre-selected alarm information of the alarm object, so as to implement the following steps:

[0264] When it is determined that the network bandwidth of the terminal meets the target network bandwidth and has an idle recognition module that carries a sub-target feature recognition model, the sub-target feature recognition model carried by the idle recognition module is used to recognize the target image information to obtain alarm information.

[0265] In other embodiments of the present application, the processor 51 is used to execute the information determination program stored in the memory 52 to implement the following steps:

[0266] In the case that the response information for the target page address information sent by the alarm object is not received, the return visit information for returning the alarm object is generated and sent to the server.

[0267] In other embodiments of the present application, the processor 51 is used to execute the information determination program stored in the memory 52 to implement the following steps:

[0268] Based on the alarm request, obtaining a first position and a second position of the alarm object; wherein the first position is determined by a base station corresponding to the terminal, and the second position is determined by a positioning module set on the terminal;

[0269] Based on the first position and the second position, a target position of the alarm object is determined.

[0270] It should be noted that the specific implementation process of the steps executed by the processor in this embodiment can refer to Figure 1 , 2 The implementation process of the information determination method provided in the embodiment corresponding to 5 will not be repeated here.

[0271] The information determination device provided in the embodiment of the present application can directly determine whether to send the target page address information to the alarm object through call information, and the alarm object can directly alarm through the target page address information after receiving the target page address information, without the need to install an alarm help App, thereby reducing the complexity of the alarm process, making the alarm process more convenient and the alarm efficiency higher, and further, the target image information of the alarm object is identified through the target feature recognition model, thereby improving the accuracy of the alarm information.

[0272] Based on the foregoing embodiments, the embodiments of the present application provide a computer-readable storage medium, which stores one or more programs, and the one or more programs can be executed by one or more processors to implement Figure 1 , 2 and 5 corresponding to the steps in the information determination method provided in the embodiment.

[0273] It should be noted that the above-mentioned computer-readable storage medium can be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), a magnetic random access memory (FRAM), a flash memory (Flash Memory), a magnetic surface memory, an optical disc, or a compact disc read-only memory (CD-ROM) and other memories; it can also be various electronic devices including one or any combination of the above-mentioned memories, such as mobile phones, computers, tablet devices, personal digital assistants, etc.

[0274] It should be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the existence of other identical elements in the process, method, article or device including the element.

[0275] The serial numbers of the above-mentioned embodiments of the present application are for description only and do not represent the advantages or disadvantages of the embodiments.

[0276] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course, by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, a magnetic disk, or an optical disk), and includes a number of instructions for a terminal device (which can be a mobile phone, a computer, a server, an air conditioner, or a network device, etc.) to execute the methods described in each embodiment of the present application.

[0277] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products of the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0278] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0279] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0280] The above are only preferred embodiments of the present application, and are not intended to limit the patent scope of the present application. Any equivalent structure or equivalent process transformation made using the contents of the present application specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present application.

Claims

1. A method for determining information, characterized in that: The method comprises: Receiving an alarm request sent by an alarm object through a terminal, and acquiring call information of the alarm object based on the alarm request; Based on the call information, sending the target page address information corresponding to the alarm request to the terminal; Upon receiving the response information sent by the alarm object with respect to the target page address information, acquiring the target image information of the alarm object; The target image information is subjected to sign language recognition based on a target feature recognition model to obtain target alarm information of the alarm object.

2. The method according to claim 1, characterized in that: The sending, based on the call information, the target page address information corresponding to the alarm request to the terminal includes: In the case that the target content is not extracted from the call information of the target call duration, the target page address information is sent to the terminal.

3. The method according to claim 1, characterized in that The step of obtaining target image information of the alarm object includes: Sending selection information for determining whether the alarmer has a physical dysfunction to the terminal; When confirmation information of the alarm target in response to the selection information is received, target image information of the alarm target is acquired.

4. The method according to claim 3, characterized in that: The step of acquiring target image information of the alarm object upon receiving confirmation information of the alarm object in response to the selection information comprises: Receiving first image information of the alarm object sent by the terminal; wherein the first image information is collected by the terminal using a first image collection module; In a case where the content of the first image information meets the target image content, determining the first image information as the target image information; In the case that the content of the first image information does not satisfy the target image content, updated second image information of the alarm object collected by the terminal is obtained, and the target image information is determined based on the second image information.

5. The method according to claim 4, characterized in that The acquiring the updated second image information of the alarm object collected by the terminal, and determining the target image information based on the second image information, comprises: Sending an adjustment instruction for adjusting the posture and / or position of the alarm object to the terminal; When the first sub-image information of the alarm object sent by the terminal is received within the target time period and the content of the first sub-image information meets the target image content, determining the first sub-image information as the target image information; wherein the second image information includes the first sub-image information; If the first sub-image information is not received within the target time period, or the content of the first sub-image information does not meet the target image content, sending a switching instruction for switching the image acquisition module to the terminal; Receive the second sub-image information of the alarm object collected by the second image acquisition module sent by the terminal, and determine that the second sub-image information is the target image information; wherein, the second image acquisition module is the image acquisition module after the terminal switches based on the switching instruction; the second image information includes the second sub-image information.

6. The method according to claim 1, characterized in that The method further comprises: Collect multiple sample images of multiple hand modes of sample objects in different environments; wherein the multiple sample images are images of hand gestures and torso images of the sample objects in communication; the hand structure used in each hand mode is the same; Based on multiple sample images of each hand pattern, the first feature extraction network and the second feature extraction network in the initial feature recognition model are trained to obtain a target feature recognition model corresponding to each hand pattern; wherein the first feature extraction network is a network capable of extracting features in time and space dimensions; and the second feature extraction network is a convolutional neural network.

7. The method according to claim 6, characterized in that The method of training the first feature extraction network and the second feature extraction network in the initial feature recognition model based on the multiple sample images of each hand pattern to obtain the target feature recognition model corresponding to each hand pattern includes: For the multiple sample images of each hand pattern, each sample image of each hand pattern is identified based on a target recognition algorithm, and a target area of ​​a key part of the alarm object in each sample image is determined; Extracting the image of the target area in each sample image of each hand pattern to obtain a plurality of to-be-processed images of each hand pattern; Model training is performed on the first feature extraction network and the second feature extraction network based on the multiple images to be processed of each hand pattern and the multiple sample images of each hand pattern to obtain a target feature recognition model corresponding to each hand pattern.

8. The method according to claim 7, characterized in that The first feature extraction network and the second feature extraction network are trained based on the multiple images to be processed of each hand pattern and the multiple sample images of each hand pattern to obtain a target feature recognition model corresponding to each hand pattern, including: Based on each layer of the time recurrent neural network, feature extraction is performed on each of the images to be processed to obtain T first feature images; wherein the first feature extraction network is composed of T layers of time recurrent neural networks; Perform feature extraction on each sample image based on each convolution layer in the second feature extraction network to obtain T second feature images; wherein the second feature extraction network includes T convolution layers; Using the fully connected layer in the initial feature recognition model to splice each of the first feature images and each of the second feature images that have a corresponding relationship, to obtain T spliced ​​feature images; The first feature extraction network and the second feature extraction network are trained based on the T spliced ​​feature images to obtain a target feature recognition model corresponding to each hand pattern.

9. The method according to claim 1, characterized in that: Performing sign language recognition on the target image information based on the target feature recognition model to obtain target alarm information of the alarm object includes: Determining a sub-target feature recognition model from a plurality of target feature recognition models based on selection information input by a user; Performing sign language recognition on the target image information based on the sub-target feature recognition model to obtain pre-selected alarm information of the alarm object; Receive correction information of the alarm object for the pre-selected alarm information, and determine the target alarm information based on the correction information.

10. The method according to claim 9, characterized in that The step of performing sign language recognition on the target image information based on the sub-target feature recognition model to obtain pre-selected alarm information of the alarm object includes: When it is determined that the network bandwidth of the terminal meets the target network bandwidth and has an idle recognition module that carries the sub-target feature recognition model, the sub-target feature recognition model carried by the idle recognition module is used to recognize the target image information to obtain the pre-selected alarm information.

11. The method according to claim 1, characterized in that The method further comprises: In the case that the response information sent by the alarm object for the target page address information is not received, a return visit information for returning the alarm object is generated, and the return visit information is sent to the server.

12. The method according to claim 1, characterized in that The method further comprises: Based on the alarm request, obtaining a first position and a second position of the alarm object; wherein the first position is determined by a base station corresponding to the terminal, and the second position is determined by a positioning module set on the terminal; Based on the first position and the second position, a target position of the alarm object is determined.

13. An information determination device, characterized in that: The device comprises: A receiving unit, configured to receive an alarm request sent by an alarm object through a terminal, and obtain call information of the alarm object based on the alarm request; A sending unit, configured to send target page address information corresponding to the alarm request to the terminal based on the call information; A processing unit, configured to obtain target image information of the alarm object upon receiving response information sent by the alarm object with respect to the target page address information; The determination unit is used to perform sign language recognition on the target image information based on a target feature recognition model to obtain target alarm information of the alarm object.

14. An information determination device, characterized in that The device comprises: a processor, a memory and a communication bus; The communication bus is used to realize the communication connection between the processor and the memory; The processor is used to execute the information determination program stored in the memory to implement the steps of the information determination method according to any one of claims 1 to 12.

15. A computer-readable storage medium, characterized in that: The storage medium stores one or more programs, and the one or more programs can be executed by one or more processors to implement the steps of the information determination method according to any one of claims 1 to 12.

Images