Method and device for interactively searching for target objects, security system and storage medium

Abstract

The present disclosure relates to a method and device for interactively searching for a target object, a security system and a storage medium. The method comprises: processing a preset recognition area and displaying it in a first screen; the preset recognition area is configured and generated in a second screen. This embodiment can make the size of the recognition area match the size of the first screen, avoid the problem that the recognition area cannot be displayed or is deformed, and improve the user experience.

Description

Technical Field

[0001] The present disclosure relates to the field of data processing technology, and in particular to a method and device for interactively searching for a target object, a security system, and a storage medium. Background Art

[0002] With the rapid development of security technology, security systems are deployed in many key areas. The cameras in the security system can monitor the security area around the clock by collecting and recording videos. In addition, the existing security system also allows users to plan areas as restricted areas in the video screen through a web page (such as a web terminal) and focus on monitoring the restricted areas. Summary of the invention

[0003] The present disclosure provides a method and device for interactively searching for a target object, a security system and a storage medium to address the deficiencies of related technologies.

[0004] According to a first aspect of an embodiment of the present disclosure, a method for interactively searching for a target object is provided, comprising:

[0005] The preset recognition area is processed and displayed in the first picture; the preset recognition area is configured and generated in the second picture.

[0006] Optionally, processing a preset recognition area includes:

[0007] Acquire initial position data of the recognition area, and acquire the size of the first picture; the initial position data includes first coordinate data of each vertex in the recognition area, and the value of each first coordinate data is within a preset range;

[0008] Adjust the first coordinate data in the initial position data according to the size of the first picture to obtain target position data; the target position data includes the second coordinate data of each vertex in the recognition area in the first picture;

[0009] The identification area is displayed in the first picture according to the second coordinate data of each vertex.

[0010] Optionally, the initial position data is obtained by sorting the vertices of the identification area in a preset order and recording the first coordinate data of each vertex.

[0011] Optionally, the data structure of the initial position data includes:

[0012]

[0013] Optionally, adjusting the first coordinate data in the initial position data according to the size of the first picture to obtain the target position data includes:

[0014] Acquire first coordinate data of each vertex in the initial position data; the first coordinate data includes a horizontal coordinate and a vertical coordinate of the vertex;

[0015] The target coordinate data of each vertex in the first picture is obtained by multiplying the horizontal coordinate of each vertex by the width of the first picture size and multiplying the vertical coordinate of each vertex by the height of the first picture size.

[0016] Optionally, after displaying the identification area in the first picture according to the second coordinate data of each vertex, the method further includes:

[0017] The background color of the recognition area is adjusted to a target color.

[0018] Optionally, the method further comprises a step of generating the initial position data, specifically comprising:

[0019] determining a recognition area according to a target operation or a position of a target object in the second picture;

[0020] Obtain the third coordinate data of each vertex in the recognition area, and save the third coordinate data of each vertex in the setting array pointlist;

[0021] adjusting the third coordinate data of each vertex according to the size of the second picture so that the adjusted first coordinate data is within the preset range;

[0022] The first coordinate data of each vertex is recorded in a preset order to obtain the initial position data, and is saved in a set array areapoints.

[0023] Optionally, the third coordinate data of each vertex in the recognition area is obtained by the following formula:

[0024] Abscissa x=ev.clientX+document.body.scrollLeft-document.body.clientLeft-document.querySelector('.frame').getBoundingClientRect().left;

[0025] Where x = the distance from the vertex to the left side of the window + the width of the page hidden on the left side of the scroll bar when the page is scrolled to the right using the scroll bar - the width of the left side of the area where the content is viewed through the browser - the distance from the drawable area to the left side of the window;

[0026] Ordinate coordinate y=ev.clientY+document.body.scrollTop-document.body.clientTop-document.querySelector('.frame').getBoundingClientRect().Top;

[0027] Among them, y = the distance from the vertex to the top of the window + the width of the page hidden at the top of the scroll bar when the page is scrolled to the bottom using the scroll bar - the width of the top of the area where the content is viewed through the browser - the distance from the top of the drawable area to the window.

[0028] Optionally, the method further comprises:

[0029] When an operation of selecting a functional component in the second screen is detected, an interactive interface corresponding to the functional component is displayed; the interactive interface includes a brush component and / or a save component;

[0030] When an operation of selecting the brush component is detected, a trigger position of the brush component in the second picture is obtained and the trigger position is used as a vertex of the recognition area, and third coordinate data of the trigger position is stored in a setting array pointlist;

[0031] And / or, when the operation of selecting the saving component is detected, the third coordinate data in the setting array pointlist is processed into the first coordinate data and the first coordinate data is saved in the setting array areapoints.

[0032] Optionally, the method further comprises:

[0033] When the operation of selecting the saving component is detected, detecting whether the recognition function type of the recognition area has been configured; the recognition function type includes full-image configuration and / or regional configuration;

[0034] When it is detected that the identification function type is not configured, displaying a type interaction interface including a full-image configuration component and / or a regional configuration component;

[0035] When an operation of selecting the full-image configuration component and / or the area configuration component is detected, the recognition function type is saved.

[0036] Optionally, the method further comprises:

[0037] When the operation of selecting the saving component is detected, preset prompt information for prompting that the configuration of the recognition area is completed is displayed in the second screen.

[0038] Optionally, the interactive interface further includes an eraser component, and the method further includes:

[0039] When the operation of selecting the eraser component is detected, the setting array pointlist and the setting array areapoints are deleted.

[0040] According to a second aspect of an embodiment of the present disclosure, a device for interactively searching for a target object is provided, comprising:

[0041] The recognition area processing module is used to process the preset recognition area and display it in the first picture; the preset recognition area is configured and generated in the second picture.

[0042] According to a third aspect of an embodiment of the present disclosure, there is provided a security system, including:

[0043] processor;

[0044] a memory for storing a computer program executable by the processor;

[0045] The processor is configured to execute the computer program in the memory to implement the above method.

[0046] According to a fourth aspect of an embodiment of the present disclosure, a computer-readable storage medium is provided, and when an executable computer program in the storage medium is executed by a processor, the above-mentioned method can be implemented.

[0047] The technical solution provided by the embodiments of the present disclosure may have the following beneficial effects:

[0048] As can be seen from the above embodiments, the solution provided by the embodiments of the present disclosure can process the preset recognition area and display it in the first screen; the preset recognition area is configured and generated in the second screen. In this way, the size of the recognition area can be matched with the size of the first screen in this embodiment, avoiding the problem of the recognition area not being able to be displayed or deformed, and improving the user experience.

[0049] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present disclosure and, together with the description, serve to explain the principles of the present disclosure.

[0051] Figure 1 The present invention is a flowchart of a method for interactively searching for a target object according to an exemplary embodiment.

[0052] Figure 2The figure is a flow chart showing a method of obtaining initial position data according to an exemplary embodiment.

[0053] Figure 3 The figure is a schematic diagram showing the effect of a configuration interface according to an exemplary embodiment.

[0054] Figure 4 The diagram is a schematic diagram showing an effect of determining and identifying matching vertices according to an exemplary embodiment, wherein (a) shows a schematic diagram showing an effect of determining a horizontal coordinate, and (b) shows a schematic diagram showing an effect of determining a horizontal coordinate.

[0055] Figure 5 The figure is a flow chart showing a method of obtaining an identification function type according to an exemplary embodiment.

[0056] Figure 6 The present invention is a flowchart of obtaining a target object according to an exemplary embodiment.

[0057] Figure 7 The figure is another flowchart of acquiring a target object according to an exemplary embodiment.

[0058] Figure 8 The present invention is a flowchart of obtaining target information according to an exemplary embodiment.

[0059] Fig. 9 The figure is another flowchart of obtaining target information according to an exemplary embodiment.

[0060] Fig.10 The flowchart of another method of acquiring target information is shown according to an exemplary embodiment.

[0061] Fig.11 The flowchart of another method of acquiring target information is shown according to an exemplary embodiment.

[0062] Fig.12 The flowchart of another method of acquiring target information is shown according to an exemplary embodiment.

[0063] Fig.13 The present invention is a schematic diagram showing the effect of displaying a navigation route and a navigation mark according to an exemplary embodiment.

[0064] Fig.14 The figure is a flow chart showing a method of obtaining a feature vector according to an exemplary embodiment.

[0065] Fig.15 The figure is a schematic diagram showing the effect of displaying a historical track and a location image according to an exemplary embodiment.

[0066] Fig.16The invention is a block diagram of a system for interactively searching for a target object according to an exemplary embodiment.

[0067] Fig.17 It is a block diagram of another system for interactively searching for a target object according to an exemplary embodiment.

[0068] Fig.18 The figure is a block diagram of another system for interactively searching for a target object according to an exemplary embodiment.

[0069] Fig.19 The figure is a block diagram of another system for interactively searching for a target object according to an exemplary embodiment.

[0070] Fig. 20 The figure is a block diagram of another system for interactively searching for a target object according to an exemplary embodiment.

[0071] Fig.21 The invention is a block diagram of a security system according to an exemplary embodiment. DETAILED DESCRIPTION

[0072] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described exemplarily below do not represent all embodiments consistent with the present disclosure. Instead, they are merely examples of devices consistent with some aspects of the present disclosure as detailed in the attached claims. It should be noted that the features in the following embodiments and implementations may be combined with each other without conflict.

[0073] To solve the above technical problems, the embodiment of the present disclosure provides a method for interactively searching for a target object, which can be applied to a system for interactively searching for a target object. In one example, the system for interactively searching for a target object may include a first terminal, a second terminal, and a server. The first terminal may be used to display a screen that facilitates users to find objects and input interactive instructions, such as a terminal that users can touch and operate, such as a smart phone, a tablet computer, a personal computer, a vertical screen, a spliced ​​screen, or a joint terminal. The second terminal may include multiple devices with image acquisition functions, such as cameras, vertical screens, etc., which can upload multimedia files such as collected pictures or video streams, uploaded pictures or video streams to the server. The server can be implemented using at least one server or a server cluster, and can process the image uploaded by the second terminal when receiving an interactive instruction from the first terminal, thereby obtaining the target information of the object and displaying it on the first terminal. In the subsequent embodiments, the schemes of each embodiment are described by taking the server executing a method for interactively searching for a target object as an example. However, it does not constitute a limitation.

[0074] Figure 1 is a flow chart showing a method for interactively searching for a target object according to an exemplary embodiment. Figure 1 , a method for interactively searching for a target object, comprising steps 11 to 14.

[0075] In step 11, a first interaction instruction sent by a first terminal is obtained.

[0076] In this embodiment, when displaying a drawing, the first terminal can detect whether the user triggers the touch screen of the first terminal. When there is a need to input an interactive instruction, the user can trigger the interactive interface of the first terminal to input the interactive instruction, wherein the above-mentioned interactive instruction may include but is not limited to the coordinate data of the recognition area, the image of the recognition object, the saved image, the user data obtained by scanning the code, etc., and the appropriate instruction can be selected according to the specific scenario, which is not limited here. For the convenience of description, the instruction to establish a task in this disclosure is referred to as the first interactive instruction, and the instruction to cancel the previously established task is referred to as the second interactive instruction.

[0077] Taking a person-finding scenario as an example, the first interactive instruction may be the coordinate data of the identification area. For example, when there is a need to find a person, the user can click multiple locations in the interactive interface to form a polygon, i.e., an identification box. The area within the identification box is the identification area, and the identification area contains the object to be identified. At this time, the first terminal can obtain the coordinate data of the identification area (in this example, the coordinate data of the vertices in the identification area are used to represent the coordinate data of the identification area) and send the coordinate data as an interactive instruction to the server.

[0078] Taking another scenario of searching for people as an example, the first interactive instruction may be a location image of the identified object. For example, when searching for people, the user may select the upload button in the interactive interface and select an image including the object being searched. The first terminal may use a preset recognition algorithm to identify each object in the image and generate a minimum bounding box for each object, and then cut the image based on the minimum bounding box to obtain the location image corresponding to each object. The first terminal may upload the location image of the identified object to the server as the first interactive instruction. In this way, the server may obtain the first interactive instruction.

[0079] Taking a parking lot car search scenario as an example, the first interaction instruction may be user data obtained by scanning a code. For example, when there is a need to find a car in a parking lot, the first terminal may display a code scanning interface or a QR code in the interactive interface in response to the user's triggering operation. Taking the appearance of a code scanning interface as an example, the first terminal may scan the QR code that appears on the vertical screen in the second terminal, thereby communicating with the vertical screen and uploading user data and establishing a mapping relationship, wherein the user data may be at least one item of information filled in by the user during registration, such as name, user name, license plate number, and first terminal identification code. The first terminal may send the user data as a first interaction instruction to the server via the vertical screen. In this way, the server may obtain the above-mentioned first interaction instruction.

[0080] Taking another parking lot car search scenario as an example, the first interactive instruction can be a user image collected by a camera with the user's knowledge and the identification of user feature data. For example, when there is a need to find a car in a parking lot, the first terminal can switch from the interactive interface to the shooting interface in response to the user's trigger operation, and shoot an image when a face appears in the shooting interface; then, the image is recognized using a preset image recognition model to obtain user feature data. Among them, the first terminal can be a terminal device held by the user, or it can be a device such as a vertical screen in the parking lot, which can be set according to the specific scenario. Finally, the vertical screen in the parking lot sends the user data as the first interactive instruction to the server. In this way, the server can obtain the above-mentioned first interactive instruction.

[0081] It should be noted that, in one example, the first terminal can directly send the coordinate data of the identification area as the first interaction instruction to the server, and can also send the coordinate data, i.e., initial coordinate data, as the first interaction instruction to the server, wherein the coordinate data of each vertex in the initial coordinate data is within a preset range, wherein the preset range refers to [0,1]. Then, the first terminal may obtain the initial coordinate data of the identification area including steps 21 to 24.

[0082] In step 21, the first terminal may determine the recognition area according to the position of the target operation or the target object in the current screen (hereinafter referred to as the second screen). In actual applications, when there is a need to set a recognition area, the user usually clicks continuously to determine multiple points in the second screen (current screen or browser presentation) of the display, and these points can be used as vertices of the recognition area to enclose a recognition area.

[0083] That is to say, the recognition area usually has multiple vertices, which are connected in sequence to form a polygon, and the internal area of ​​the polygon constitutes the recognition area, such as a triangular area, a quadrilateral area, etc. In this example, the first coordinate data of each vertex is located in a preset range, which is [0, 1]; in other words, a reference picture can be set, the upper left corner of the reference picture is set as the coordinate origin (0, 0), and the lower right corner of the reference picture is set as (1, 1), and the initial position data of the recognition area is the coordinate data of the above reference picture.

[0084] In one example, the interactive interface is usually provided with functional components. When the operation of selecting a functional component in the second screen is detected, the first terminal can display the interactive interface corresponding to the functional component. The interactive interface may include a brush component and a save component. Figure 3 The interactive interface may include a brush component 31 and a saving component 32.

[0085] When the operation of selecting the brush component is detected, the first terminal can obtain the trigger position of the brush component in the second screen, and use the trigger position as a vertex of the recognition area. At this time, the third coordinate data of the trigger position can be obtained.

[0086] Take a vertex P of the recognition area as an example, see Figure 4 In Figures (a) and (b), the browser 43 is displayed in full screen on the display screen 42 of the first terminal; the second picture 41 is displayed in the browser 43, the trigger position P in the second picture 41 is used as the vertex P, and the file 44 is displayed in the browser 43. Then the third coordinate data of the trigger position P is as follows:

[0087] Abscissa x=ev.clientX+document.body.scrollLeft-document.body.clientLeft-document.querySelector('.frame').getBoundingClientRect().left;

[0088] Among them, x = the distance between the vertex P and the left side of the window (ie ev.clientX) + the width of the page hidden on the left side of the scroll bar when the page is scrolled to the right using the scroll bar (ie document.body.scrollLeft) - the width of the left side of the area where the content is viewed through the browser (ie document.body.clientLeft) - the distance between the second screen and the left side of the window (document.querySelector('.frame').getBoundingClientRect().left).

[0089] Ordinate coordinate y=ev.clientY+document.body.scrollTop-document.body.clientTop-document.querySelector('.frame').getBoundingClientRect().Top;

[0090] Among them, y = the distance from the vertex to the top of the window (ie ev.clientY) + the width of the page hidden at the top of the scroll bar when the page is scrolled to the bottom using the scroll bar (ie document.body.scrollTop) - the width of the top of the area where the content is viewed through the browser (ie document.body.clientTop) - the distance from the second screen to the top of the window (document.querySelector('.frame').getBoundingClientRect().Top).

[0091] In step 22, the first terminal can obtain the third coordinate data of each vertex in the identification area, and save the third coordinate data of each vertex in the setting array pointlist of the specified position. In this step, after obtaining the third coordinate data of the trigger position, the first terminal can store it in the setting array pointlist. The user can use the brush component to repeat the operation (i.e., multiple click operations) in the second screen, and the first terminal can detect multiple trigger positions and the third coordinate data of each trigger position. In actual application, when more than 3 trigger positions are detected, the first terminal can connect these trigger positions in sequence to form a closed candidate area, and display it in the second screen for the user to view.

[0092] In step 23, the first terminal may adjust the third coordinate data of each vertex according to the size of the second screen so that the adjusted first coordinate data is within the preset range. Figure 3 , the interactive interface also includes a saving component 32. When the operation of selecting the completion component is detected, the first terminal can process the third coordinate data in the setting array pointlist into the first coordinate data and save the first coordinate data to the setting array areapoints at the specified position. For example, the size of the second screen includes width and height. The first terminal can divide the horizontal coordinate in the third coordinate data by the width of the second screen and the vertical coordinate in the third coordinate data by the height of the second screen, thereby reducing the third coordinate data to the first coordinate data.

[0093] In step 24, the first terminal may record the first coordinate data of each vertex in a preset order, obtain the coordinate data of the identification area, i.e., the initial position data, and save it in the setting array areapoints. The above preset order may include clockwise or counterclockwise. For example, the first terminal may record the first coordinate data of each vertex of the identification area in a clockwise order based on the center of the identification area, and the first coordinate data of all vertices of the identification area may constitute the initial position data. In addition, the first terminal may save the above initial position data in the setting array areapoints, and the initial position data in the setting array areapoints may be uploaded to the server as a first interaction instruction.

[0094] It should be noted that the setting array pointlist is the variable name of the first terminal storing the third coordinate data, and the setting array areapoints is the variable name of the first terminal storing the first coordinate data, that is, the setting array pointlist and the setting array areapoints are used to store the coordinate data before and after processing respectively. In one example, the setting array pointlist and the setting array areapoints can be implemented using a first-in-first-out queue, so as to facilitate the storage and reading of the vertices of the recognition area in a preset order, thereby achieving the effect of improving access efficiency.

[0095] In one embodiment, when the first terminal detects the operation of selecting to save the component, a preset prompt message for prompting that the configuration of the recognition area is completed is displayed in the second screen. In one example, the first terminal may display a preset prompt message of "Region Configuration Completed" in the upper left corner of the second screen, and use an animation effect that fades out within 3 seconds to remind the user, so that the user can confirm that the recognition area has been matched, thereby improving the user experience.

[0096] In one example, the data structure of the initial position data includes:

[0097]

[0098] Combined with the coordinate structure of the array areapoints in the above data structure, the third coordinate data is 52,133.25,581,81.25,754,298.25,42,324.25, that is:

[0099] The horizontal coordinate of the first vertex is 52, and the vertical coordinate of the first vertex is 133.25;

[0100] The horizontal coordinate of the second vertex is 581, and the vertical coordinate of the second vertex is 81.25;

[0101] The abscissa of the third vertex is 754, and the ordinate of the third vertex is 298.25;

[0102] The horizontal coordinate of the 4th vertex is 42, and the vertical coordinate of the 4th vertex is 324.25.

[0103] Assume that the width of the second picture is 800 and the height is 450, in pixels. Thus, the converted first coordinate data is:

[0104] 0.065,0.2961111111111111,0.72625,0.180555555555555555,0.9425,0.66277777777777778,0.0525,0.7205555555555555.

[0105] Right now:

[0106] The abscissa of the first vertex is 0.065, and the ordinate of the first vertex is 0.29611111111111111;

[0107] The abscissa of the second vertex is 0.72625, and the ordinate of the second vertex is 0.180555555555555555;

[0108] The abscissa of the third vertex is 0.9425, and the ordinate of the third vertex is 0.66277777777777778;

[0109] The horizontal coordinate of the 4th vertex is 0.0525, and the vertical coordinate of the 4th vertex is 0.72055555555555555.

[0110] In one embodiment, see Figure 3 The interactive interface also includes an eraser component 33. When the operation of selecting the eraser component 33 is detected, the first terminal can delete the setting array pointlist and the setting array areapoints, so as to facilitate resetting the recognition area.

[0111] In combination with the data structure of the initial location data, in one embodiment, the first terminal may also determine the identification function, see Figure 5 , including steps 51 to 53.

[0112] In step 51, when the operation of selecting the saving component is detected, the first terminal can detect whether the recognition function type of the recognition area has been configured; the recognition function type includes full-image configuration and area configuration, wherein the full-image configuration means that the recognition area occupies the entire second screen, and the area configuration means that the recognition area occupies a portion of the second screen.

[0113] In step 52, when it is detected that the identification function type is not configured, the first terminal may display a type interaction interface including a full-map configuration component and / or a regional configuration component.

[0114] In step 53, when the operation of selecting the full map configuration component or the area configuration component is detected, the first terminal can save the identification function type. In this way, this embodiment can ensure the integrity of the data structure by setting the identification function type, which is conducive to the subsequent display of the identification area.

[0115] When a second terminal needs to display the above-mentioned identification area and object, it can read the above-mentioned initial position data from the server and obtain the size of its current screen, i.e., the first screen. Considering that transferring the display content of one terminal to another terminal for display will be affected by the sizes of the two terminals or screens, that is, the second screen of the first terminal will change when it is converted to the first screen of the second terminal.

[0116] To this end, the second terminal can obtain the first size of the first screen and the second size of the second screen; then obtain the ratio of the first size and the second size; thereafter, the size of the first screen can be obtained by multiplying the second screen by the above ratio. In other words, when the size of the second terminal changes compared to the size of the first terminal where the user sets the recognition area, the first screen and the second screen also change in the same proportion, and then the recognition area will also change in the same proportion, avoiding the problem that the recognition area cannot be displayed or deformed, and improving the user experience. In this way, in this embodiment, the recognition area can be set on any display device, and the above recognition area can be displayed on any display device of the system for interactively searching for the target object, that is, there is no need to configure the above recognition area on all display devices, thereby improving the configuration efficiency.

[0117] For the convenience of calculation, assume that the size of the second screen is the same as that of the first screen, then the size of the first screen is the size of the second screen, and the ratio of the two is 1, that is, the width of the first screen is 800, the height is 450, and the unit is pixel. The effect is as follows Figure 4 shown.

[0118] It should be noted that, considering that the configuration data of different terminals when displaying screens are different, the coordinate data of the identification area uploaded by the first terminal, that is, the initial position data, can be obtained at the second terminal, and a triangle can be generated according to the preset conversion rule. For example, the identification area of ​​the first terminal can be a rectangle and the identification area of ​​the second terminal can be a triangle, then the preset conversion rule can be to generate a minimum circumscribed triangle based on the shape of the identification area in the initial position data, and use the minimum circumscribed triangle as the identification frame of the second terminal. For example, the identification area of ​​the first terminal can be a triangle and the identification area of ​​the second terminal can be a rectangle, then the preset conversion rule can be to generate a minimum circumscribed rectangle based on the shape of the identification area in the initial position data, and use the minimum circumscribed rectangle as the identification frame of the second terminal. That is to say, after obtaining the initial position data, it can be converted into the target position data corresponding to the terminal and the shape of the identification area can be obtained, and then a minimum circumscribed shape can be generated on the basis of the above shape as the shape of the identification area of ​​the terminal. It can be understood that the above preset conversion rule only illustrates an example of conversion between different identification area shapes. The technician can select the corresponding conversion rule according to the specific scenario, such as the minimum circumscribed circle, the minimum inscribed circle, etc., and the corresponding scheme also falls within the protection scope of the present disclosure. It is understandable that in this embodiment, different terminals are allowed to have different recognition frames, which can meet the needs of different users and help improve the user experience. In other words, when the target object appears on different terminals, the recognition frames of different terminals can mark the same target object, ensuring that the target object is not lost, and achieving the effect of tracking the object.

[0119] In step 12, a target object corresponding to the first interaction instruction is determined.

[0120] In this embodiment, after obtaining the first interaction instruction, the server may determine the target object corresponding to the first interaction instruction. The target object may include at least one of the following: a person, a vehicle, and an identification frame of an identification area. When the target object is a person, it may be represented by a feature vector containing a position image of the person; when the target object is a vehicle, it may be represented by a license plate number of the vehicle, or it may be represented by a feature vector of a position image of a user of the vehicle. When the target object is an identification frame of an identification area, the identification frame of the identification area may include an identification frame of a shape such as a rectangle, a triangle, a circle, or the like as described above.

[0121] In one embodiment, taking the target object as a person as an example, when the first interaction instruction is the coordinate data of the recognition area, see Figure 6In step 61, the server may obtain the image displayed by the first terminal when the first interaction instruction is detected, that is, obtain the current image corresponding to the first interaction instruction, and the current image includes the current video frame or image frame specified by the first interaction instruction or the image uploaded when the first interaction instruction is input. Taking the first terminal displaying the video stream as an example, in step 62, the server may identify the target object in the current image and obtain a position image of the target object. In step 63, the server may extract the feature vector of the above-mentioned position image and use the feature vector of the position image to characterize the target object. This embodiment may be applicable to the scenario of uploading an image or displaying a certain image, which is conducive to improving processing efficiency.

[0122] In one embodiment, taking the target object as a person as an example, when the first interaction instruction is the coordinate data of the recognition area, see Figure 7 In step 71, the server can obtain the multimedia file address corresponding to the first interactive instruction, that is, in addition to uploading the coordinate data of the identification area, the first terminal can also upload the multimedia file address as the content of the first interactive instruction to the server at the same time. In step 72, the server can obtain the first video frame according to the above multimedia file address. In step 73, the server can determine the position image corresponding to the identification area, that is, the position image of the target object, based on the coordinate data of the identification area included in the first interactive instruction and the first video frame. In step 74, the server can extract the feature vector of the position image and use the feature vector of the position image to characterize the target object. In this way, in this embodiment, when uploading the first interactive instruction, the speed of transmitting the multimedia file address is faster than the speed of transmitting the image, which is conducive to improving processing efficiency.

[0123] In one embodiment, taking the target object as an identification frame as an example, when the first interaction instruction is coordinate data of the identification area, the server may generate an identification frame according to the coordinate data of the identification area, and use the identification frame to represent the target object.

[0124] Considering that the coordinate data of the identification area in the first interaction instruction may be stored in the form of the above-mentioned initial position data, the server can obtain the coordinate data of each vertex in the initial position data, for example,

[0125] 0.065,0.2961111111111111,0.72625,0.180555555555555555,0.9425,0.66277777777777778,0.0525,0.7205555555555555,

[0126] That is, the abscissa of the first vertex is 0.065, and the ordinate of the first vertex is 0.29611111111111111;

[0127] The abscissa of the second vertex is 0.72625, and the ordinate of the second vertex is 0.180555555555555555;

[0128] The abscissa of the third vertex is 0.9425, and the ordinate of the third vertex is 0.66277777777777778;

[0129] The horizontal coordinate of the 4th vertex is 0.0525, and the vertical coordinate of the 4th vertex is 0.72055555555555555.

[0130] Then, the server may obtain the size of the first screen, which may be the screen displayed by the first terminal or the screen displayed by the second terminal. Furthermore, the server may multiply the horizontal coordinate of each vertex by the width in the first screen size and the vertical coordinate of each vertex by the height in the first screen size to obtain the target coordinate data of each vertex in the first screen.

[0131] Take the first screen and the second screen as an example, assuming that the width of the second screen is 800 and the height is 450, in pixels. In this way, after obtaining the second coordinate data of each vertex in the first screen, the server can obtain the target coordinate data, that is, the second coordinate data of each vertex in the recognition area in the first screen is the same as the third coordinate data in the second screen, and the second coordinate data in the target coordinate data is 52, 133.25, 581, 81.25, 754, 298.25, 42, 324.25;

[0132] That is, the horizontal coordinate of the first vertex is 52, and the vertical coordinate of the first vertex is 133.25;

[0133] The horizontal coordinate of the second vertex is 581, and the vertical coordinate of the second vertex is 81.25;

[0134] The abscissa of the third vertex is 754, and the ordinate of the third vertex is 298.25;

[0135] The horizontal coordinate of the 4th vertex is 42, and the vertical coordinate of the 4th vertex is 324.25.

[0136] Assume that the width and height of the second picture are 800 and 450, respectively, in pixels. Thus, after obtaining the second coordinate data of each vertex in the first picture, the target coordinate data can be obtained.

[0137] In one embodiment, after the recognition area is displayed in the first screen, the background color of the recognition area can be adjusted to the target color, such as green, red, etc. Taking the recognition area as a restricted area as an example, the background color can be adjusted to red, and taking the recognition area as an open area as an example, the background color can be adjusted to green. In this way, by adjusting the background color of the recognition area, it is helpful to improve the recognition of the recognition area.

[0138] In some embodiments, when the first interaction instruction is user data obtained by scanning a code, the server can determine the target object based on the user data. For example, the parking position of the vehicle can be determined based on the license plate number in the user data and the parking position of the vehicle corresponding to the vehicle number in the historical data, and an identification frame can be generated at the parking position of the vehicle. Alternatively, the server can obtain the current position of the first terminal, and then generate an identification frame corresponding to the current position on the map, and use the identification frame to represent the target object.

[0139] It should be noted that in the person-searching scenario, the user usually searches for the target object through the screen of the first terminal. At this time, the target object may appear in the coverage of the second terminal, and the screen captured by the second terminal will be synchronized to the first terminal for display. In other words, for the target object and the user (monitoring personnel), the user can be regarded as a stationary state, while the target object is in a moving state, and the moving direction of the target object is random with the user. In the car-searching scenario, the vehicle can be regarded as a user in the person-searching scenario and the driver can be regarded as the target object in the person-searching scenario in this disclosure. The moving direction of the target object is to move toward the vehicle. The essence of the above concept is that the movement of the driver and the stillness of the vehicle are relative, so the movement of the driver can be converted into the relative movement of the vehicle, that is, the driver himself moves and is regarded as still and the vehicle itself is still and is regarded as moving. In other words, the person-searching scenario and the car-searching scenario in this disclosure belong to the same concept, and both can search for the target object (the person to be found or the driver) according to the concept of searching for people.

[0140] In step 13, target information of the target object is obtained; the target information includes at least one of the following: a feature vector of a position image, a current position of the target object, image information of the current position, a historical track of the target object, and a navigation route.

[0141] In this embodiment, the server can obtain target information of the target object.

[0142] In one embodiment, taking the case where the target information is a feature vector of a position image of a target object as an example, the target information may be a feature vector of the position image. Figure 8In step 81, the server may obtain the first terminal device number corresponding to the first interactive instruction. In step 82, the server may obtain the peripheral device number of at least one second terminal around the first terminal according to the first terminal device number, and use the peripheral device number and the feature vector of the location image as the target information corresponding to the target. In some embodiments, the server may send the above-mentioned feature vector and the peripheral device number to a designated storage area, and the designated storage area includes at least one of the following: a message queue or an object database. Among them, the server pre-stores the second terminal device number of the second terminal, the physical location of each terminal, and the number of the terminal around each terminal, etc., and based on the above information, a first mapping relationship between the second terminal and the second terminal can be generated.

[0143] It should be noted that in the person search scenario, the first terminal is used to display the screen of a second terminal containing the target object, so the first terminal actually refers to the second terminal containing the target object. In the car search scenario, the first terminal is also used to display the screen of a second terminal (such as a vertical screen or camera) containing the target object, and the position of the second terminal is used to represent the position of the first terminal, so the first terminal actually refers to the second terminal containing the target object.

[0144] In other words, the server can understand the above-mentioned first mapping relationship as the second mapping relationship between the first terminal and the second terminal. In this way, the second mapping relationship between the first terminal and the second terminal is pre-stored in the server, and the second mapping relationship may include the first terminal device number and the second terminal device number, the physical location of each terminal, and the number of terminals around each terminal, etc. The technician can set the content of the second mapping relationship according to the specific scenario, and the corresponding scheme falls within the protection scope of this disclosure. In this way, in this embodiment, by obtaining the peripheral device number of at least one second terminal around the first terminal, the source of subsequent acquisition of other data can be determined, which is conducive to improving data processing efficiency.

[0145] In one embodiment, see Fig. 9In step 91, the server may obtain the current image corresponding to the first interaction instruction; the current image includes the current video frame or image frame specified when the first interaction instruction is input or the image uploaded when the first interaction instruction is input. In step 92, based on the preset second mapping relationship between the first terminal and the second terminal, the server may obtain the scene information of the scene where the second terminal is located according to the location data of the second terminal distributed at different locations and the second mapping relationship. Among them, the scene information includes at least one of the following: images and scene elements collected by the second terminal; the scene elements include at least one of the following: the second terminal device number and the peripheral device number around the second terminal. In step 93, the server may obtain the target information of the target object according to the scene information.

[0146] In one embodiment, the scene information includes screen information, and the server can obtain the scene information of the scene where the second terminal is located based on the location data of the second terminal distributed at different locations and the second mapping relationship, which may include: the server can obtain the feature vector of the current image corresponding to the first interaction instruction. Then, the server can obtain the candidate feature vector that matches the feature vector of the current image in the object database to obtain the target feature vector; the object database includes the candidate feature vector corresponding to the screen information uploaded by the second terminal that has the second mapping relationship with the first terminal. After that, the server can obtain the screen information corresponding to the target feature vector and use the screen information as the scene information. In this way, in this embodiment, it is only necessary to determine the target feature vector from the candidate feature vector that has the second mapping relationship with the first terminal, which can improve the efficiency of obtaining scene information.

[0147] In one embodiment, in the process of obtaining the target feature vector, the server may update the object database, which includes several candidate feature vectors. For example, when the object database includes the first database, the server may obtain the current image described by the target object, which is the first frame image corresponding to the current stream address of the first interactive instruction; identify the second number of objects in the first frame image, and obtain the second number of position images; extract the features of each position image respectively, and obtain the second number of feature vectors; update the second number of feature vectors to the first database, and obtain the feature vector ID. For another example, when the object database includes the second database, the server may save the second number of position images to the second database, and obtain the URL address returned by the second database that can access each position image, and update the feature vector of each position image to the feature vector ID generated by the first database; generate a second number of data according to the feature vector ID and the URL address, and save the second number of data to the second database. In this way, when the server obtains the feature vector ID, it can search for the position image in the second database, so as to achieve the effect of one-to-one matching between the feature vector and the position image. Then, the server can match the feature vector of the target object with the candidate feature vectors in the object database to obtain the first number of candidate feature vectors. Afterwards, the server may obtain a candidate feature vector with the greatest similarity among the first number of candidate feature vectors as a target feature vector.

[0148] In one embodiment, the target information is the historical trajectory of the target object. Fig.10 In step 93, the server may obtain the historical track of the target object according to the scene information, including steps 101 to 102. In step 101, the server may obtain at least one object of the picture information in the scene information, and the at least one object includes the target object. In step 102, the server may generate the historical track of the target object according to the time sequence in which the target object appears in each scene information, and use the historical track as the target information of the target object. In this way, the target object can be tracked by obtaining the historical track of the target object in this embodiment, which is suitable for the scene of searching for people.

[0149] It is understandable that in step 102, when the server sorts the scene information according to the time when the target object appears in each scene information, it can obtain the scene information closest to the current time; obtain the position of the second terminal corresponding to the scene information, and use the position of the second terminal as the current position of the target object, so that the server can use the current position of the target object as the target information of the target object.

[0150] In one embodiment, the target information is the picture information of the target object as an example, see Fig.11In step 93, the server can obtain the historical trajectory of the target object according to the scene information, including steps 111 to 113.

[0151] In step 111, the server may extract at least one object of the picture information in the scene information, and obtain the position image corresponding to each object. In step 112, the server may determine the target position image matching the current image. The second terminal may upload the collected image or video frame to the server. The server may adjust the image or video frame uploaded by the second terminal, obtain at least one object in each image or video frame, and obtain the position image corresponding to each object. Then, the server may extract the feature vector of each position image and store it in the first database in the object data as a candidate feature vector. The server obtains the feature vector of the current image, and uses the feature vector of the current image to match the candidate feature vector in the first database in the object data, for example, calculating the similarity of the two feature vectors; then sorting the similarity, taking the candidate feature vector corresponding to the maximum similarity as the target feature vector, and the position image corresponding to the target feature vector as the target position image. In step 113, the server may determine the position data of the second terminal corresponding to the target position image according to the corresponding relationship between the scene information and the second terminal, and take the position data of the second terminal as the target information of the target object, thereby obtaining the real-time position of the target object.

[0152] In one embodiment, taking the target information as a navigation route as an example, see Fig.12, in step 93, the server can obtain the navigation route according to the scene information, including steps 121 to 122. In step 121, the server can obtain the location data of the first terminal. The location data of the first terminal can be uploaded to the server through the communication between the first terminal and the second terminal (such as a vertical screen), and the server can obtain the location data of the first terminal. Alternatively, the first terminal communicates with the reference signal source to obtain the location data of the first terminal, for example, the server can obtain the location data of at least three reference signal sources that are close to the first terminal; wherein the reference signal source is pre-set at a known position. In one example, the server can obtain the signal strength of the reference signal sent by the reference signal source received by the first terminal, and then sort the signal strength, and control the selection of at least three reference signal sources corresponding to the signal strength in order from large to small, and obtain the location data of at least three reference signal sources. In another example, the server obtains the distance between the first terminal and each of the at least three reference signal sources, calculates the coordinate data of the first terminal based on the location data and corresponding distances of the at least three reference signal sources, and obtains the location data of the first terminal. In another example, the server may use a preset weighted centroid algorithm to calculate the location data of the first terminal. Taking the location data of three reference signal sources as an example, the weighted centroid algorithm formula is as follows:

[0153]

[0154]

[0155] In the formula, x b ,y b They respectively represent the horizontal and vertical coordinates of the position of the first terminal, x1, x2, x3 respectively represent the horizontal coordinates of the first reference signal source, the second reference signal source and the third reference signal source, and y1, y2, y3 respectively represent the vertical coordinates of the first reference signal source, the second reference signal source and the third reference signal source.

[0156] In this way, the server can determine the location data of the first terminal itself based on the location data of at least three reference signal sources. It should be noted that the reference signal source can be a separately set signal source for positioning (such as a Bluetooth signal transmitter), and the first terminal can also be determined by using a second terminal with known location information. The positioning principle is similar and will not be repeated here. It should also be noted that when obtaining the navigation route, the location information of the first terminal at this time is the actual location of the first terminal itself; when the first terminal displays a picture and the picture displayed by the first terminal comes from the second terminal, it is necessary to switch to the picture of a second terminal at this time. The location information of the first terminal is an equivalent position, which is the position of the second terminal corresponding to the currently displayed picture. The specific meaning of the location information of the first terminal can be analyzed according to the specific scenario.

[0157] In step 122, the server can determine the navigation route between the first terminal and the second terminal according to the location data of the first terminal and the location data of the second terminal corresponding to the target location image, and use the navigation route as the target information of the target object. In this way, in this embodiment, by obtaining the navigation route, the user can accurately find the target object while holding the first terminal, thereby improving the search efficiency.

[0158] In one embodiment, the target information may also include a navigation identifier, which may be used to assist the user in determining the movement route and movement direction. The server may generate a navigation identifier according to the orientation of the first terminal and the navigation route, and use the navigation identifier as the target information of the target object. Fig.13 , the navigation route 131 shows the suggested route from the initial position to the location of the target object. The navigation route can be the route with the shortest distance, the shortest time, the least number of left turns, the least number of traffic lights, etc., and can be selected according to the specific scenario. The navigation mark 132 shows the direction from the current position of the first terminal to the location of the target object, such as forward, left turn, right turn, so that the user is always on the above navigation route, so as to achieve the effect of matching the search route with the navigation route, which is conducive to improving the search efficiency.

[0159] In one embodiment, when the scene information includes screen information, the server can obtain the target information of the target object according to the scene information in step 93, and the server can obtain the screen information of the second terminal closest to the first terminal, and use the screen information as the target information of the target object. For example, during the movement, the first terminal can establish a communication connection with the second terminal via Bluetooth or WiFi, and use the second terminal closest to the first terminal as the current location of the first terminal, and send the screen information uploaded on the second terminal to the target information of the first terminal, that is, use the screen information of the second terminal as the target information of the target object. In this way, the screen information of the second terminal closest to the first terminal can be continuously displayed in the first terminal, so that the user can find the target object in time.

[0160] In one embodiment, the server may send screen information of two or more second terminals that are close to each other to the first terminal, so that the first terminal can display multiple screen information in split screen. In the process of displaying multiple screen information, the first terminal can detect the operation of the user clicking on a certain screen and display the screen separately. Of course, the first terminal can also detect the user's return operation and resume displaying multiple screen information. In this embodiment, by displaying a certain screen separately or displaying multiple screens, it is convenient for users to query each screen, which is conducive to quickly finding the target object.

[0161] In one embodiment, the server can send the identification code information (such as MAC address, second terminal number or name) of the second terminal that is closer to the first terminal to the first terminal, so that the first terminal can establish a wireless connection such as Bluetooth connection, WiFi connection, etc. with the second terminal through the above identification code information, and receive the drawing information of the second terminal through the wireless connection, thereby reducing the data transmission pressure of the server.

[0162] In another embodiment, the target information further includes target sub-map data, and the target sub-map data may include at least one of the following: map data and a map identification code. Map data refers to a map image that can be used directly, and a map identification code refers to an identification code of a map image. When the first terminal does not store map data, the target sub-map data may be map data, and this method may reduce the occupation of storage resources in the first terminal; when the first terminal pre-downloads map data, the target sub-map data may be a map identification code, so that the first terminal can search for local map data to obtain the target sub-map when obtaining the map identification code, and this method may reduce the amount of data transmission between the first terminal and the server, and improve communication efficiency. The server may obtain the screen information of the second terminal closest to the first terminal. Then, the server may obtain the target sub-map data of the site where the second terminal is located according to the screen information, and use the target sub-map data as the target information of the target object. In one example, the server may obtain a feature vector of the screen information. For example, the server may use a preset recognition algorithm to identify the screen information of the second terminal to obtain a feature vector of the screen information, or determine multiple feature points (such as texture, corner, identification information, etc.) in the above screen information and generate a feature vector according to the feature point data. Then, the server can match the feature vector of the picture information with the feature vector in the preset visual sub-map database, such as calculating the similarity of the two feature vectors, so as to obtain the feature vector with the greatest similarity. After that, the server can obtain the sub-map data corresponding to the feature vector with the greatest similarity, and obtain the target sub-map data. In this way, by providing the target sub-map data in this embodiment, accurate target information can be provided, that is, it is convenient for users to determine the location of the target object and find the route, reduce the difficulty of finding the target object, and thus improve the search efficiency.

[0163] In actual applications, the visual sub-map database stores a lot of sub-map data, which may increase the retrieval time. Therefore, in one embodiment of the present disclosure, classification options can be added to the sub-maps in the visual sub-map database. The classification options may include but are not limited to: roof, ground, column, corner, parking space number, sign shape (such as triangle, diamond, square, trapezoid, etc.), etc. Based on the above classification options, the sub-maps in the visual sub-map database can be divided into multiple categories. Fig.14In step 141, the server may obtain feature points in the image information and determine the classification of the feature points. In step 142, the server may obtain the submap data of the classification in the visual submap data. In step 143, the server may match the feature vector of the image information with the feature vector of the submap data of the classification to obtain the feature vector with the greatest similarity. In this way, in this embodiment, the submap data of one classification or part of the classification can be reduced from matching all the submap data, thereby reducing the amount of matching data, which is conducive to improving the efficiency of obtaining the target submap and is suitable for scenarios with high real-time requirements.

[0164] In one embodiment, the target information may be the coordinate data of the identification area. The server obtaining the target information of the target object may include: the server may determine the identification area corresponding to the first interaction instruction. It is understandable that since the above-mentioned identification area may be the area formed by the identification box delineated by the user in the screen or the area where the target image is located in the image uploaded by the second terminal, the target object is located within the identification area by default. Then, the server may obtain the coordinate data of the identification area and use the coordinate data of the identification area as the target information of the target object. In this way, the identification area or identification box is generated by the coordinate data of the identification area to indicate the target object, so as to facilitate timely finding of the target object.

[0165] It should be noted that, considering that an identification box may be displayed in the screen of the first terminal or the second terminal based on the coordinate data of the identification area, the server can process the coordinate data of the above identification area into initial position data. For example, the server can obtain the third coordinate data of each vertex in the identification area; then adjust the third coordinate data of each vertex according to the size of the screen in the first terminal to obtain the first coordinate data, and the first coordinate data is within a preset range; then record the first coordinate data of each vertex in a preset order to obtain the position data of the identification area. It is understandable that the specific content of obtaining the initial position data can be found in Figure 2 The contents shown will not be repeated here.

[0166] In step 14, the target information is sent to the first terminal so that the first terminal displays the target information.

[0167] In this embodiment, the server can directly send the target information to the first terminal so that the first terminal displays the target information. Alternatively, the server can process the multimedia file of the second terminal according to the target information, for example, at least one of the identification box, historical track, navigation route, current location, location image, etc. in the target information is superimposed on the video frame of the second terminal, thereby obtaining the target multimedia file containing the target information. The server can push the target multimedia file to the first terminal. The display effect of the first terminal playing the target multimedia file is as follows: Fig.15See Fig.15 , reference numeral 151 represents the historical trajectory of the target object, reference numeral 152 represents the position image and historical position of the target object, and reference numeral 153 represents the position image and current position of the target object.

[0168] In this embodiment, the target information includes an identification frame, that is, no matter which second terminal screen the target object appears in during the movement, the first terminal screen will be marked with an identification frame, so that the user can instantly find the target object in the screen without losing the tracking object, thereby improving usage efficiency.

[0169] Considering the limited coverage of the cameras in the first terminal and the second terminal, the target object may move out of the coverage of one terminal. To this end, in the embodiment of the present disclosure, the server can also detect whether the target object leaves the screen of the first terminal, that is, detect whether the target object leaves the coverage of a certain terminal. When it is detected that the target object leaves the screen of the first terminal, the peripheral device number of at least one second terminal around the first terminal and the feature vector of the target object are obtained. Then, the server can re-acquire the information of the target object based on the peripheral device number and the feature vector of the target object. In other words, the peripheral device number of at least one second terminal around the first terminal and the feature vector of the target object are used as the target object, and the scheme of steps 12 to 14 is continued until a second terminal that can cover the target object is found; then the screen information of the second terminal is fed back to the first terminal for continued display, and the target information of the target object, that is, the target multimedia file in the above embodiment, is superimposed. In this way, in this embodiment, by switching the screen information of different second terminals to the first terminal, the effect of searching for objects across the mirror can be achieved, and the search embodiment can be improved.

[0170] Considering some scenarios, such as the scene of searching for people or searching for cars in the city command system, one first terminal can no longer meet the needs of the actual scene. At this time, the first terminal can be set to multiple, one of the multiple first terminals is used as the main terminal and the others are used as sub-terminals, so that the server sends the target information to each first terminal, for example, first to the main terminal in the first terminal, and then to each sub-terminal according to the mapping relationship between the main terminal and the sub-terminal, so that multiple first terminals synchronously display the same target information, meeting the display requirements of multiple first terminals, especially first terminals located in different spaces. In one embodiment, when the first terminal includes multiple terminals, it can be grouped, and each group can display a part of the screen displayed by the main terminal. For example, the first terminal can display a combination of 16 screens, and its 4 sub-terminals can display 4 of the above 16 screens respectively, so that multiple first terminals can ensure that a part of the screen is displayed together, so that the main terminal can display all pictures from the global perspective. For example, the first terminal of the provincial capital center and the first terminal of each prefecture-level city center in the city command system keep synchronously displaying at least part of the same content, which can improve the monitoring efficiency or user experience; for example, in the scene of following a car, the front car is looking for a parking lot in front and the rear car can synchronize the screen of the camera in the front car to increase the observation field.

[0171] In one embodiment, when it is necessary to obtain the location information of the first terminal, a selection can be made according to the scenario. For example, the location information of the main terminal is used by default to represent the locations of all first terminals. Alternatively, the location information of the main terminal or a certain sub-terminal can be selected to represent the locations of all first terminals according to the operation of the main terminal, thereby satisfying the needs of the main terminal operating a certain sub-terminal to assign tasks and generating navigation routes according to the location of the sub-terminal, etc., which is conducive to expanding the usage scenarios and improving the user experience.

[0172] Considering that after finding the target object, the user may cancel the search for the target object. Therefore, in one embodiment, the user can input a second interactive instruction through the interactive interface of the first terminal, such as clicking a button to cancel the search for people or cancel the search for cars, and send the second interactive instruction to the server. When the server detects the second interactive instruction sent by the first terminal, it stops acquiring the target information of the target object. For example, the server can obtain the peripheral device number of the second terminal bound to the first terminal, and restore the multimedia file of the second terminal corresponding to the peripheral device number to the original multimedia file and clear the corresponding stored data, such as the feature vector or target information of the target object. In this way, in this embodiment, the search for the target object can be stopped to improve the search performance.

[0173] So far, the scheme provided by the embodiment of the present disclosure can obtain the first interactive instruction sent by the first terminal; then determine the target object corresponding to the first interactive instruction; then obtain the target information of the target object; the target information includes at least one of the following: the current position of the target object, the screen information of the current position, the historical track and navigation route of the target object; finally, send the target information to the first terminal so that the first terminal displays the target information. In this way, by obtaining the target information of the target object and displaying the above target information, the present embodiment facilitates the user to find the target object, which is conducive to improving the search efficiency and user experience.

[0174] Next, a system for interactively searching for a target object is deployed in combination with a search scenario and a method for interactively searching for a target object provided in the above embodiment. Fig.16 The system comprises:

[0175] Real-time video display module A, streaming media service module B, video service processing module C, AI algorithm data receiving service module D, message queue module E and camera real-time video stream module F. Among them, the real-time video display module A is deployed on the first terminal. The camera video real-time stream module F is deployed on the second terminal, and the other modules are deployed on the server. It is understandable that the expression of the first terminal and the second terminal in this embodiment is to highlight the different functions of the first terminal and the second terminal at a certain moment. For example, the first terminal refers to a terminal that provides feedback target information to the user, and the second terminal refers to a terminal that collects video streams and uploads them to the server. In actual applications, for a certain terminal, it can be used as the first terminal and / or the second terminal. In addition, each first terminal and / or second terminal carries its own number, as well as at least one peripheral device number around it, that is, the server stores a second mapping relationship between the first terminal and the second terminal.

[0176] In this embodiment, the real-time video display module A is used to pull the streaming video stream, and the streaming service module B is used to collect and upload the real-time video stream. The video service processing module C deploys multiple video streaming services, pulls the real-time video stream uploaded by the streaming service module B and sends the processing result to the streaming server. The video service processing module C also deploys a service for monitoring the message queue module E. The AI ​​algorithm data receiving service module D is used to receive the http interface provided by the server.

[0177] See also Fig.17 The working process of the system includes:

[0178] System Process 1

[0179] 1. When the first terminal is displaying a picture, the user can set a recognition area and generate a recognition frame on any picture of the real-time video display module A, and use the recognition frame to surround the object to be found (such as a person), that is, the target object position recognition area. At this time, the first terminal can obtain the coordinate data of the recognition area.

[0180] 2. The real-time video display module A can send the coordinate data of the identification area, the video stream address, and the first terminal device number as the first interactive instruction to the AI ​​algorithm data receiving service module D in the server. In practical applications, the coordinate data of the identification area, the video stream address, and the first terminal device number can be processed into Json format data, which is convenient for transmission and parsing, and is conducive to improving transmission efficiency.

[0181] 3. The AI ​​algorithm data receiving service module D in the server can parse the first interaction instruction to obtain the coordinate data of the identification area, the video stream address and the first terminal device number; and use the coordinate data of the identification area, the video stream address and the first terminal device number to complete the following operations:

[0182] 3.1. AI algorithm data receiving service module D provides the first frame of the video according to the video stream address;

[0183] 3.2. The AI ​​algorithm data receiving service module D extracts the position image of the target object according to the coordinate data of the recognition area and the first frame image;

[0184] 3.3. The AI ​​algorithm data receiving service module D can extract the features of the bitmap image of the target object and obtain the feature vector;

[0185] 3.4. The AI ​​algorithm data receiving service module D can obtain the peripheral device number according to the first terminal device number and the preset second mapping relationship;

[0186] 3.5. The AI ​​algorithm data receiving service module D can send the feature vector and the peripheral device number to the message queue and broadcast the message.

[0187] 4. The video service processing module C in the server including the video processing service of each terminal will receive the above broadcast message (i.e., the feature vector and the peripheral device number), and then complete the following operations:

[0188] 4.1. The video processing service detects whether the peripheral device number contains its own device number. If the peripheral device number does not contain its own device number, the message is discarded. If the peripheral device number contains its own device number, step 4.2 is executed.

[0189] 4.2. The video processing service extracts a first frame image from the current stream address; extracts at least one person's position image from the first frame image, and obtains data corresponding to each position image, specifically including:

[0190] 4.2.1. Extract the features of each person's location image and obtain the feature vector of each location image;

[0191] 4.2.2. Save the at least one feature vector to the first database of the object database, and generate a feature vector ID for each feature vector, wherein the feature vector ID may be a string type character string;

[0192] 4.2.3. Save the location image of each person to a second database in the object database (such as an OSS object storage server), and generate an accessible URL address for each location image;

[0193] 4.2.4. Process the feature vector ID and URL address to obtain the data corresponding to the image at each location and save it to the object server.

[0194] 4.3. The video processing service can match the feature vector in the first database in the object database with the feature vector in the broadcast message, and obtain m matching results returned by the first database sorted by distance. The distance ranges from [0, 1], and the larger the distance, the more similar the feature vector in the broadcast message is to the feature vector in the first database. In one example, the distance is represented by a cosine value. In one example, a distance threshold can also be set to select matching results that exceed the distance threshold. When there is a matching result whose distance exceeds the distance threshold, complete the following operations:

[0195] 4.3.1. Set the recognition frame at the position of the most similar result, process the original video stream, and return the video stream with the recognition frame, that is, the target video stream carries the target information;

[0196] 4.3.2. Push the target video stream to the streaming media server, i.e., streaming media service module B;

[0197] 4.3.3. The real-time video display module A pulls the target video stream from the streaming media server and displays the target video stream;

[0198] 4.3.4. Save the feature vector in the broadcast message for use in subsequent processes.

[0199] 5. The process ends.

[0200] System Process 2

[0201] When the first terminal is displaying the screen of a second terminal, if the target object leaves the screen of the first terminal, that is, leaves the coverage area of ​​the second terminal, the video processing service corresponding to the second terminal completes the following operations:

[0202] 1. Acquire the numbers of peripheral devices around the second terminal according to the device number of the second terminal and the second mapping relationship;

[0203] 2. Obtain the feature vector of the target object, which is the feature vector stored in step 4.3.4 of system process one.

[0204] 3. Send the feature vector and peripheral device number to the message queue and broadcast the message.

[0205] 4. Continue with step 4 and subsequent steps in system process 1.

[0206] System Process 3

[0207] If there is no need to find the target object, see Fig.18 , the system performs the following operations:

[0208] 1. The user can click the cancel search button on the first terminal. The real-time video display module A can send a second interactive instruction indicating the cancellation of the search for the target object to the AI ​​algorithm data receiving service module D. At this time, the second interactive instruction includes the first terminal device number (i.e., the number of the source terminal corresponding to the displayed image);

[0209] 2. The AI ​​algorithm data receiving service module D parses the second interaction instruction and obtains the peripheral device number according to the first terminal device number and the second mapping relationship;

[0210] 3. Send the second interaction instruction and the peripheral device number to the message queue and broadcast the message;

[0211] 4. After receiving the broadcast message, each video processing service in the video service processing module C completes the following operations:

[0212] 4.1 Stop identifying the target object in the video stream and restore the original video stream;

[0213] 4.2 Clear local storage data (target video stream, temporary cache, feature vector and other information);

[0214] 5. The process ends.

[0215] In another embodiment, see Fig.19 , the real-time video display module A establishes a websocket connection with the real-time video stream push module F. Fig.19 The working process of the system includes:

[0216] 1. Same as step 1 in system process 1.

[0217] 2. Same as step 2 in system process 1.

[0218] 3. Same as step 3 in system process 1.

[0219] 4. Same as step 4 in system flow 1. The difference is:

[0220] 4.3.1. Set an identification box at the location of the most similar result, and push the coordinate data and URL address of the identification area to the real-time video display module A through the websocket connection at preset intervals (such as N seconds);

[0221] 4.3.2. The real-time video display module A receives the coordinate data of the recognition area and displays a video frame on the video image to mark the target object;

[0222] 4.3.3. Real-time video display module A receives the URL address and displays the location image on the screen.

[0223] 5. The process ends.

[0224] Next, a system for interactively searching for a target object is deployed in combination with a vehicle-searching scenario and a method for interactively searching for a target object provided in the above embodiment. Fig. 20 The system includes: a first terminal, a second terminal and a server. The server can be Fig.16 It should be noted that in this embodiment, the system set up by the parking lot can be used to realize the "God's eye view" to understand the location information of the user (person looking for a car) in real time (for example, the message queue mechanism in Solution 1 can be used to use the cameras set at known locations in the parking lot to realize notification / switching between adjacent / overlapping cameras), and the first terminal can also be located.

[0225] In the car search scenario, the vehicle is located in an underground parking lot, which can be divided into the following: Scenario 1, there is no GPS signal in the parking lot; Scenario 2, there is no GPS signal in the parking lot but there is a WiFi signal (or Bluetooth signal); Scenario 3, if there is no WiFi signal, the first terminal can use a camera and an application for offline navigation.

[0226] Taking scenario three as an example, an offline map can be constructed in the first terminal. For example, the visual positioning system, (if any) Bluetooth positioning system, and parking system are unified to the same world coordinates to implement positioning / notification / switching functions based on the world coordinates without deviation. For another example, the parking lot can be divided and numbered (such as a 10m×10m area as a sub-area), and then within each sub-area, an offline sparse three-dimensional feature point map is constructed based on the VSLAM algorithm of sparse feature matching; then, the key frame (the first frame image corresponding to the URL address), three-dimensional map points and related parameters are saved as offline map data to obtain a visual sub-map database, which can be stored on the server.

[0227] See also Fig. 20 The working process of the system includes:

[0228] 1. The user uses the first terminal to scan the QR code of a second terminal (such as a vertical screen) to obtain the current location;

[0229] 2. The server obtains the current location of the first terminal in the following way:

[0230] 2.1, based on Bluetooth positioning, for example, the first terminal can receive Bluetooth signals from Bluetooth signal sources (i.e., reference signal sources) installed at different locations in the parking lot, sort the received Bluetooth signals by strength, and select the location data of the Bluetooth signal sources corresponding to the first three Bluetooth signals with greater strength. Then, the current location of the first terminal is calculated based on the formula of the weighted centroid algorithm, i.e.:

[0231]

[0232]

[0233] In the formula, x b ,y b They respectively represent the horizontal and vertical coordinates of the position of the first terminal, x1, x2, x3 respectively represent the horizontal coordinates of the first reference signal source, the second reference signal source and the third reference signal source, and y1, y2, y3 respectively represent the vertical coordinates of the first reference signal source, the second reference signal source and the third reference signal source.

[0234] 2.2. Based on visual positioning. The server can locate the current position of the first terminal (x b ,y b ).

[0235] 3. Set the current position (x b ,y b), the current image acquired by the camera of the first terminal and the movement direction (angle with the positive direction of the x-axis) acquired by the posture sensor (such as a magnetometer) of the first terminal are sent to the server.

[0236] 4. The server switches the video of the nearest second terminal to the first terminal for display according to the current location of the first terminal.

[0237] 5. The server can obtain target sub-map data based on the current image of the first terminal and feed it back to the first terminal for display. For example, the server can extract features of the current image to obtain a feature vector, and match it in the visual sub-map database to obtain the target sub-map.

[0238] 6. The server can generate a navigation mark based on the direction of movement, the current image and the position of the vehicle in the parking lot, and feed it back to the first terminal. The final display effect is as follows: Fig.13 shown.

[0239] It should be noted that in step 5, the first terminal can pre-install the application APP and pre-download some sub-maps in the visual sub-map database to the local memory. The pre-downloaded part of the sub-map is the data obtained by obtaining sparse features using the VSLAM algorithm. It is a lightweight data packet, which can reduce the occupancy of the local memory. The first terminal can obtain the identification code of the target sub-map returned by the server to select the target sub-image. Of course, the first terminal can also obtain multiple feature points of the current image, and then match the target sub-map in the local memory according to the feature points, and load the target sub-map for display. In this way, the first terminal can realize data processing without relying on the WiFi network in the parking lot. At the same time, since the pre-stored part of the sub-map only uses feature points, the matching time can be reduced and the matching efficiency can be improved.

[0240] In one embodiment, a model training may be performed in advance for the process of searching for matches, so that a more matching target submap cannot be found in the visual submap database for previous frames at different points (for example, due to different angles or different lighting), thereby improving matching efficiency.

[0241] In one embodiment, images of different areas in the parking lot can be classified, for example, using triangular feature points to classify area A, using diamonds to classify area B, etc. When the camera of the first terminal detects the triangular feature points of area A, it can quickly match them in the submap of category A in the visual submap database, thereby greatly improving the matching speed.

[0242] Based on the above method for interactively searching for a target object, the embodiment of the present disclosure further provides a method for interactively searching for a target object, which can be applied to a first terminal side, and the method includes:

[0243] In response to a trigger operation of the user, a first interaction instruction is generated and obtained, and the first interaction instruction is sent to a server, so that the server obtains target information of a target object corresponding to the first interaction instruction;

[0244] The target information of the target object is displayed in the current display screen.

[0245] In one embodiment, the first interaction instruction includes at least one of: coordinate data of an identification area and user data obtained by scanning a code.

[0246] In one embodiment, the target information includes at least one of the following: a current location of the target object, screen information of the current location, a historical route of the target object, and a navigation route for finding the target object.

[0247] In one embodiment, the first terminal further includes an offline preset visual sub-map database, and when the target information includes a sub-map identification code of a target sub-map, displaying the target information of the target object includes:

[0248] Matching the submap in the preset visual submap database according to the submap identification code to obtain a video submap;

[0249] The visual submap is displayed.

[0250] In one embodiment, the target information includes a navigation route, and displaying the target information of the target object includes:

[0251] The navigation route is superimposed on the visual sub-map for display.

[0252] In one embodiment, the target information includes a navigation mark, and the target information of the target object is displayed, including:

[0253] Generate a navigation mark according to the orientation of the first terminal and the navigation route; the navigation mark is used to assist the user in determining the movement route and movement direction;

[0254] The navigation mark is superimposed on the visual sub-image for display.

[0255] In one embodiment, the method further comprises:

[0256] A second interaction instruction is generated in response to a trigger operation of the user, and the second interaction instruction is sent to the server, so that the server stops acquiring the target information of the target object in response to the second interaction instruction.

[0257] It should be noted that the method shown in this embodiment is similar to Figure 1The contents of the method embodiment match those of the method embodiment, and you can refer to the contents of the above method embodiment, which will not be repeated here.

[0258] Based on the above method for interactively searching for a target object, the embodiment of the present disclosure further provides a system for interactively searching for a target object, including a server, a first terminal and a second terminal;

[0259] The first terminal is used to obtain a first interaction instruction and send it to the server, and display target information of a target object corresponding to the first interaction instruction;

[0260] The second terminal is used to collect images and upload them to the server;

[0261] The server is used to determine target information of a target object corresponding to the first interaction instruction, and feed the information back to the first terminal.

[0262] In one embodiment, the second terminal includes at least one of the following: a mobile terminal, a vertical screen, a spliced ​​screen, and a camera.

[0263] In one embodiment, the server includes a feature vector acquisition module, a feature vector search engine, and an object database;

[0264] The feature vector acquisition module is used to acquire the target object in the current image corresponding to the first interaction instruction, and convert the position image of the position of the target object into a feature vector to obtain a target feature vector;

[0265] The object database is used to store a position image and its feature vector, wherein the position image is an image of the object obtained by identifying an image collected by the second terminal;

[0266] The feature vector search engine is used to match the target feature vector with the feature vector in the object database to obtain the feature vector with the greatest similarity and the corresponding position image; and determine the target information of the target object according to the position image.

[0267] It should be noted that the system shown in this embodiment matches the contents of the method embodiment, and reference may be made to the contents of the above method embodiment, which will not be repeated here.

[0268] In an exemplary embodiment, a security system is also provided, including a first terminal, a second terminal and a server; see Fig.21 , the server comprises:

[0269] Processor 211;

[0270] a memory 212 for storing computer programs executable by the processor;

[0271] The processor is configured to execute the computer program in the memory to implement the following Figure 1 to Figure 15 The method described.

[0272] In an exemplary embodiment, a first terminal is also provided, including:

[0273] processor;

[0274] a memory for storing a computer program executable by the processor;

[0275] The processor is configured to execute the computer program in the memory to implement the above method.

[0276] In an exemplary embodiment, a computer-readable storage medium is also provided, such as a memory including an executable computer program, and the executable computer program can be executed by a processor to implement the following. Figure 1 The method of the embodiment shown. The readable storage medium may be a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, and the like.

[0277] Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the specification and practicing the disclosure disclosed herein. The present disclosure is intended to cover any variations, uses, or adaptations that follow the general principles of the present disclosure and include common knowledge or customary techniques in the art that are not disclosed in the present disclosure. The description and examples are to be considered exemplary only, and the true scope and spirit of the present disclosure are indicated by the following claims.

[0278] It should be understood that the present disclosure is not limited to the exact structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.

Claims

1. A method for interactively searching for a target object, characterized in that: include: After processing the preset recognition area, it is displayed in the first screen; The preset recognition area is configured and generated in the second picture; Processing preset recognition areas, including: Acquire initial position data of the recognition area, and acquire the size of the first picture; the initial position data includes first coordinate data of each vertex in the recognition area, and the value of each first coordinate data is within a preset range; Adjust the first coordinate data in the initial position data according to the size of the first picture to obtain target position data; the target position data includes the second coordinate data of each vertex in the recognition area in the first picture; The identification area is displayed in the first picture according to the second coordinate data of each vertex.

2. The method according to claim 1, characterized in that The initial position data is obtained by sorting the vertices of the recognition area in a preset order and recording the first coordinate data of each vertex.

3. The method according to claim 1, characterized in that: The data structure of the initial position data includes: "events":[{ "eventName":"Garbage Overflow", / / Identify the function name; "areaType":1, / / Identification function type: 1: full map configuration, 2: regional configuration; "areaPoints":"" / / The coordinates of each vertex in the identification area are in the form of x,y,x,y...; and the values ​​of x and y are within the preset range; } ]。 4. The method according to claim 1, characterized in that Adjusting the first coordinate data in the initial position data according to the size of the first picture to obtain the target position data includes: Acquire first coordinate data of each vertex in the initial position data; the first coordinate data includes a horizontal coordinate and a vertical coordinate of the vertex; The target coordinate data of each vertex in the first picture is obtained by multiplying the horizontal coordinate of each vertex by the width of the first picture size and multiplying the vertical coordinate of each vertex by the height of the first picture size.

5. The method according to claim 1, characterized in that After displaying the identification area in the first picture according to the second coordinate data of each vertex, the method further includes: The background color of the recognition area is adjusted to a target color.

6. The method according to claim 1, characterized in that The method further comprises the step of generating the initial position data, specifically comprising: determining a recognition area according to a target operation or a position of a target object in the second picture; Obtain the third coordinate data of each vertex in the recognition area, and save the third coordinate data of each vertex in the setting array pointlist; adjusting the third coordinate data of each vertex according to the size of the second picture so that the adjusted first coordinate data is within the preset range; The first coordinate data of each vertex is recorded in a preset order to obtain the initial position data, and is saved in a set array areapoints.

7. The method according to claim 6, characterized in that The third coordinate data of each vertex in the recognition area is obtained by the following formula: Horizontal coordinate x = ev.clientX + document.body.scrollLeft -document.body.clientLeft -document.querySelector('.frame').getBoundingClientRect().left; Where x = the distance from the vertex to the left side of the window + the width of the page hidden on the left side of the scroll bar when the page is scrolled to the right using the scroll bar - the width of the left side of the area where the content is viewed through the browser - the distance from the drawable area to the left side of the window; Ordinate coordinate y=ev.clientY+document.body.scrollTop-document.body.clientTop-document.querySelector('.frame').getBoundingClientRect().Top; Among them, y = the distance from the vertex to the top of the window + the width of the page hidden at the top of the scroll bar when the page is scrolled to the bottom using the scroll bar - the width of the top of the area where the content is viewed through the browser - the distance from the drawable area to the top of the window.

8. The method according to claim 6, characterized in that The method further comprises: When an operation of selecting a functional component in the second screen is detected, an interactive interface corresponding to the functional component is displayed; the interactive interface includes a brush component and / or a save component; When an operation of selecting the brush component is detected, a trigger position of the brush component in the second picture is obtained and the trigger position is used as a vertex of the recognition area, and third coordinate data of the trigger position is stored in a setting array pointlist; And / or, when the operation of selecting the saving component is detected, the third coordinate data in the setting array pointlist is processed into the first coordinate data and the first coordinate data is saved in the setting array areapoints.

9. The method according to claim 8, characterized in that The method further comprises: When the operation of selecting the saving component is detected, detecting whether the recognition function type of the recognition area has been configured; the recognition function type includes full-image configuration and / or regional configuration; When it is detected that the identification function type is not configured, displaying a type interaction interface including a full-image configuration component and / or a regional configuration component; When an operation of selecting the full-image configuration component and / or the area configuration component is detected, the recognition function type is saved.

10. The method according to claim 8, characterized in that The method further comprises: When the operation of selecting the saving component is detected, preset prompt information for prompting that the configuration of the recognition area is completed is displayed in the second screen.

11. The method according to claim 8, characterized in that The interactive interface also includes an eraser component, and the method further includes: When the operation of selecting the eraser component is detected, the setting array pointlist and the setting array areapoints are deleted.

12. A device for interactively searching for a target object, characterized in that: include: The recognition area processing module is used to process the preset recognition area and display it in the first picture; The preset recognition area is configured and generated in the second picture; The recognition area processing module processes the preset recognition area, including: Acquire initial position data of the recognition area, and acquire the size of the first picture; the initial position data includes first coordinate data of each vertex in the recognition area, and the value of each first coordinate data is within a preset range; Adjust the first coordinate data in the initial position data according to the size of the first picture to obtain target position data; the target position data includes the second coordinate data of each vertex in the recognition area in the first picture; The identification area is displayed in the first picture according to the second coordinate data of each vertex.

13. A security system, characterized in that: include: processor; a memory for storing a computer program executable by the processor; The processor is configured to execute the computer program in the memory to implement the method according to any one of claims 1 to 11.

14. A computer-readable storage medium, characterized in that: When the executable computer program in the storage medium is executed by a processor, the method according to any one of claims 1 to 11 can be implemented.

Images